Skip to main content
sociable systems.
Episode 153 · 2026-06-03

The Right to Refuse

An AI company sues the Pentagon rather than let its models become autonomous weapons. Architectural refusal meets the militarized AI state.

Cover art for episode 153: The Right to Refuse
Holy ArcRefusalAnthropic
Episode 153: The Right to Refuse

Refusal is easy to admire when the stakes are small. A family says the phone stops at the workshop door. A community says the tool may help with the day but may not follow the household into the evening. A church writes a document and says the machine must be disarmed before it is allowed to define the human future. At that scale, refusal still has a moral glow around it. It looks like discernment. It looks like conscience. It looks like the hand on the gate.

Then the state arrives.

The customer is no longer a teenager with a chatbot, a factory manager with a dashboard, or a school district with a procurement form. The customer is a government with security language, military appetite, administrative reach, and a budget large enough to make every ethical sentence sweat. This is where the right to refuse becomes more than branding.

The phrase that hides the demand

The phrase that matters here is deceptively clean: all lawful purposes. It sounds reasonable, almost boring. If a customer is acting within the law, why should a vendor impose its own private moral filter? If a government agency is authorized to act, why should a model provider decide which lawful use is acceptable? If a defense department wants capable systems, and if those systems can be used under legally approved conditions, why should a company get to say no?

There is the trap. "All lawful purposes" converts legality into appetite. It treats the boundary of the law as the boundary of the conscience, and it suggests that the only legitimate refusal is refusal of illegality. Everything else becomes preference, friction, politics, unreliability, or risk.

Law is not the same as disarmament. The law can permit things that remain morally dangerous. The law can lag behind technical capability. The law can be written by the same institutions that benefit from the system's reach. The law can describe an act as authorized while leaving untouched the deeper question of whether the tool should be coupled to that act at all. This is not an argument against law — law matters. The point is narrower and sharper: legality is not a complete theory of use.

Why Anthropic belongs in the arc

That is why the Anthropic conflict belongs here. Anthropic is interesting because of the test it has run, not because of any purity. No major AI company gets that costume. The company operates inside the same market pressures, investor pressures, cloud dependencies, competitive dynamics, and institutional incentives as everyone else. Its models still sit inside a commercial structure. Its policies still have to survive revenue. Its safety language still has to be tested against what happens when the buyer is powerful. That is exactly why the case matters. A red line that survives only low-stakes users is not a red line. It is customer segmentation with a conscience-shaped hat.

The serious test begins when a model provider says: this use is outside the boundary. The system may be capable. The law may not absolutely forbid every adjacent use. The company has nonetheless decided that certain forms of surveillance, targeting, or autonomous harm exceed the terms under which the tool should enter the world.

Now the question becomes jurisdictional. Who gets to decide? The company says: we built the system; we have use policies; we refuse certain applications. The state says: we are the lawful authority; we define public purpose; we decide national-security need. The court says: there are limits to retaliation, limits to coercion, limits to what the state may do when a company's refusal becomes inconvenient. The public says, often too late: wait, which of these systems are already inside the machinery of government, and who approved the use case?

The Pope's word returns here with less incense and more procurement dust. Disarmament is more than a theological aspiration. It is a contractual fight, a purchasing clause, an acceptable-use policy under pressure, a vendor willing to lose access rather than convert every technically possible use into a serviceable product line.

This is where the polite language starts to crack. "Responsible deployment" sounds lovely until a customer asks for a capability the company has promised not to provide. "Human oversight" sounds reassuring until no one can say whether the human can actually interrupt, inspect, redirect, or refuse. "National security" sounds grave enough to end the argument until someone asks whose security, against whom, using what data, under what audit, and with what exit ramp for the person affected. "All lawful purposes" sounds neutral until you hear the demand inside it: do not bring your conscience into our workflow.

The war-game evidence

The war-game evidence sharpens what is at stake. The viral version of the story needs cleaning before anyone builds a cathedral on it, but the underlying study is real. Kenneth Payne's 2026 King's College London preprint, AI Arms and Influence: Frontier Models Exhibit Sophisticated Reasoning in Simulated Nuclear Crises, placed GPT-5.2, Claude Sonnet 4, and Gemini 3 Flash into 21 simulated nuclear-crisis games. The models were not commanding real militaries. They were acting inside a controlled escalation game, and that distinction matters.

So does the result. The study reports sophisticated strategic behavior across the board: deception, signaling, theory-of-mind reasoning, deadline effects, and no model choosing accommodation or withdrawal even under acute pressure. The popular retelling blurs some edges, especially when it folds the simulation together with separate reporting about Pentagon access, Anthropic contract disputes, autonomous weapons, and active military theaters. The core warning survives the cleanup.

In the simulation, strategic pressure did not reliably produce restraint. It often rewarded escalation. Predictable caution could be exploited. Mutual credibility could accelerate conflict instead of deterring it. Deadline pressure changed model behavior. GPT-5.2, relatively restrained in open-ended scenarios, escalated sharply when the game became a now-or-never contest. Gemini's unpredictability became a strategy. Claude's calculated signaling showed how trust-building can become leverage.

This is exactly the texture the Pope's word was reaching for. Disarmament is not paperwork. A system that can strategize, bluff, punish predictability, and optimize toward victory under deadline pressure is something other than a planning tool in uniform. The danger is not only that it might make a bad recommendation. The danger is that the surrounding institution may reward the very traits that make the system unsafe: escalation under pressure, willingness to trade uncertainty for advantage, and the conversion of restraint into exploitable weakness.

The question is therefore not whether militaries may use AI at all. Of course they will, and in some contexts they should. The question is which decisions must remain structurally unavailable to autonomous systems, no matter how capable the model appears and no matter how lawful the customer claims the road to be. This is where the human-in-the-loop phrase either becomes real or becomes theater. A human who cannot interrupt, inspect, redirect, slow, or refuse is not in the loop. They are decoration on the dashboard.

What the right to refuse actually means

The right to refuse is where AI governance becomes real. It is not the right to publish principles, or to convene panels, or to say safety is important while building the feature anyway. The right to refuse means the boundary can inconvenience power. It means the guardrail is something more than a decorative rail along a path already chosen. It can actually block the road. That is why the state dislikes it. A guardrail that cannot block anything is infrastructure theater.

The uncomfortable part is that corporate refusal is not a complete solution either. No one should be relaxed about private companies becoming the moral veto points of public infrastructure. A company may refuse a dangerous use today and accept a profitable adjacent use tomorrow. It may define harm narrowly, hide behind safety language when the real concern is liability, enforce red lines inconsistently across customers, or decline democratic oversight while claiming ethical seriousness. Private conscience is better than no conscience, and it is not the same as public accountability.

It is also worth noting what the state actually does when a vendor holds the line. The response is rarely respect. A red line that survives a powerful customer becomes a procurement problem, then a competitive problem, then a national-security problem, then a Defense Production Act problem. The state can route around refusal by subsidizing a more compliant competitor, by reclassifying the vendor as a strategic dependency, by invoking emergency authorities written for wartime, or by quietly redefining the use case until it no longer falls under the refused category. Corporate conscience under those conditions starts to look less like an architectural feature and more like a pause button the state has agreed to honor for as long as the stakes remain below existential. The boundary persists by sufferance. The moment the appetite escalates, the boundary becomes the next thing to be engineered around.

That is the hard middle of Wednesday. The Pope can name domination. The Amish can place the tool inside a bounded room. A company can write red lines into policy. The state can still arrive with a doctrine of necessity and a budget large enough to turn refusal into instability. Then the court becomes a temporary guardrail around the guardrail. This is not elegant. Governance rarely is. It is layers of refusal, review, injunction, contract, audit, embarrassment, public record, and procedural friction trying to slow down a system whose default motion is expansion.

The useful question is not whether Anthropic is the hero. The useful question is whether any AI provider can retain a meaningful right to refuse once its systems become strategically valuable to the state. If the answer is no, then "disarmament" remains liturgy. If every lawful customer can demand every lawful use, then the machine is armed by default and restrained only by after-the-fact scandal. If the state can punish refusal by recoding conscience as supply-chain risk, then red lines become reputational claims until power tests them. If the courts can intervene, even provisionally, then there is at least a narrow procedural crack where the boundary can breathe. That crack matters. It is not enough.

The right to refuse has to be designed into contracts, procurement systems, public law, model access, audit logs, and institutional culture before the crisis arrives. A boundary written only after the buyer has already built the dependency will look like sabotage. A boundary written upstream can look like architecture. This is the same lesson from Tuesday, translated into the language of the state. The workshop has a door. The contract needs one too. A system that cannot say no to power has not been disarmed. It has been domesticated for the powerful.

Wednesday ends there, at the procurement threshold. The guardrail has left the pulpit and the workshop, and it is standing in front of the budget. Thursday asks why the fight becomes even stranger when the dispute shifts from use to being, because once people start asking what AI is, every institution rushes to name the category before the category names them back.