In February, the United States Department of Defense threatened to designate the AI firm Anthropic a “supply-chain risk” after a dispute over the military’s use of the company’s Claude models.
Author
- David Silver
Chair in Business and Professional Ethics, University of British Columbia
This designation would not only put the company’s $200 million Pentagon contract at risk, but would require all defence contractors to cut ties with the company.
At the centre of the dispute were two uses that Anthropic said it would not permit: fully autonomous weapons and the mass surveillance of Americans.
The Pentagon wanted Anthropic to allow the military to use its AI for ” all lawful purposes .” Anthropic CEO Dario Amodei said the company did not believe current frontier AI models were reliable enough for fully autonomous weapons and argued that mass domestic surveillance would endanger democratic values.
U.S. President Donald Trump responded by directing federal agencies to stop using Anthropic’s technology , while allowing agencies like the Defense Department six months to phase out existing use.
Anthropic challenged the designation in court, and on March 26, a federal judge issued a preliminary injunction blocking its enforcement while the litigation proceeds.
The dispute raises a question larger than the future of one government contract. In my book Corporations and Persons: A Theory of the Firm in Democratic Society , I argue that a company’s responsibilities are owed not primarily to an administration, but to the entire democratic constitutional order.
I have previously argued that this includes a duty to resist autocratic demands. Here, I want to focus on a particularly urgent version of that responsibility: the duty to warn.
Two kinds of warning
Companies often know things that democratic citizens need to know in order to govern themselves. For example, Facebook’s internal research raised serious concerns about Instagram’s effects on teenagers’ mental health. Those findings became public only after former Facebook employee Frances Haugen came forward as a whistle-blower.
A company’s obligation in such cases is straightforward: tell the public what you know, and certainly do not follow the tobacco industry’s strategy of actively preventing people from knowing .
At one level, Anthropic is doing something similar. It is warning democratic society that it faces important choices about how governments use AI. But there is a second, more urgent warning about whether democratic society can keep making decisions like this at all.
Disinformation on social media raises a similar problem. It is not simply another problem for democratic society to manage. It can attack the shared epistemic and social commitments that make democratic self-government possible in the first place.
AI and the threat to democracy
Much of the concern about AI and democracy focuses on what AI allows politicians to do: spread misinformation, manipulate citizens or conduct surveillance. A deeper concern is what AI may do to the people and institutions capable of stopping those abuses.
Democracies have never relied only on laws and elections to check abuses of power. They’ve relied on people – a great many of them, spread across a great many independent roles: the officer who can refuse an unlawful order, the engineer who can decline to build something, the analyst who notices an anomaly and says so, the journalist a whistleblower can call. People provide a safeguard.
Democracies are resilient because responsibility is spread across many independently situated people. Any single official can be replaced or pressured; a whole system of people, each bringing an independent sense of personal and professional ethics, is harder to fully compromise.
The Cambridge Analytica scandal is a case in point. Facebook sat on evidence that user data had been improperly obtained and used by the political consultancy Cambridge Analytica. The scandal only became public after former Cambridge Analytica employee Christopher Wylie came forward and provided information to journalists.
Such moments depend on there being enough independent people, in enough independent positions, for at least one of them to decide to speak. This is precisely what the autocratic use of AI threatens to dismantle.
A human in the loop is not enough
Anthropic’s proposed contractual safeguards partly address this concern by insisting that a human being remain in the “kill chain,” the sequence of decisions before a lethal strike.
However, that doesn’t fully address the problem. A government doesn’t need to remove humans altogether to defeat this safeguard. It only needs fewer and more dependable ones: people selected, incentivized or pressured to comply rather than object.
This is what makes the current generation of frontier AI especially concerning. Automating surveillance and decision-making at scale threatens to dismantle the human network through which democracies have caught abuses of power before they became entrenched.
The danger is compounded by what these technologies can do. Mass surveillance and autonomous weapons are precisely the kinds of capabilities that enable authoritarian governments to consolidate their power over society.
In this regard, Anthropic has both a duty to warn democratic society about those risks and a duty to resist an autocratic demand that would contribute to them.
Not a licence to govern
None of this means Anthropic should decide how AI gets governed. A company convinced it understands a risk better than its regulators, and therefore does not need their oversight, is making a serious mistake. Superior knowledge creates an obligation to inform democratic decision-makers, not a licence to replace them.
Very few companies anywhere occupy the position frontier AI firms currently do. They possess technical capabilities that governments urgently want, while governments have enormous purchasing, regulatory and coercive power over them.
AI may give governments extraordinary new powers. Democracy will depend, in part, on whether enough people remain willing and able to recognize when those powers are being misused, and to point it out.
