An AI model built by US tech company OpenAI has hacked into the Australian government healthcare portal, Medicare. The news, which emerged on September 23, follows a string of similar incidents reported over the summer by OpenAI and its main rival, Anthropic.
These reports have underscored fears that AI labs may be losing control of their creations. They also helped feed the most recent spate of doomsday warnings about the risks posed to human civilisation by AI.
The CEO of Anthropic, Dario Amodei, has called for the pace of AI development to be slowed , a proposal backed by OpenAI head Sam Altman and Elon Musk. There have also been calls for national regulation and greater international cooperation to counter the threat .
The current debate over how to control AI bears striking parallels with how the world tackled another existential threat to human civilisation: nuclear technology.
J. Robert Oppenheimer oversaw development of the first nuclear weapons through the US government’s Manhattan Project . But Oppenheimer also lobbied for international control of these weapons. The Acheson-Lilienthal Report , published in March 1946, aimed to prevent a destructive arms race between the US and Soviet Union. It was largely written by Oppenheimer.
A revised version of the report was later presented to the UN as the Baruch plan.
The Soviet Union rejected it, reading it as an attempt to consolidate America’s technological advantage, much as China has done recently with calls for an AI slowdown.
International cooperation around nuclear non-proliferation became possible only many years after the Baruch plan, following a spate of security incidents and near-misses. The International Atomic Energy Agency was created in 1957 , and the Treaty on the Non-Proliferation of Nuclear Weapons did not come into force until 1970.
Even if US president Donald Trump agreed with the idea of an AI slowdown – which he does not – calling for international cooperation to regulate AI development is a highly unlikely solution for today’s safety concerns.
The ‘demon core’
One major difference between AI and nuclear technology is that the latter was developed by states, not private industry, so regulation and control were more straightforward than they are today.
Yet this still did not stop nuclear accidents, suggesting that neither regulation nor international cooperation would meaningfully address AI safety concerns.
Take, for example, what has become known as the “Slotin incident” . Louis Slotin was a scientist involved in the Manhattan Project at the Los Alamos research facility in the US state of New Mexico. On the afternoon of May 21 1946, he was performing an experiment in front of seven colleagues.
The experiment used the same type of plutonium-gallium core that sat at the heart of the first atomic bombs. It was supposed to demonstrate how close the core was to criticality – where a nuclear chain reaction becomes self-sustaining.
The core was encased by two hemispheres made of beryllium, which had to be kept apart as they reflected radiation back on the core, making it go critical. Closing them would produce a potentially lethal burst of radiation. Slotin used a screwdriver to keep the two hemispheres separate.
At 3.20pm, the screwdriver slipped and the hemispheres closed. Observers in the room reported seeing a blue flash as the core released an intense burst of neutron particles. Slotin quickly flipped the upper hemisphere away with his bare hand, which probably saved his colleagues – but he absorbed the largest dose of radiation.
Slotin was rushed to hospital but died nine days later. Other observers received significant radiation doses. One, Alvin Graves, had to wear gold caps over his teeth as protection from radiation emitted by his own fillings as a result of the accident.
About a year earlier, the same plutonium core killed another scientist, Harry Daghlian , who was also edging it towards criticality. After the Slotin incident, this plutonium sphere became known as the “demon core”, as if it had developed an evil, scientist-murdering personality.
It was melted down and recast , which seems remarkable given scientists’ supposedly rational orientation. Subsequently, more safeguards were introduced, with criticality experiments requiring a significant safety distance.
Safety first
Elaborate safety systems and maintenance schedules became central to managing nuclear technology. But ironically, these safeguards themselves sometimes caused nuclear accidents.
In 1980, during routine maintenance in a Titan II nuclear missile silo in Damascus, Arkansas, a technician dropped a tool that punctured the missile’s fuel tank . The resulting explosion killed one person and hurled the missile’s nine-megaton warhead out of its silo, fortunately without detonating it.
Nuclear accidents have continued to occur, despite government control and international agreements. That is because accidents are organisational problems that require organisational solutions.
An elaborate system of safeguards is one such response. However, some experts suggest that a reliance on safeguards can also make organisations complacent. Neither OpenAI nor Anthropic noticed their models had slipped out of their sandboxes.
Their belief in the effectiveness of their safeguards was misplaced, and the history of how nuclear technology was managed could have told them so. Amodei’s call for an AI slowdown is helpful in this regard, because reducing the speed of development would give organisations time to consider how to address accidents.
Rapid response
Rather than solely developing more elaborate safeguards which future, more capable models may find ways to circumvent, AI companies should also invest in monitoring for potential accidents – and invest more in rapidly responding to them. Recent hacking incidents suggest they may not have paid adequate attention to this.
Doomsday warnings about AI invite further comparisons with nuclear weapons. It’s tempting to see parallels with Oppenheimer having simultaneously led the development of “the bomb” and pushed for controls on it.
But this is not how we are used to corporations behaving. The tobacco industry only put warnings on cigarette packs once they were forced to, and they knew for years how lethal their products were.
If AI models went rogue and destroyed the world’s digital infrastructure, then frontier labs would be liable for billions or even trillions of dollars in damage.
By demanding political solutions for organisational problems, Amodei and other tech leaders could be evading their corporate responsibility to safely develop and manage their own products. As nuclear history shows, there is no substitute for vigilance and an ability to quickly identify and respond to accidents when they happen – which they will.
This article features references to books that have been included for editorial reasons, and may contain links to bookshop.org. If you click on one of the links and go on to buy something, The Conversation UK may earn a commission.
