Some in the AI community call it a “Chernobyl moment”: the fear that a catastrophic AI-related incident — from a hack that crashes the financial system to the release of a lethal bioweapon — could stop development of the technology in its tracks. Anthropic’s Claude Mythos is the first AI model that has made such risks seem real. Warning that Mythos’ exceptional ability to sniff out flaws in coding could, in the wrong hands, be used to bring down critical infrastructure, Anthropic has restricted its initial release to a group of chosen partners. Other risky models will follow, quickly.
White House recently scrapped an executive order that would have standardised pre-release safety testing of advanced AI models. Once again, those advocating a trust-but-verify approach lost to the no-holds-barred camp. AI as a whole requires light and agile regulation that does not stifle innovation in a world-changing technology. But even with Pope Leo XIV warning that AI must be “disarmed”, cutting-edge frontier models — those with “nation-state” capabilities — are an area where setting up a vetting system is a matter of urgency. A formal mechanism is needed to understand just what these systems are capable of and decide how, or whether, to release them.
AI currently operates in an accountability-free zone. As it replaces existing systems throughout the economy, its black-box algorithms cannot easily be bound by rules written for humans. The question is no longer whether these tools will proliferate, but whether the terms are set by democratic choice or the companies moving fastest. The risks of getting this wrong are serious. Chatbots giving medical advice face neither the malpractice liability nor the licensing oversight of human doctors. In the US, governments can use error-prone AI systems to deny benefits without citizens having recourse to an explanation from a human or an appeal to a human.
US courts are struggling to apply existing law. A lawsuit against OpenAI alleges product liability and competition violations, which it denies, for conduct for which “every therapist, teacher, and human being would face criminal prosecution.” The court must decide whether a chatbot is even a product, and if it can be held liable for competing unfairly with humans by doing what humans legally cannot.
This “accountability arbitrage” to gain market dominance is not new. US social media companies perfected the playbook by using a legal loophole to sidestep media norms requiring news outlets to carry liability insurance and uphold editorial standards. Uber said it was a technology platform, not a taxi company; its drivers were gig workers, not employees. Airbnb argued it owed no hotel taxes and was not bound by zoning or building-safety rules.
That makes it all the more unfortunate that the Trump administration postponed, at short notice, an executive order that would have established a voluntary testing regime between US-based frontier AI companies and the government to study new models for 90 days before public release. President Donald Trump said he “didn’t like certain aspects of it” and wanted nothing to “get in the way” of America’s AI lead over China.
The Trump White House has generally scorned AI “safety” policies as unwarranted hindrances to US tech giants. But some officials were said to have been persuaded of the need to act on frontier models after Treasury Secretary Scott Bessent met Wall Street bosses in April to discuss the risks posed to the financial system by Mythos. The resulting executive order was then reportedly delayed after lobbying from other officials and some tech titans.
The draft order was flawed and narrow. It focused largely on cyber security rather than broader risks, relied on voluntary co-operation by US firms, and limited pre-release testing to US agencies and “select trusted” partners chosen together with the government. Not mentioned in leaked drafts was a body set up by the Biden administration specifically to monitor AI risks and stress-test frontier models, but which Trump officials have rebranded and refocused on prioritising innovation. Perhaps inevitably, given that most (though not all) frontier model developers are American, the draft treated AI as, in essence, a proprietary US technology and its security as a US affair.
The order, however, represented an important first step. In the interests of international safety, a version that is not watered down from the original should be issued without delay. It is in the interests of the White House, too; a catastrophic incident could torpedo the industry on which the administration is pinning hopes for enhanced growth and continued superiority over China, and which has become a mainstay of the US stock market.
A US-led framework may be, for now, the best the world can hope for. But the longer-term aspiration must be to develop an international vetting network. Though other countries lack AI companies the size of America’s, they do not lack AI knowhow. The UK’s AI Security Institute — the only non-US government agency given access to Mythos — has become the kind of repository of expertise that its US counterpart, in other circumstances, might have done. When it comes to frontier AI, the goal must be to put in place a monitoring system that prevents a Chernobyl moment from ever happening — not to be forced to do it in the aftermath.
