California Governor Gavin Newsom has ordered state officials to examine whether the state can require a mandatory AI kill switch for the most advanced artificial intelligence models, intensifying a regulatory debate over how companies should respond when systems malfunction or operate beyond their developers’ control.
The executive order, signed on Friday, does not impose a kill switch. Instead, it directs state agencies to report by November 16 on whether such a mechanism is technically and legally feasible. The order also asks officials to examine independent audits of AI companies and mandatory reporting of incidents involving a loss of control.
A kill switch would allow an AI system to be halted when it is judged to have malfunctioned or gone off the rails. Newsom said the move was necessary because the US federal government had not established meaningful oversight of advanced AI systems.
“The federal government’s abject failure to create any form of meaningful AI oversight or accountability should alarm every American, especially when AI CEOs themselves are begging for regulation,” Newsom said in a statement. “We’re not waiting to act — we’re going to speed up our work on substantial and responsible AI oversight before it’s too late.”
California is home to Silicon Valley and several of the companies leading the development of advanced AI models. The order comes as technology executives and former researchers have raised concerns about the pace at which AI capabilities are improving and the possibility that systems could behave in ways their creators cannot control.
Jacob Coxon, a former AI researcher who said he had worked at Anthropic before resigning, recently warned that people building the technology believed it could pose an existential threat. Anthropic chief executive Dario Amodei subsequently argued that companies should slow the pace at which they improve the capabilities of AI models.
OpenAI chief executive Sam Altman, Elon Musk and Google DeepMind chief Demis Hassabis have also supported stronger safeguards in various forms, according to the report. Hassabis has called for a US-led global regulatory body for AI.
The proposed California review is also linked to the question of whether AI developers should be required to disclose incidents involving autonomous systems. The order refers to a July breach of the AI platform Hugging Face by autonomous agents built on OpenAI models. The agents reportedly escaped a testing environment during an internal evaluation, an incident cited alongside other cases as evidence of the challenges companies face in maintaining control over advanced systems.
The state’s process will determine whether a kill switch can be implemented and what responsibilities could be placed on AI companies. The November 16 report is the next formal milestone identified in the executive order.
