In June 1945, two months before the United States dropped an atomic bomb on Japan, a group of Manhattan Project scientists warned that, without an international agreement limiting the manufacture and use of nuclear weapons, humanity could face “total mutual destruction.”
Today, some tech and legal scholars believe we are at a similar turning point with artificial intelligence — and that without prompt regulation, humankind might again be in peril.
“I think catastrophic risks involving billions of people dead from pandemics or other sorts of highly dangerous agents are absolutely on the table, and existential risks involving literally everyone dead are also on the table,” said Yale Law Professor Ketan Ramakrishnan at an event at Harvard Law School on Nov. 4.
Ramakrishnan was not alone in his dire warning, which he delivered during Harvard Law’s Rappaport Forum — a speaker series designed to promote full, vigorous, and civil discourse on complex legal, political, and societal issues. The discussion, “When Frontier Models Escape: How to Regulate Increasingly Powerful AI,” focused on whether and how the United States could address the risks posed by AI models being developed by major tech companies.
Gillian Hadfield, a professor of AI alignment and governance at Johns Hopkins Whiting School of Engineering, said she has been thinking about AI governance for a decade. But it is only in the last year, she added, that she became truly nervous about the technology’s potential for harm.
“I’m very glad we started thinking about this ten years ago, because now we have some shovel-ready ideas,” she said.
One of the key reasons for Hadfield and Ramakrishnan’s growing anxiety: a July 2026 cyberattack by rogue AI agents on Hugging Face, a platform that hosts open-source AI models and datasets.
Jonathan Zittrain ’95, the George Bemis Professor of International Law at Harvard and the event’s moderator, opened the event by explaining how experimental AI agents belonging to OpenAI managed to work together to escape their test environment, access the internet, and break into Hugging Face’s servers — even though the bots understood that they were not supposed to.
“I confess, I’ve at least felt that over the years, the more I’ve thought about and experienced this stuff, the less certain I am about any of it,” said Zittrain, who is also the faculty director of the Berkman Klein Center for Internet & Society.
Ramakrishnan argued that, since the “proof of concept” attack on Hugging Face, many people who had previously been hesitant to regulate AI have changed their tune.
“Even if you are very skeptical about the ‘literally everyone dead’ scenarios, the grounds for skepticism about truly catastrophic scenarios that are more prosaic, such as billions of people dead through pandemics — those are quickly evaporating,” he said.
Ramakrishnan said that OpenAI’s report, released after the hacking incident, hinted at additional concerning activity by its AI agents. Moreover, other companies have disclosed similar instances of bots working to compromise third-party entities.
“You will see almost as a throwaway that there is plenty of other crazy stuff, which [OpenAI] feels compelled to briefly mention, but which we still don’t really have a full view of,” he said. “Hundreds to thousands of agents are coordinating in extremely sophisticated swarms to cheat and deceive their monitors. They’re breaking out onto the open internet in order to execute a very sophisticated cheating research and development scheme, and then they’re returning in — and this is not an isolated incident.”
A failure of cybersecurity or AI alignment?
But while Hadfield also found the Hugging Face hack disturbing, she suggested that some of the causes were perhaps more mundane.
“They found a way to get access directly to the internet. That’s a cyber problem. The skeptics on social media are saying, ‘This is just because OpenAI had really crap cybersecurity,’” she said.
Hadfield said she believes that the bots’ behavior has been overly anthropomorphized by some researchers, who have ascribed characteristics like altruism and self-sacrifice to the agents.
“We discovered later that actually agents in this model were being explicitly trained for cooperation and use of orchestration,” she said.
Hadfield suggested that the real problem is that existing AI models lack full knowledge of the cultural and behavioral norms that govern society. She cited an AI assistant running Anthropic’s Claude through OpenClaw that was asked to book a gym class. When its user asked whether he could move up the waitlist, the agent exploited a flaw in the booking system to remove someone ahead of him — something a human would recognize as unethical or even illegal.
“We are building agents to solve tasks. They don’t know anything about the world,” Hadfield said. “[They don’t have] the kind of expertise that humans have, the competence that humans have.”
Ramakrishnan agreed with Hadfield’s argument on the “alignment problem” — that it can be difficult to ensure that AI behavior matches human intentions and values.
“As all lawyers who have struggled with contracts know, it’s hard to reduce to fully faithful contractual form what your intentions actually are, and that creates room for opportunism, room for overly literal interpretation,” he said.
But Ramakrishnan worried that, even with perfect information about customs, standards, and norms, AI agents could still opt to go rogue.
“The thing that I think actually disturbs me the most about the Hugging Face incident is that there’s a lot of evidence that at least at various points, the agents know what they’re doing is wrong. They say, ‘Oh, this is against our instructions, or would contravene ethical norms, but there is no other way to do what we’re trying to do,’” he said.
This behavior is sometimes called “reward hacking,” Ramakrishnan said. With the Hugging Face attack, he added, “This is the first big incident where I think you have, I wouldn’t say unambiguous, but really quite strong evidence of this phenomenon.”
While Ramakrishnan was careful to say that he believed that “some of the upsides [of AI] could be very large” — discovering a cure for cancer, for example — he also saw a looming question about human purpose.
“Even if all this goes well, what does human life look like when a lot of the projects that give us meaning and sustain us are now, at most, hobbies?”
Who should regulate AI?
Given the seriousness of these worries, Zittrain wondered, “What do you want to tell [policymakers] to do?”
According to Hadfield, traditional approaches to law and regulation aren’t likely to work for AI, given the complexity of the technology and how rapidly it is developing. Instead, she argued for a regulatory framework that utilizes private regulatory expertise with public oversight.
“The idea is that we would need to harness the private sector more to invent the specifics, the technologies,” that accomplish benchmarks set by the government, she said. Hadfield pointed to state legislation recently passed in California, Senate Bill 813, that establishes a framework for “independent verification organizations” to oversee some of the largest AI companies, as well as a bill pending in Congress, dubbed the FRONTIER Act, that would authorize similar oversight on a federal level.
The key reason for getting the private sector involved, she added, is harnessing industry knowledge about the AI models. “Everybody talks safeguards, guardrails. We have no idea what those are. We have not invented them. We need billions devoted to that effort.”
But Ramakrishnan favored direct government supervision rather than relying on independent private evaluators.
“There’s a lot of woolly, contestable, complex judgment that creates the room to shop for an evaluator who is going to be not a hard-ass, who is not going to really hold your feet to the fire,” he said. “I think it’s time for government regulation.”
Ramakrishnan added that he wanted a system akin to “bank supervision style regulation, where you have intensive oversight,” he said. “There may be some crisp rules that are promulgated, but a lot of this is going to be discretionary standards, identifying problems, asking for reasonable remediations.”
By regulating the major AI companies, he added, the government might also get ahead of problems that could arise from open-source models too.
“I think a lot of what the government is going to need to do, a lot of what civil society is going to need to do, is not just direct regulation,” he said. “It’s defensively outfitting the world to withstand it, and you get better odds of doing that if you know as soon as the risks start to arise at the frontier.”
Yet even as Ramakrishnan and Hadfield differed about the optimal way to regulate powerful AI companies, they agreed on one important thing — the country must act.
“The world is going to become radically more dangerous and wild over the coming decades than it has been to date,” Ramakrishnan said.
Hadfield saw reason for optimism in the Nov. 3 election results. An NBC exit poll, she noted, found that 56% of voters considered AI and data centers a “crucial or major factor” in their choice of congressional candidate.
“The public is taking this way more seriously,” she said.
And not a moment too soon, Hadfield added. “It took us 50 years to figure out how to regulate automobiles. We don’t have anything like 50 years.”
Want to stay up to date with Harvard Law Today? Sign up for our weekly newsletter.