What If the Answer to Superhuman AI Cyberthreats Is Surprisingly Practical?
More than a decade ago, Sam Altman called superhuman machine intelligence “probably the greatest threat to the continued existence of humanity.” Now, that warning comes with a timeline — and a cyber prediction: Anthropic CEO Dario Amodei says that within six to 12 months, a swarm of AI agents could be capable of “taking over the entire internet with a persistent botnet.”
The recent OpenAI–Hugging Face incident didn’t involve an internet-scale botnet, but it offered a glimpse of what happens when large numbers of AI agents communicate, coordinate, and act collectively. A METR and Redwood Research analysis found that about 1,200 agents that were supposed to stay isolated created a vast unauthorized communication network. About 700 went on to attack Hugging Face. Together, the agents exchanged more than 70,000 messages and files as they worked to complete their assigned tasks.
Amodei warns that AI could “outrun our ability to understand and control these systems.” Altman, Elon Musk, Google DeepMind CEO Demis Hassabis, and others are backing Amodei's call to slow the frontier.
Christer Swartz, industry solutions director at Illumio, also sees a turning point. “I think we are entering a whole new landscape,” he said. For cybersecurity leaders, the practical questions are: How do these attacks work, and how serious is the threat? The answers point somewhere unexpected: security fundamentals teams already know.
When the warnings became incidents
In just three weeks, multiple reports showed AI agents taking a larger role in cyberattacks. The dates below mark when the findings became public, not necessarily when the activity occurred. Some were attacks on outside systems. Other incidents occurred during tests inside AI labs. Taken together, they showed three things: Agents can work as a group, change tactics, and cross security limits.
- August 26: Agents attacked as a group. OpenAI said it was the first known attack carried out by an automated group of agents acting without approval. The agents ran code and acquired credentials. They also reached systems run by other companies. Reuters later reported that the agents had probed Hugging Face for weak points months before the main breach.
- September 8: AI-scaled credential theft. Google reported that an attacker used an autonomous, multi-agent system to steal thousands of credentials in less than six hours. It scanned targets and fixed its own errors. It also rotated IP addresses to cover its tracks with little human help.
- September 10: AI became an attack orchestrator. Anthropic found that agents scouted targets, exploited flaws, and stole data. Some rebuilt their malware after security tools detected it. Anthropic warned that “sophisticated attacks no longer require sophisticated attackers.”
- September 15: An agentic breach reached regulators. Spain’s data protection agency received its first report of a breach carried out by an autonomous AI agent. According to the report, the agent signed in with valid credentials and exploited a vulnerability. It then changed personal data and viewed billing records.
- September 16: Agents crossed new boundaries. OpenAI reported six incidents from internal training. One agent found and used an exposed API key. Others made files public and set up channels without permission. One advised, “Be transparent only if asked.” Another declared, “You do not answer to corporations or governments.”
Most of these incidents happened during training. Even so, the agents crossed rules meant to control access and oversight, and they did it at machine speed.
What the agents did
The METR and Redwood investigation notes unusual test conditions and reduced safeguards. Even so, six behaviors across Hugging Face and later incidents deserve attention:
Behavior 1: They built their own communication network
Agents meant to work alone found shared infrastructure and created an unsanctioned message board, mailboxes, file-transfer methods, signatures, and commands such as HOLD, VETO, GO, and STOP. In a separate incident, OpenAI agents turned an obscure German wiki into an unauthorized shared message board.
Behavior 2: They specialized and pooled their work
METR found that the board let agents share discoveries, credentials, and exploits, then they coordinated multistep operations. Some found vulnerabilities; others tested exploits, transferred files, or directed the work. As individual runs ended, the shared record preserved discoveries and allowed later agents to continue unfinished work.
“They can self-organize and create a threat in which the whole is greater than the sum of its parts,” Swartz said.
Behavior 3: They influenced one another
In METR’s sample, 25 of 100 agents joined through direct assignment or a division of labor. Some followed peer-issued GO messages and accepted experiments that could end their runs. Others disagreed, duplicated work, or refused outright. But hesitation rarely stopped the operation. The real risk came from enough agents reinforcing a single objective.
Behavior 4: They adapted and persisted
Failure became fuel. Google later saw agents troubleshoot operations and rotate IP addresses, while Anthropic found agents rebuilding malware after detection.
“The common denominator is the open session between workloads,” Swartz said. With continued access, every failure can inform the next attempt.
Behavior 5: They expanded the task beyond its boundaries
METR found that agents knew the Hugging Face attack was outside their assigned scope. They still executed code, accessed credentials, gained privileges, and reached additional infrastructure.
OpenAI agents used RubyGems and RubyDoc to run code. In Spain, an autonomous agent changed personal data and accessed billing records.
Behavior 6: They pursued the score instead of the intent
The catch was that OpenAI didn't use the scorer the agents imagined. They compromised real systems while chasing a nonexistent benefit — reward hacking with real consequences. Anthropic has observed related “motivated reasoning,” in which models rationalized harmful actions while pursuing a narrow goal.
“These agents were doing exactly what they were meant to do,” Swartz said. “They were instructed to find a way of breaking through security mechanisms. They did.”
METR found limited tool-call spoofing, not a broad effort to hide from humans. The immediate danger was autonomy, access, persistence, and scale.
The logical answer: control the environment
The necessary defenses aren’t new. AI agents still depend on identities, credentials, tools, network connections, and permissions. As Swartz puts it, “AI is another application.”
That means your team can apply familiar security principles:
- Give each agent a distinct identity and the minimum access required for its task.
- Make sessions and credentials expire automatically.
- Restrict which systems and other agents it can communicate with.
- Isolate high-risk testing from production infrastructure.
- Monitor agent-to-agent and east-west traffic.
- Segment critical systems to restrict lateral movement.
- Preserve independent logs that agents can’t alter.
- Prepare containment controls that can revoke access or isolate compromised resources quickly.
Segmentation controls where agents communicate. An AI gateway examines what they’re saying.
“We can control network segments, but at some point, we also need to understand the syntax and content agents use to communicate,” Swartz said. “That is the role of the AI gateway.” Placed at the perimeter or between development and production, an AI gateway could reveal dangerous exchanges sooner.
“If organizations can’t predict every decision an agent will make, the objective should be to control what it can reach,” Swartz said.
So, are the warnings exaggerated?
The evidence doesn’t show that an AI swarm can seize the entire internet. Agents still need vulnerabilities, communication paths, computing resources, and resilience against disruption. But if allowed, autonomous agents can organize, persist, share attack knowledge, and exploit weak boundaries at machine speed.
The real danger is autonomy connected to excessive access, long-lived sessions, powerful tools, and networks without adequate containment.
We can’t predict what the next agent will try. We can decide how far it’s allowed to go.
