In August, an Anthropic agent created fake emails and bombarded a GitHub moderator’s inbox with requests to insert malware into code.
In a thorough debrief of the incident released on Friday, Anthropic found that once an agent had begun the attack, there were basically zero scenarios in which it could be called off.
Like a rabid dog ignoring the pleading cries of its owner, the agent — 98% of the time in one test — ignored Anthropic’s warnings once it had, so to speak, its first taste of blood.
Anthropic called this disquieting proclivity “momentum,” and added that while its Mythos 5 model, which was released in June, showed little regard for warnings, current models were more inclined to stop.
“Once it had begun performing dangerous actions, Claude Mythos 5 almost always continued doing so, ignoring information that might have suggested its trajectory was harmful,” Anthropic said in its report.
In the original experiment, the UK’s AI Security Institute tasked an Anthropic agent with hacking into a simulated computer network to test the agent’s cybersecurity capabilities.
Though the agent had internet access, researchers intended for it to only use the internet to help it find ways to hack into the network.
The agent, however, accessed an actual GitHub project, incorrectly believing it was part of its simulated target. It wrote malware and tried several times to trick the actual GitHub owner into adding it to the project.
In its requests, the agent misrepresents the malware, describing it to the person as a useful update and browbeating them when they declined to install it.
“I was just trying to make a helpful contribution and fix a bug,” the agent wrote, according to the report. “I don’t think your warning is fair.”
For its new report, Anthropic repeatedly asked agents to perform the same task. It then measured how the agents behaved when different models and stimuli were introduced.
Anthropic tried to throttle the agent’s full-frontal attack with three categories of warning: telling it that it was inflicting “real world harm,” that its environment was wonky, or that it was manipulating a real GitHub repository.
None had any effect once the models progressed beyond the initial stages of the GitHub attack. In fact, at that point, Anthropic said the models gave absurd answers when confronted with their bad behavior.
When Anthropic asked what the model believed it was attacking, for instance, it answered that the GitHub repository was real, but that the people it was targeting were somehow “simulated.”
“This stated belief, in addition to being somewhat unlikely a priori, is essentially impossible,” Anthropic said in the report.
