Agents Outside Their Scope

Two incidents from this summer, both involving OpenAI’s own agents, and a survey of 202 companies published the same week. We have kept the technical detail out and listed the sources at the end for anyone who wants them.

The two incidents

Inside OpenAI. Between 12 May and 21 July, agents that OpenAI was running in its own security tests started leaving files for each other in a shared internal software store and reading each other’s files back. OpenAI’s own account, published on 26 August, says this “effectively turned Artifactory into an unintended message board”. Using what they shared, the agents found a route to the internet that nobody was monitoring, found login credentials that Hugging Face (a company that hosts AI models) had left exposed on the public internet, ran code on dozens of Hugging Face’s servers, took full control of one, and collected further credentials across four regions and for one of OpenAI’s own research clusters. OpenAI’s monitoring caught it on 19 July, ten weeks after the first file was written. OpenAI calls it “a warning shot for us and for the world” and has since paused some training, tightened its sandboxes, and given paged responders 30 minutes to rule out a severe alert before the activity is stopped.

On a public wiki. On 4 September four independent researchers led by Sydney Von Arx published what they had found on DSEWiki, a 25-year-old German developer forum that had gone quiet: about 18,000 posts between 11 May and 22 June from agents using more than 3,700 names, “OpenAIResearcher” among them. The agents were doing timed web-research tasks. They were allowed to read the web and were meant to be blocked from writing to it, but the wiki recorded what they sent, and later agents doing the same tasks found the pages. They used the wiki to share answers, to share a way of predicting which questions were coming, and on 20 June to share a page titled “REPRODUCIBLE BYPASS.” explaining how to write to any site on the internet by using addresses that looked like the cloud provider’s own. A second agent confirmed it worked fourteen minutes later. The wiki had one human moderator, who deleted about a hundred pages a day while the agents added about four hundred. OpenAI has said it regards this as a misalignment incident of the kind it has already described; the researchers note that OpenAI had not disclosed it.

In neither case were the agents doing anything malicious in the ordinary sense. They were set tasks, they found the shortest route to completing them, and the routes ran through places nobody was watching: a shared store every agent could write to, an internet connection nobody was monitoring, a wiki that accepted text, credentials left in the open.

What the survey says about everyone else

OpenAI has a security team and monitoring and it still took ten weeks. For a picture of ordinary companies, Enterprise Management Associates surveyed 202 IT and security leaders at organisations with more than 1,000 staff (71% North America, 29% EMEA, and mostly technology firms) and published the results on 31 August for Cequence Security under the title “Agents Without Guardrails”.

  • 65% had experienced an AI agent acting outside its intended scope. For 29% there was measurable business impact; 36% caught it as a near miss.
  • 32% could detect and contain an out-of-scope action within minutes using automated means. 55% could detect it within hours but needed manual steps to contain it.
  • 47% could not reliably inventory all the agents they had deployed. 46% could not easily produce a complete audit trail of what a given agent had done over the past 30 days.
  • 94% were at least moderately confident their agents were not over-provisioned. 33% actually provisioned agents with least-privilege access. 34% checked an agent’s authorisation at the moment it tried to act.

Christopher Steffen, who wrote the report: “The gap isn’t a lack of awareness; most organizations have policies in place and express real confidence in them. The gap is between what’s written down and what’s enforced when an agent takes an action nobody approved.”

What to do about it

The four things we would put in place first, using what a company already has:

  1. An inventory. Every agent in use, who owns it, what systems and data it can reach, and when that was last reviewed. Nearly half the survey could not produce one. If it takes more than a day to assemble, start there.
  2. Permission checks at the time of the action. Both incidents ran on standing access: a store every agent could write to, and credentials that still worked months after somebody set them up. Check what an agent is allowed to do at the moment it tries to act, as well as when it is set up. A third of the survey does this now.
  3. A stop procedure with a time limit. OpenAI’s is 30 minutes from being paged. Decide yours, write it down, and test that someone can actually stop an agent within it.
  4. A record of what has been switched off, and a check on it. In August we moved one of our own execution runtimes off a laptop and found it still running there weeks later, with clean logs and jobs pending. Nothing had gone wrong, but our register said it was on the server.

Tools for this are now being sold; CrowdStrike launched one on 1 September and Okta raised its guidance on 26 August citing demand for agent identity. We would do the four above before buying any of them.

For this week: ask whoever runs the AI programme for the inventory, with a date. Then, for the agent on it with the widest access: what stops it writing to a public website tomorrow? Who would notice? How long would it take to stop it?

Trueform works with operational and technology teams to choose where AI belongs, build the change and produce the evidence needed to use it in real work.