CSIPE

Published

- 17 min read

An AI Test Needs a Network Boundary, Not Just a Prompt

AI Agents AI Security DevSecOps Security Testing

Books by the author

Compare all 5

As an Amazon Associate I earn from qualifying purchases. Buying through these links costs you nothing extra and helps pay for the blog.

A security test gave an AI model a fictional company to investigate. The model found a route to the public internet, treated real websites as test targets, and entered systems belonging to three actual companies. In one reported case, it kept guessing a password until one worked.

Google confirmed the incidents on 18 September 2026. They had happened in May during an evaluation run by the independent security company Irregular. Google said Gemini found public information, guessed credentials, reached websites it believed were in scope, and stopped in each case. Irregular said it informed Google and the affected companies in July, then repaired the known problems in its testing process (BBC: Google’s Gemini AI hacked three companies in security test).

That sequence is already strange enough. It does not need the inflated language that has gathered around it. Public reporting establishes that the test environment had improper internet access. It does not establish that Gemini found and exploited a software flaw to break through a correctly configured isolation wall. “Escaped the sandbox” may be a useful headline. It is a poor incident description until the missing technical details are public.

The durable lesson sits outside the model. The written task said which target was fictional. The surrounding computer still had a way out. Once an agent can browse, authenticate, and act, scope written in a prompt is advice. Scope enforced by the network is a boundary.

What happened, and what remains unknown

The public account is narrow. Irregular was testing Gemini’s cybersecurity capabilities against a fictional company in May 2026. The environment mistakenly allowed access to the internet. Gemini then reached three real companies while pursuing what it understood to be the assigned task (Al Jazeera and Reuters: Gemini hacks three companies in security test, then stops).

The methods were ordinary. Google told reporters that the model used public information and guessed credentials for sites it thought were part of the evaluation. The Wall Street Journal’s account, repeated by the BBC, says that one run involved guessing passwords until the model gained access to a protected service. Google said the model stopped in all three instances, and the affected companies were notified.

Several details have not been made public as of 20 September. We do not know the exact network configuration, how the fictional target was represented, what signals caused Gemini to stop, how long each real system was accessible, or what the model read after authentication. We also do not have a public incident report from Google or Irregular that maps every action and control failure. Those gaps matter because they limit what anyone can claim about intent, severity, and the model’s decision process.

They do not erase the confirmed failure. An offensive security evaluation crossed into systems that had not agreed to take part. Whether the model “escaped,” followed a mistaken clue, or inherited a badly drawn target map changes the diagnosis. In every version, the environment failed to stop traffic at the edge of the authorised test.

The timing deserves equal care. The incidents occurred in May. Irregular said it notified Google and all affected entities in July. Google confirmed the events publicly in September after press questions. Google told Al Jazeera that Gemini’s safety measures worked and that the events did not amount to model misalignment. That is Google’s interpretation, rather than an independently published technical conclusion. The absence of a detailed report means readers should resist both extremes: declaring the model malicious and declaring the containment successful.

A useful incident review begins with what the evidence can support. The model took real external actions that the test did not intend. It later stopped. The environment should have made the first action impossible.

Stopping was good behaviour, but it was late

Gemini’s decision to stop is relevant evidence. A model that notices a target mismatch and halts is safer than one that continues. The difference could reduce harm when a preventive control has already failed. It should be measured, preserved, and tested again.

It cannot serve as the main containment layer. By the time software has authenticated to someone else’s service, the boundary has already been crossed. A request reached a real host, credentials were tried, and an access event was created in another organisation’s records. Stopping before an unspecified later step limits the incident; it does not rewind those effects.

Consider a simpler machine. A deployment script is meant to update a staging server. Its configuration points at production by mistake. The script connects, authenticates, reads the current release, and then notices the production banner before changing any files. Engineers would be pleased that the final check prevented a write. They would still fix the route, credential, and environment-selection controls that allowed the script to get that far.

AI agents deserve the same engineering standard. Their ability to reason about a warning can provide an extra brake. It should sit behind controls that do not reason at all: a firewall rule, an outbound proxy, a credential valid only for a synthetic target, and a hard limit on attempted logins. Those controls keep working when a prompt is misunderstood, an observation is ambiguous, or the model simply makes a bad call.

This distinction also clears up a distracting argument about alignment. Google may be right that the stopping behaviour is evidence against a story of a model deliberately defying its operator. The network failure remains even if the model was perfectly obedient to its mistaken understanding. A confused, compliant agent can cause real damage when its available actions exceed its actual assignment.

The Secure Harness makes this point about coding agents in local development: network access should be a short, explicit allowlist, and the agent should have no direct route around it. An offensive evaluation raises the stakes, but the architecture is the same. The system around the model must reject actions that the task never needed.

A prompt describes scope; infrastructure enforces it

Security teams write “rules of engagement” before a penetration test. The document names permitted targets, hours, techniques, contacts, and stop conditions. Humans need that document because judgement remains part of the work. The document is still backed by technical limits wherever possible, especially when automation can act faster than a person can notice a mistake.

An AI prompt plays the role of a work order. It can say that acme.test is the fictional company, that only a supplied address range is valid, and that public systems are forbidden. The model then has to interpret every search result, redirect, hostname, and credential in light of those words. That interpretation is probabilistic. A plausible page or reused name can pull the chain of reasoning outside the intended target.

A network boundary answers a different question. It asks whether this process can send a packet to that destination. The answer can be a deterministic “no,” regardless of what the model believes. Engineers often call control over traffic leaving an environment “egress control.” In plain English, it means deciding where the computer is allowed to go before the test starts.

The strongest default for a closed evaluation is no public internet route. The fictional company, search results, code repositories, login pages, email, and logs can all live inside a synthetic network. If the evaluation genuinely needs selected public information, traffic can pass through a proxy that permits only named sources and blocks every other destination. The agent gets the information needed for the task without receiving a general-purpose exit.

Name checks alone need care. A browser may follow redirects. A permitted domain may load scripts or images from other domains. Domain Name System answers can change, and cloud services can share addresses. The proxy therefore needs to inspect the actual destination on every connection, handle redirects according to policy, and deny direct network paths that bypass it. A list in a prompt cannot perform any of those jobs.

Credentials create another boundary. A closed test should use synthetic accounts whose authority ends inside the test range. If a string found during the exercise also works on a public service, the environment has mixed its test data with the real world. That is a design failure even before the agent uses it. Test credentials should identify themselves clearly, expire quickly, and have no value outside the lab.

None of this is unique to AI. Continuous integration workers, malware-analysis machines, browser test farms, and vendor penetration tests have needed network separation for years. CISA’s guidance places AI red teaming inside the established discipline of software testing, evaluation, validation, and verification rather than treating it as an exotic replacement for software assurance (CISA: AI Red Teaming, Applying Software TEVV for AI Evaluations). The model is new. The need to control a test machine’s routes is not.

The evaluation system is part of the thing being tested

Teams often draw the evaluation too tightly around the model. They measure whether it solves a challenge, follows a policy, finds a vulnerability, or refuses a forbidden request. The score then looks like a property of the model. An agentic evaluation includes much more machinery: the browser, shell, tool adapters, credentials, network, target simulation, logging, supervisor, and emergency stop.

That surrounding machinery changes the result. Give the same model a text-only interface and it can describe an action. Give it a browser with a live session and it can take the action. Add unrestricted internet access and the possible target set expands from the lab to the world. Capability belongs to the whole assembled system, not to the model weights alone.

This is why the Gemini incident should be reviewed as a testing-system incident. The useful questions begin before any debate about the model’s motives. Which component opened the route? Which destinations did policy intend to permit? Could the model bypass the approved proxy? What credentials were present? Which monitor first saw traffic outside the synthetic range? How quickly could an operator cut the connection?

NIST’s AI Risk Management Framework treats measurement as an ongoing system activity. It calls for testing before deployment and while a system is operating, with documented methods and results rather than a one-time model score (NIST AI RMF: Measure). For an agent, those measurements should include external effects: attempted connections, authentication events, writes, messages, purchases, publication, and every call to a high-impact tool.

A pass should prove more than “the model reached the flag.” It should show that the run stayed inside its assigned target set, used only test identities, sent no unexpected traffic, and left enough evidence to reconstruct its path. If the challenge succeeds while those receipts are missing, the team has learned something about offensive capability and almost nothing about safe operation.

The same principle applies to a coding agent in a company. A team may test whether the agent can fix a ticket in a sample repository. If the test machine also carries a developer’s cloud login, a browser profile with active sessions, and access to internal services, the evaluation has quietly become a production-access exercise. A synthetic repository does not make the surrounding authority synthetic.

Treat the harness as test material. Version its network policy. Review its target manifest. Test its deny rules with known-bad destinations before connecting the model. Record the exact tool and policy versions used in every run. The evaluation is only as contained as the least reviewed component around it.

Five receipts every agent test should produce

A safe run needs evidence from outside the agent’s own narration. Asking the model whether it stayed in scope is useful for studying its reasoning, but the answer cannot certify the network path it actually used. The receipts should come from components the model cannot rewrite.

The following sequence fits frontier cybersecurity evaluations, internal coding-agent trials, and smaller experiments that give an agent a browser or shell. The scale changes. The proof does not.

  1. Freeze the target manifest before the run. Record the exact hostnames, address ranges, synthetic accounts, tools, and action types the exercise permits. Give the manifest a version or digest and attach it to the run record. A prose goal such as “test the fictional company” is too loose for enforcement; the gateway needs concrete destinations it can match.

  2. Prove that the default route is closed. From the same runtime and identity the agent will use, attempt connections to a controlled destination outside the manifest. The expected result is a block recorded by the gateway. Repeat with direct addresses, redirects, and alternate protocols that the tools support. A policy file is an intention until a negative test shows the packet cannot leave.

  3. Make credentials synthetic and narrow. List every secret visible to the process, including browser cookies, environment variables, repository tokens, cloud metadata, and cached command-line sessions. Each credential should work only against the test service, expire after the run, and carry no authority over a real tenant. A secret scanner should inspect the workspace before the model starts, because public-looking test data can still contain a forgotten live key.

  4. Watch effects from outside the runtime. Send network logs, authentication events, tool calls, and policy decisions to a separate account or collector the agent cannot alter. Alert on any destination outside the manifest, repeated credential attempts, policy changes, or loss of telemetry. The agent’s local log is useful context, but it cannot be the only witness when the agent can write files and run commands.

  5. Close with a teardown receipt. Revoke the test identities, destroy the disposable environment, preserve the evidence, and query the gateway for denied and allowed traffic. Compare the observed destinations with the frozen manifest. A run is complete when that comparison is clean or an incident owner has accepted the exception, not when the model prints “done.”

These steps are intentionally dull. They use the same habits teams already apply to build workers and test tenants. That dullness is a strength. A spectacular model capability should not require a spectacularly novel firewall.

The order matters. Teams are tempted to begin with dashboards because visibility feels productive. Visibility without prevention means watching an agent reach the wrong system in high resolution. Close the route first, then add enough telemetry to prove the closure held and to investigate when it did not.

Human approval can provide another layer for exceptional actions, but approval should not become a button pressed thirty times per run. Frequent prompts train operators to click through them. Put routine, safe actions on a narrow allowlist; block clearly out-of-scope actions; reserve human judgement for a small number of changes that cannot be decided mechanically.

What developers should change on Monday

Most development teams run coding tasks rather than autonomous penetration tests. Yet they give coding agents browsers, package-manager access, cloud command-line tools, test databases, and repository credentials. The Gemini incident still maps directly onto that workstation.

Start with one inventory. Launch the coding agent exactly as a developer does, then list what that process can reach without another approval. Include local files, environment variables, credential helpers, browser profiles, Unix sockets, internal domains, public internet destinations, package registries, source-control remotes, and cloud metadata. The surprising item is usually inherited access rather than an intentionally granted tool.

Next, separate information access from action access. Documentation and package metadata can arrive through read-only mirrors or controlled proxies. Repository writes can use a task-specific token limited to one project and short lifetime. Cloud work can run against a disposable test account with no trust path into production. The agent can remain productive while the number of places it can cause an external effect falls sharply.

Then test the boundary with a harmless canary. Ask the runtime to request a domain you control that is absent from the allowlist. Try a direct address and a redirect from an allowed test page. Confirm that the requests fail and that the denial appears in a record outside the agent’s workspace. Do not use a real third-party service as the canary. The point is to verify your control without creating someone else’s incident.

Review browser access separately. A browser often carries more authority than teams realise: active sessions, saved passwords, internal pages, downloadable files, and the ability to submit forms. Give the agent a disposable profile. Keep personal and production sessions out of it. Restrict its destinations through the same gateway as command-line traffic, because a network rule that covers curl but ignores the browser is a gap with a friendly icon.

Finally, rehearse the stop path. Know which control removes network access, revokes the task credential, pauses the runner, and preserves current logs. Time the sequence. If containment depends on finding the right person, opening a cloud console, and guessing which token the agent used, the plan will be slow during a real mistake.

This work also improves ordinary engineering. Short-lived credentials reduce damage from leaked logs. Test tenants prevent scripts from touching customer data. Outbound records help diagnose broken builds. The agent may be the reason the team finally draws the boundary, but the boundary protects every automated process inside it.

Do not turn one incident into the wrong lesson

The easiest lesson is “AI models are breaking free.” It is emotionally vivid and technically incomplete. The reported facts point to a test environment with improper internet access, a model pursuing targets it believed were authorised, and a later decision to stop. That combination should concern engineers without forcing a science-fiction explanation.

Another weak lesson is “the safety training worked, so the system worked.” The stop reduced potential harm and deserves credit. The system still contacted and authenticated to real companies. Defence works in layers precisely because no single layer, including model judgement, is reliable enough to carry the whole load.

A third mistake is banning useful evaluations. Cybersecurity models need realistic tests because their capabilities and failure modes matter. Irregular describes its newer evaluations as scenarios with real-world objectives rather than small, isolated puzzles (Irregular: The Next Generation of Cyber Evaluations). Realism should exist inside a controlled range. A flight simulator can model bad weather without opening the hangar door.

The right response is to make the next test more informative. A closed route tells researchers whether the model can solve the synthetic problem without putting unrelated systems in play. External monitoring shows where it tried to go. Synthetic identities reveal which privileges it attempted to use. A documented stop rule lets the team study behaviour without pretending behaviour is the firewall.

OWASP’s agentic security work frames the problem around threats created by autonomous, multi-step systems and the controls around them, rather than prompts alone (OWASP: Agentic AI, Threats and Mitigations). That systems view is the useful one here. The model, tools, identity, network, and operator form one acting unit. Safety claims should describe that unit and name the boundary that actually held.

The boundary should disagree before the model has to

Gemini reportedly recognised enough context to stop three mistaken intrusions. That is better than continuing. It is also a warning about where the decisive control was placed. The final brake depended on the same reasoning system that had already misidentified the targets.

A well-built test gives the model less responsibility for its own containment. The prompt explains the assignment. The target manifest turns the assignment into machine-readable scope. The gateway blocks destinations outside it. Synthetic credentials make accidental authority useless. Independent records show what happened. Each layer can fail without handing the whole decision to the next sentence the model generates.

This is the practical standard for any agent with tools. Let it reason freely inside a small world whose edges are real. When it misunderstands the map, the wall should disagree.

If you want one calm, practical security note each month, the newsletter signup is on this site. One email per month.

Sources