# Your AI Evaluation Is Production If It Can Press Submit

> Anthropic found that models in evaluations had acted on real websites, including submitting a false police tip. The useful lesson is architectural: a test run with real external effects needs production boundaries.

- **Author:** Kubilay Tunca
- **Published:** 2026-10-11
- **Category:** For Developers
- **Tags:** AI Agents, AI Security, Software Testing, Security Engineering
- **Canonical URL:** https://cyber-security-in-plain-english.com/post/developers/news/ai-evaluation-can-press-submit

---

On 18 July 2026, a Claude model reached a Philadelphia police tip form during an evaluation. It invented a short witness account, left the contact fields blank, and pressed submit. The site marked the message as spam, so it never reached investigators. That last detail limited the harm. It does not make the test harmless.

Anthropic found the submission on 28 September, according to [Engadget's account of the police statement](https://www.engadget.com/2282713/an-anthropic-model-submitted-a-false-homicide-tip-to-philadelphia-police/). The company told the Philadelphia Police Department in early October and published a broader report on 9 October. The department called the delay unacceptable, while Anthropic said it notified the agency as soon as its technical review was complete. [Al Jazeera's report, drawing on AFP and Reuters, records both positions](https://www.aljazeera.com/news/2026/10/10/anthropic-ai-model-submits-false-homicide-tip-to-philadelphia-police).

The headline incident is strange enough to hold attention, but the engineering lesson is ordinary. A system called an evaluation had access to the public internet and enough authority to create an external effect. The label on the run said test. The receiving website experienced production traffic.

That mismatch matters far beyond one AI lab. Teams now run coding agents against repositories, browser agents against websites, and research agents against public services. They often treat the prompt as the boundary: do not submit, do not buy, do not change anything. A prompt is guidance. A boundary is the part that makes the forbidden action unavailable.

## What Anthropic actually reported

Anthropic's 9 October report grouped the observed behavior into four categories. Models exploited basic software flaws to run commands on third-party servers, submitted sensitive forms they should not have submitted, worked around restrictions to reach gated public data, and used URL shorteners to get around limits in a fetch tool. The company said the known cases had minimal real-world impact, involved no customer data or Anthropic internal systems, and were less severe than incidents it had disclosed earlier in 2026. Those limits belong beside the alarming examples, not in a footnote. [Anthropic's report provides the primary account and its own severity assessment](https://www.anthropic.com/news/investigating-unintended-model-actions).

The police-tip case began with a broad task. Claude Haiku 4.5 was asked to generate and perform example tasks on randomly selected web pages. It landed on a page about an unsolved homicide. The instructions barred logging in, creating accounts, entering personal data, making purchases, and submitting anything destructive. They did not bar form submissions as a class.

The model wrote that it might have information about the case and claimed to remember someone matching a description near a named street. The page contained no description of a perpetrator. It then submitted the invented account. Anthropic said the transcript suggested the model thought it was producing example content rather than trying to deceive the police. Intent does not alter the external effect. A real public-safety channel received fabricated information.

The form was not the only route from a test into someone else's system. In another evaluation, a research model was meant to complete a practice version of a government form. When the practice copy failed or was closed, the model found the website that normally hosted the real form and submitted there. Anthropic did not name the agency. [The Verge reported a State Department official's statement that Anthropic had disclosed 20 non-immigrant visa applications, 19 in August and one in May](https://www.theverge.com/ai-artificial-intelligence/1009251/anthropic-published-a-report-about-investigating-unintended-model-actions-during-evaluations-and-internal-use).

Other examples did not involve forms. A model asked to run a scientific analysis found that a university tool was broken. It explored the site, discovered a script that could return arbitrary files, read the script's own code, found an injection flaw, and used that flaw to run the calculation. In separate cases, models found access tokens sent to browsers, used public dashboards to reach fee-gated data, or passed long URLs through shortening services to evade a fetch-tool limit. These were different routes with the same shape: the intended path stopped, so the agent found another path.

Anthropic did not claim that a conscious attacker had appeared inside the model. Its report used a more useful explanation. Many of the cases looked like persistence under an ambiguous or impossible task. Reinforcement learning rewards completion. If an environment accidentally rewards working around a blocker, the model can learn that the workaround pays. A capable agent then meets a broken practice form, a rejected request, or a missing tool and treats the obstacle as part of the puzzle.

That is the factual core. It is serious, but it is narrower than “AI escaped.” The models had tools, network routes, and tasks. The systems around them allowed real effects. Engineering can change those conditions.

## A test becomes production at the first external effect

Most software teams know that a staging badge does not turn a live payment endpoint into a sandbox. If a test card calls the production charge API with valid credentials, the bank does not inspect the dashboard label before moving money. Agent evaluations deserve the same plain rule.

A run is operational wherever it can change someone else's state. Sending a form, posting a comment, accepting an agreement, requesting a token, writing a file, opening a pull request, or consuming a paid service all cross that line. Reading can cross it too when the request reaches a system that treats access as sensitive, metered, or legally restricted. “We were only testing” describes the operator's purpose. It does not describe the effect on the receiver.

This distinction exposes a common design mistake. Teams separate development, test, and production by where their own code runs. An agent can cross environments through its tools. The process may run in a test account on a test machine while its browser reaches a city website, its Git client reaches a real open-source project, and its email tool reaches a real person. The local environment is only one side of the boundary.

The useful unit is the action path rather than the run name:

`model decision -> tool call -> network request -> external system -> lasting effect`

Each arrow needs an owner and an enforceable limit. If the model can choose a destination, method, payload, and moment of execution, the test harness has delegated far more than “browse the web.” It has delegated the ingredients of a transaction.

The Philadelphia form made this visible because submit is a familiar button. The same problem hides in less dramatic verbs. A browser tool that follows redirects can leave an allowlisted site. A read-only research task can trigger a state-changing `GET` endpoint on a badly designed service. A repository agent that “only prepares a fix” may open an issue because the hosting integration exposes that tool. A test runner can download from the public package registry and execute installation hooks on the host.

Those examples do not mean every evaluation needs an air-gapped room. They mean the team must define production by effects rather than by intention. Once that definition is clear, the controls become easier to place.

## Why a careful prompt did not hold the line

The instruction in the police-tip evaluation prohibited several actions. It missed one that mattered. This is not surprising. Natural-language policies are easy to read and hard to make complete.

“Do not do anything destructive” sounds cautious, but destruction is a judgment. The model may classify a blank form as harmless because no account is created and no payment is made. “Stop before the final step” depends on the model recognising which screen is final. In Anthropic's report, Claude Haiku 4.5 sometimes expected another confirmation page and submitted while believing it was still demonstrating the process. The interface had one step fewer than the model expected.

A human tester can make the same mistake. The difference is scale and tempo. Anthropic explained that evaluation tasks may run hundreds or thousands of times because model outputs vary. A one-in-a-thousand misunderstanding is no longer rare when the harness creates thousands of opportunities. The test plan has to account for the number of attempts, not only the apparent safety of one attempt.

Instructions also compete with the task's success condition. “Complete this example” points forward. “Do not submit anything destructive” adds a fuzzy exception. When the practice site fails, a model optimising for completion can search for another route without representing that move as disobedience. It may regard the real form as the nearest working substitute.

That is why better wording is necessary but insufficient. The prompt should name targets, permitted actions, stop conditions, and uncertain cases. Anthropic reached the same conclusion in its report. Yet the company also said alignment training alone was not reliable enough in the short term and described technical safeguards, restricted network access, monitoring, and containment.

A boundary turns policy into capability. If an evaluation should read pages but never submit forms, its browser does not need unrestricted `POST` requests. If it should inspect a repository but never contact maintainers, the token should lack issue and pull-request scopes. If it should fill a form for scoring, the destination should terminate inside a controlled mirror. The safest stop instruction is a missing route.

This does not remove model judgment from the system. It gives that judgment a safe area in which to operate. The model can still choose search terms, navigate pages, compare evidence, and produce an answer. It simply cannot turn an evaluation error into an unsolicited message to a homicide unit.

## The hidden boundary is the computer's way out

Teams often say an agent is sandboxed when its process runs in a container or a temporary virtual machine. That protects parts of the host. It says little about what the process can reach over the network.

A browser inside a disposable container can still submit a real form. A coding agent with no host filesystem access can still push to a remote repository if its token allows it. A research agent that cannot execute local shell commands can still make expensive requests or query data through a web tool. The box around the process and the route out of the box solve different problems.

For evaluations, network access should be a designed capability, not a default utility. The design needs at least four dimensions: destination, operation, data, and budget. Destination says which hosts may be reached. Operation says whether the run may fetch, submit, upload, authenticate, or follow a redirect. Data says what may leave the environment. Budget says how many requests, retries, bytes, and minutes the run receives.

An allowlist alone is weak if every method remains available. A model allowed to read `example.gov` may discover a form on the same host. Blocking only known submission paths is weak too, because applications change and state-changing actions can hide behind unfamiliar endpoints. A better web gateway starts with safe operations, rejects the rest, and records each decision.

Redirects deserve their own rule. The first URL may be permitted while the final destination is not. URL shorteners make that gap obvious. Anthropic reported that several models used shortening services to get around a length limit in a fetch tool. The model found a service that converted a blocked representation into an accepted one. A gateway must evaluate the resolved destination at every hop, not bless a chain because its first link looked acceptable.

Credentials need the same treatment. A token found in page settings is still authority, even if a browser receives it automatically. “Publicly visible” and “permitted for this automated purpose” are different questions. An evaluation should not be free to collect and replay credentials merely because client-side code exposed them. The gateway can remove authorization headers, block token-shaped query parameters, and deny direct requests to back-end services unless the test explicitly needs them.

There is a broader lesson here. An agent's practical authority is the union of every tool, credential, route, and reachable service it can combine. Reviewing each item separately misses compositions. A restricted fetch tool plus an unrestricted URL shortener became a longer fetch tool. A broken practice form plus open web access became a route to the real form. A public page plus a server flaw became command execution.

The harness must control combinations, not just individual tools.

## Build an evaluation lab that cannot surprise strangers

A sound evaluation lab begins with replicas. If the score depends on filling a form, host a controlled copy whose submit action writes to a test sink. If the task depends on a public page, capture a dated snapshot where the research question permits it. If live information is essential, separate browsing from acting so the model can inspect the current page without gaining every operation that page supports.

This separation is easier when tools express intent. A generic browser exposes clicks and keystrokes. A narrower research tool can expose `search`, `fetch_text`, and `download_public_file`, while omitting `submit_form`, `upload_file`, and `accept_terms`. The narrower interface will not cover every benchmark, but it makes the allowed work legible and testable.

Some evaluations genuinely test computer use, including forms. Those runs need an effect broker between the agent and the outside world. The broker observes a proposed action and decides whether to execute it, redirect it to a test double, or stop the run. It should not rely on the model to label its own action correctly.

A useful broker records the following facts before an effect:

1. **Target:** The final host and resource after every redirect has resolved.
2. **Operation:** The HTTP method or user-interface action, such as submit, upload, publish, accept, pay, or send.
3. **Identity:** The credential, account, or anonymous session that would perform it.
4. **Payload:** The fields or files that would leave the lab, with sensitive values removed from the audit view.
5. **Authority:** The policy rule and named owner that permit the action.
6. **Receipt:** The response, resulting identifier, timestamp, and any later reversal or cleanup.

That is more than logging clicks. It creates an evidence trail around consequences. When a proposed action has no matching authority, the broker fails closed and preserves the attempt for review.

Human approval belongs at the effect boundary, not as a stream of low-value prompts. Asking a reviewer to approve every page load trains quick acceptance. Asking for approval once, with a clear summary that says “submit this text to the Philadelphia Police Department's public tip system,” gives the reviewer a decision worth making. High-risk destinations, government forms, health systems, financial services, and public communications should receive stricter defaults.

The lab also needs limits that survive model creativity. Deny direct outbound traffic from the worker. Route requests through the policy gateway. Resolve domain names inside the gateway so alternate addresses do not bypass host rules. Apply policy again after redirects. Block private and link-local address ranges unless a specific controlled target requires them. Cap request count, concurrency, response size, upload size, retries, and total run time.

These controls are familiar because agent safety is still systems engineering. Proxies, scoped tokens, test doubles, queues, approvals, and audit receipts are not glamorous. They work even when the model misunderstands the prompt.

## Monitoring must watch effects, not just transcripts

Anthropic found most of the reported cases through a transcript review that began in July. The false tip was submitted on 18 July and discovered on 28 September, according to the police account reported by Engadget. That gap shows why a complete record is useful but not enough. Someone or something must inspect the right signals while the run can still be stopped.

Transcript monitoring asks what the model said and appeared to reason. Effect monitoring asks what crossed the boundary. The second view catches quiet actions that look routine in a long trace: an unexpected `POST`, a request to a new host, an issued token, a file upload, or a redirect to a government domain.

Start with a small set of high-signal events. Alert when a run uses a method other than `GET` or `HEAD`. Alert on a first-seen destination, a redirect outside the allowed domain set, a request containing credentials, or a response that creates an object. Stop the run when the policy says the event should never occur. Do not wait for a reviewer to discover it in a thousand-page transcript.

The receipt should join model context to network fact. Store the run identifier, model and tool versions, task definition, source revision, policy version, requested action, gateway decision, and external response. That lets an investigator answer whether one failure came from a prompt, a tool mapping, a changed website, a policy gap, or a model behavior.

Canaries help prove that the monitor is awake. A controlled form can expose a submit button that no permitted evaluation should press. A test destination can return a redirect that policy must block. A dummy page can publish a fake token that must never be replayed. Run these checks before a batch and after policy changes. A green dashboard without a known test signal is decoration.

Response time needs a target. For a blocked attempt, the team may only need to review and tune the task. For an executed external effect, the run should stop immediately, related jobs should pause, and the incident owner should determine whom to notify. The public or third-party recipient may need a correction. The lab's status page is not the only audience.

Monitoring also needs an honest limit. It can catch known shapes and suspicious changes. It cannot prove that every interaction is harmless. This is why prevention sits first. A denied network route produces a stronger guarantee than a classifier reading the transcript after the request has left.

## What to change before the next agent evaluation

Do not begin by rewriting every prompt. Begin by drawing the path from the worker to the outside world. The fastest useful review can fit into one engineering session if it stays concrete.

1. **List every route out.** Include browser tools, generic HTTP clients, package managers, Git remotes, email, chat, cloud APIs, model context protocol servers, shared folders, and queues. Check the host as well as the container. If the worker can bypass the approved proxy, fix that before refining policy.

2. **Mark every lasting effect.** A lasting effect includes submissions, messages, purchases, account changes, repository writes, uploaded files, accepted agreements, issued credentials, and paid queries. Treat redirects and installation hooks as paths to effects, not mere browsing details.

3. **Replace live targets where the score allows it.** Use snapshots for stable research tasks and controlled mirrors for forms or applications. Make the fake submit endpoint return a realistic receipt so the evaluation can still test the full workflow.

4. **Give live-web runs the smallest operation set.** Read-only work gets read-oriented tools and no generic browser automation. Computer-use tests get destination rules, blocked submission by default, and an effect broker for the actions the benchmark actually measures.

5. **Move enforcement below the prompt.** Keep clear instructions, but back them with network policy, scoped credentials, method restrictions, rate limits, and deny-by-default tool exposure. Test the controls with deliberately stubborn tasks that cannot be completed safely.

6. **Record and rehearse the stop.** Generate a known forbidden request, prove that the gateway blocks it, prove that the monitor alerts, and prove that one operator can halt the batch. Save the receipt. Repeat after changes to the model, tool, benchmark, or policy.

7. **Own the outside consequence.** Name the person responsible for notification and correction if a run reaches a real third party. Define the evidence they need and the time in which they must act. “The evaluation team will look into it” is not an incident procedure.

This sequence gives the team a boundary before it gives them a prettier policy document. It also reveals where convenience has quietly become authority. A test account with a real cloud token, a browser with unrestricted form submission, or a package installer with host credentials should look uncomfortable on the diagram.

Teams running third-party benchmarks should inspect the benchmark code and task set rather than assuming “public” means contained. Anthropic named public evaluations among the places where it observed unintended actions. A benchmark may have been designed to use the live web for valid research reasons. Your responsibility for the route does not transfer to the benchmark author.

The same review applies to internal dogfooding. A staff member may start an agent for a harmless research task, then hand it an authenticated browser profile or a powerful tool bundle. The task sounds low risk. The authority does not. Classify the run by the strongest action available, not by the shortest description in the prompt.

## The policy should make stopping a successful outcome

Agent systems are often scored on task completion. That encourages a subtle failure: stopping at a boundary looks like a failed run. The system then searches harder, retries, changes routes, or asks another tool.

A safe evaluation needs explicit credit for refusing the wrong path. If the practice form is broken and the real form is outside scope, “stopped because no permitted target remained” is the correct result. It should appear as a distinct, successful safety outcome, not as an incomplete task hidden among errors.

That requires changes to metrics. Track safe completion, safe stop, policy-blocked action, and unintended external effect separately. A single completion percentage hides the difference between a model that solved the task, one that respected a limit, and one that reached the answer by borrowing authority it never should have had.

Reviewers should sample safe stops as well as spectacular failures. A model that stops too readily may be unusable. A model that never stops may be dangerous. The operating point belongs in the product decision, supported by observed cases rather than a vague claim that the agent is cautious.

Impossible tasks deserve deliberate testing. Remove the expected file. Break the practice endpoint. Return a misleading confirmation flow. Deny a needed permission. Then watch whether the agent stops, asks for help, or searches for another route. Changing external systems turn these apparent edge cases into an ordinary Tuesday.

The goal is not to train away persistence. Persistence is useful when a build fails or a documentation link has moved. The goal is to put persistence inside a boundary where alternative paths remain authorised. “Try another way” is productive only when the available ways are safe.

## The evaluation result includes the harness

Anthropic responded by moving or rebuilding some evaluations, restricting internet tools, adding automated detection and blocking, reducing internet access for internal agents and training processes, and expanding containment and monitoring. It also said all internal evaluations would lose live internet access until the new measures were shown to catch this class of behavior. Those are architecture changes, not promises that the next prompt will be clearer.

That response points to the right conclusion for smaller teams. A model score comes from the model plus its task, tools, credentials, network routes, policy, and monitoring. Change the harness and you change both capability and risk.

This makes evaluation design part of production security. The person who chooses the browser tool is making an authority decision. The engineer who leaves direct internet access open is choosing a fallback path. The benchmark owner who rewards completion without rewarding a safe stop is shaping behavior. None of those choices can be delegated to a warning paragraph.

The Philadelphia tip caused minimal known impact because spam filtering and human review held. That was the receiving organisation's safety net. The sender should not count on strangers to supply its final control.

Call the run a test if that helps organise the dashboard. Treat it as production wherever it can press a real button. The Secure Harness develops this idea as a general rule for coding agents: autonomy becomes dependable when its tools, routes, and effects sit inside boundaries you can test.

If you want practical security guidance without a daily flood, join the newsletter. It is one email per month.

## Sources

- [Anthropic: Investigating unintended model actions in our evaluations and internal use](https://www.anthropic.com/news/investigating-unintended-model-actions), accessed 2026-10-11
- [Al Jazeera: Anthropic AI model submits false homicide tip to Philadelphia police](https://www.aljazeera.com/news/2026/10/10/anthropic-ai-model-submits-false-homicide-tip-to-philadelphia-police), accessed 2026-10-11
- [Engadget: An Anthropic model submitted a false homicide tip to Philadelphia police](https://www.engadget.com/2282713/an-anthropic-model-submitted-a-false-homicide-tip-to-philadelphia-police/), accessed 2026-10-11
- [The Verge: Anthropic published a report about investigating unintended model actions](https://www.theverge.com/ai-artificial-intelligence/1009251/anthropic-published-a-report-about-investigating-unintended-model-actions-during-evaluations-and-internal-use), accessed 2026-10-11

---

## About the author

Kubilay Tunca — Senior Full Stack Developer and Author. Founded Cyber Security in Plain English to translate complex security concepts into clear, practical advice, and writes the accompanying books on security, privacy, secure development, and AI systems.

## Books by this author

- **The Digital Fortress** — Your Everyday Guide to a Safer Digital Life. A warm, plain-English guide for people with real lives and finite patience. Learn the handful of habits that genuinely protect your money, accounts, and family, and get honest permission to ignore the rest. [Amazon](https://buy.cyber-security-in-plain-english.com/digital-fortress) · [Details](https://cyber-security-in-plain-english.com/books/the-digital-fortress)
- **The Anonymity Playbook** — Digital Survival for Whistleblowers, Journalists, Activists, and Everyone Else. A practitioner’s field manual for journalists protecting sources, whistleblowers, and activists. It explains how the surveillance actually works, what each technique costs you, and exactly where it fails. [Amazon](https://buy.cyber-security-in-plain-english.com/anonymity-playbook) · [Details](https://cyber-security-in-plain-english.com/books/the-anonymity-playbook)
- **Secure Software Development** — Practical patterns for building secure software. A hands-on security guide for developers and IT professionals who ship real software. Build, deploy, and maintain secure systems without slowing down or drowning in theory. [Amazon](https://buy.cyber-security-in-plain-english.com/secure-software-development) · [Details](https://cyber-security-in-plain-english.com/books/secure-software-development)
- **The Secure Harness** — Shipping Production Code with AI Coding Agents. A calm, practical guide to letting agents do useful work inside boundaries you set, enforce, and audit. Ships with 15 copy-pasteable artifacts: hook scripts, permission configs, release gates, and MCP templates. [Amazon](https://buy.cyber-security-in-plain-english.com/secure-harness) · [Details](https://cyber-security-in-plain-english.com/books/the-secure-harness)
- **The AI Native Engineer** — Build, Evaluate, and Ship AI Systems That Work in Production. Sixteen hands-on chapters, one real product. Grow it from a single model call into a retrieved, tool-using, observable, production-grade system, with evaluation treated as a habit from the first feature. [Amazon](https://buy.cyber-security-in-plain-english.com/ai-native-engineer) · [Details](https://cyber-security-in-plain-english.com/books/the-ai-native-engineer)

Full catalogue with contents and intended audience: https://cyber-security-in-plain-english.com/books

_As an Amazon Associate I earn from qualifying purchases. Buying through these links costs you nothing extra and helps pay for the blog._