23,019 AI Bug Reports Need a Measurement Pipeline

Claude Mythos produced 23,019 candidate vulnerabilities in an early open-source scanning run. That number is big enough to win a launch slide and ruin a security team’s quarter.

The more useful number arrived later. In an analysis reported on 4 September 2026, independent reviewers had checked 1,900 candidates from that May snapshot. They confirmed 1,726 as real vulnerabilities, but human scoring also changed the severity on 14 of 27 findings that had received public CVE records. Thirteen had been rated too high by the model. Only one of the eight findings that Mythos called Critical remained Critical after review (Help Net Security).

That result supports two conclusions at once. The model found a substantial amount of real security work. Its first severity label was still a poor substitute for a decision. Teams adopting AI vulnerability discovery need to preserve both facts, because either one on its own leads to bad engineering.

The right product measures more than scanner output. A measurement pipeline turns uncertain candidates into reproducible evidence, routes them with deployment context, and learns from every human decision. Discovery is one stage. If the stages after it cannot keep up, the impressive number becomes queue length.

What the September scrutiny actually showed

The story began before this week’s coverage. In a 22 May 2026 update, Anthropic said Claude Mythos Preview had scanned more than 1,000 open-source projects and produced 23,019 candidates. The model initially rated 6,202 of them High or Critical. At that point, independent security firms had carefully assessed 1,752 from that higher-severity slice and confirmed 1,587 as real, a 90.6% true-positive rate within the selected group (Anthropic).

That last phrase matters: within the selected group. A triage team rarely samples a queue at random. It pulls forward the reports that look clear, severe, reproducible, or relevant to a project that can respond. A high confirmation rate among those candidates tells you that the front of the queue contains useful work. It does not establish the accuracy of everything behind them.

The September reporting examined the gap using counts from the earlier snapshot. Of 23,019 candidates, 21,119 had received no review outside Anthropic. Maintainers had received 1,596 reports, acknowledged 1,451, landed 97 fixes, and published 88 security advisories. Those figures describe different gates, not a single accuracy score. A candidate can be real yet duplicated, outside a project’s threat model, unreachable in normal builds, already fixed, or too costly to address immediately.

Severity exposed the missing context. Help Net Security described a Temporal Server finding that Mythos rated Critical. Maintainers scored it 2.3, Low, because exploitation required an attacker-controlled namespace that already possessed a privileged internal credential, and the reachable impact was narrower than the source-level report implied. A MinIO finding moved from the model’s Critical rating to High during outside review and then Medium with maintainer context because the path required an existing cluster root token and allowed read access only (Help Net Security).

The code contained a flaw in each case. The operational question was how much authority an attacker needed before reaching it and what changed after success. Source code can reveal the dangerous operation. It often cannot reveal whether production exposes that operation, which identity can call it, what data sits behind it, or which compensating controls block the route.

Anthropic’s live disclosure dashboard makes the pipeline easier to see. As of 26 August 2026, a newer and broader snapshot covering Mythos Preview and other Claude models listed 26,153 candidates. External firms had reviewed 5,008, confirmed 4,576 as valid, and produced a 91.4% true-positive rate in that reviewed set. Anthropic reported 2,300 disclosed findings, 1,815 maintainer acknowledgements, 421 upstream patches, and 462 CVE or GitHub advisory identifiers (Anthropic disclosure dashboard).

Do not subtract the May numbers from the August numbers as if they were two identical benchmark runs. Anthropic says the newer dashboard covers additional Claude models and changed scope. The dates, populations, and workflow states differ. What survives the snapshot change is the shape: discovery creates candidates much faster than people can reproduce, assess, disclose, fix, release, and deploy them.

That shape is the engineering story. AI reduced the cost of opening tickets. It did not reduce every cost between a suspicious code path and a safe production fleet.

A true positive can still be the wrong next job

Security scanners have always produced more work than the first number suggests. AI raises the volume and improves the quality of some findings, but it does not repeal the distinction between existence, severity, and priority.

Existence asks whether the reported behavior can happen under some defined conditions. A reviewer can often answer it by building the affected revision, supplying a controlled input, and observing a crash, policy bypass, or unexpected data flow. A strong code model can save hours here. It can trace unfamiliar code, propose a reproducer, and explain why a guard does not cover one path.

Severity asks what an attacker gains and what preconditions the attack requires. The Common Vulnerability Scoring System separates base characteristics from threat and environmental context. Its environmental metrics exist specifically to refine severity for a particular computing environment (FIRST CVSS v4.0 specification). A model reading a public repository does not automatically know that environment.

Priority asks what your team should work on next. That decision combines the vulnerability with facts outside the report: whether you run the affected feature, whether an attacker can reach it, whether exploit activity exists, whether the asset signs releases, whether an upgrade breaks a customer workflow, and whether another control already interrupts the path. Two companies can receive the same valid finding and make different sound decisions.

Consider a parser bug in a command-line utility. The code defect may be undeniable. At Company A, the utility processes files uploaded by anonymous users inside a privileged conversion service. At Company B, it appears only in a developer image and reads files created by the same developer. The same source finding enters two different systems. Company A may need an emergency patch; Company B may schedule the update with its next image rebuild.

Now add a model-generated Critical label. If both companies sort their queues by that label alone, they have replaced judgement with a string. Company A might still act correctly by accident. Company B may interrupt planned work while a reachable High finding on an internet-facing service waits below it.

This is why the September severity mismatch deserves more attention than the headline candidate count. A finder can be excellent at locating code patterns and weak at estimating the surrounding system. Those capabilities should be measured separately. Combining them into one “AI scanner accuracy” percentage hides the part that decides who gets paged.

The mismatch also creates a social cost. Maintainers already receive low-quality automated reports, often with confident prose and little evidence. Anthropic said in May 2026 that several maintainers were severely capacity constrained and some had asked it to slow disclosure while they designed patches (Anthropic). A valid but badly scoped report can still waste scarce review time. A duplicate with a dramatic rating can be worse than silence.

The scanner therefore owes the recipient more than a plausible explanation. It owes them a buildable case, a bounded claim, and enough provenance to decide whether the issue belongs in their queue.

The metric that flatters the model

A single true-positive rate is tempting because everyone understands it. Confirmed findings divided by reviewed findings yields a percentage that fits neatly in a dashboard. The denominator decides what that percentage means.

Suppose a system emits 10,000 candidates. An internal ranker puts 500 clear memory-safety crashes at the top, and reviewers inspect those first. If 450 are real, the reviewed slice has a 90% true-positive rate. The result says the ranker found a productive front of the queue. It says nothing reliable about the remaining 9,500 until you sample or review them.

Selection is not a minor statistical footnote here. Security teams intentionally select. They prioritize reports with convincing traces, likely impact, responsive maintainers, important dependencies, and code they can reproduce. That policy is reasonable for remediation, but it makes the reviewed set a biased evaluation set.

The fix is not to abandon the operational queue in favour of random review. Run two lanes. The first lane serves the business and reviews the highest expected-value findings. The second takes a small, stratified sample across scores, repositories, languages, bug classes, and model confidence bands. The operational lane fixes important software; the measurement lane tells you what the scanner is doing.

A useful evaluation report should publish both. For the operational lane, show confirmed findings, time to reproduce, accepted reports, fixes, and deployed patches. For the measurement lane, estimate precision by score band and bug class with uncertainty intervals. If the low-confidence tail is mostly noise, you can stop sending it. If one language family performs badly, you can route it through a stricter verifier.

Recall, the share of real flaws the system finds, is harder because the complete set of flaws in a real codebase is unknown. Do not fake it. Measure recall on seeded test suites, historical vulnerabilities hidden from the model’s context, or code snapshots with known fixes removed. Keep that benchmark separate from live discovery counts.

Then measure ranking. If engineers can review only 50 reports this week, how many confirmed, reachable, material findings appear in the first 50? Precision at 50 is closer to the team’s actual constraint than precision across every candidate. Normalized discounted cumulative gain, a ranking metric that rewards putting more relevant results earlier, can help when findings carry graded value. A simpler weighted score is fine if everyone can inspect how it is calculated.

Calibration matters too. Among findings assigned 80% confidence, roughly eight in ten should survive the defined review gate over time. If only half do, the confidence number is decoration. Plot predicted confidence against observed outcomes, split by bug class, and update the model or threshold when the curve drifts.

Severity needs its own scorecard. Compare the model’s proposed vector and rating with the external reviewer and maintainer decisions. Record why they differ. Missing authentication precondition, unreachable feature, reduced impact, unusual deployment, and existing mitigation are useful labels. “Human disagreed” teaches the system almost nothing.

One number can still lead the dashboard, but choose it honestly. I would lead with accepted, reproducible findings per reviewer-day. That metric makes the model share the page with the human bottleneck it creates.

Build evidence before you build a queue

A candidate report should arrive as a small research package rather than a paragraph of model confidence. The package must let another person reproduce the claim without trusting the model’s narration.

Start with immutable provenance. Record the repository, exact commit, build configuration, dependency lockfile, model version, prompt or agent policy version, tool image, and scan time. A report against “latest main” decays as soon as the next commit lands. A report against commit abc123 can be replayed.

Keep the original candidate separate from later edits. Reviewers will improve the explanation, narrow the claim, and sometimes change the reproduction. Those are good changes, but overwriting the model output destroys the evidence needed to evaluate the system. Store an append-only event trail or versioned artefacts.

The reproducer should be minimal and safe. It needs the shortest input or test that reaches the behaviour, the expected safe result, the observed result, and the commands needed to build and run it in an isolated environment. For memory faults, include sanitizer output and the relevant stack. For access-control flaws, define the starting identity and prove the unexpected operation without touching real data.

A report also needs a reachability statement. Name the public entry point, configuration flag, build option, or call chain that connects normal use to the flawed code. If reachability has not been established, say so. “Potential issue in parser” is a legitimate candidate state. It is not yet a production-impact claim.

The final piece is the smallest defensible impact. Avoid jumping from a write primitive to “full system compromise” unless the chain has been demonstrated under the target’s actual mitigations. State what the reproducer proves, then list stronger consequences as hypotheses that need separate tests. This discipline makes reports less exciting and far easier to trust.

An evidence package might carry these fields:

1. Candidate identity: Give the finding a stable ID tied to the repository and commit so duplicates can converge on one record.
2. Provenance: Store model, policy, tools, build image, dependency state, and timestamps as machine-readable metadata.
3. Reproduction: Include a safe test, exact commands, expected result, observed result, and captured logs.
4. Reachability: Name the required feature, caller, privilege, and deployment assumption, with unknowns left explicit.
5. Impact: Separate demonstrated effect from possible escalation and record the evidence for each statement.
6. Ownership: Identify the person or team responsible for the next decision and set a review deadline based on evidence, not adjectives.

That looks heavier than dropping Markdown into an issue tracker. It is cheaper than asking five maintainers to reverse-engineer what the model meant. The package turns model output into an inspectable unit of work.

The package also supports automation after discovery. A verifier can rebuild the commit, run the reproducer in a disposable worker, check that the expected failure appears, and attach the result. A deduplication stage can compare call paths and patches. A policy stage can suppress findings from disabled features without deleting them from the research record.

Every automated stage must preserve failure as a state. A timed-out build does not mean false positive. A blocked reproducer does not mean safe. “Unverified because the test environment failed” should remain visible and should not silently improve the scanner’s accuracy metric.

Give deployment context a formal place

The model’s source report and your production context are different data sets. Teams often join them in an analyst’s head, then lose the reasoning when the ticket closes. Put the join in the system.

An asset record should answer a few concrete questions. Do we run the affected version? Is the vulnerable component loaded? Which entry points can reach it? What identity does the process hold? Which data, credentials, or release actions sit behind it? What controls interrupt the path? Who owns the service, and how quickly can they test a change?

Freshness matters. A service catalogue last updated nine months ago can make a precise vulnerability report useless. Pull version evidence from running workloads, signed build manifests, endpoint inventory, or package attestations. When evidence conflicts, show the conflict. Do not let the newest API response overwrite a deploy manifest without explanation.

Represent preconditions explicitly. A report that requires an authenticated tenant administrator belongs in a different lane from a report reachable by an unauthenticated internet request. A flaw that needs a feature your build excludes should remain recorded but should not page anyone. A flaw in a release-signing service deserves attention even when the initial entry point looks narrow because the service’s downstream authority is unusually large.

CVSS can communicate vulnerability characteristics, but it cannot know your Tuesday. The official version 4.0 specification includes Threat and Environmental metric groups because base severity alone is incomplete (FIRST). Use the vector as structured input. Do not turn the final decimal into an unquestionable work order.

A practical priority function can remain simple:

priority = evidence_quality × reachability × likely_impact × asset_authority × threat_signal

Each factor should have a written rubric and an “unknown” value. Unknown must not collapse to zero. For an internet-facing package with incomplete inventory, uncertainty may increase the need for an owner to investigate. For an unreachable development tool, the same uncertainty may justify a normal review rather than an incident page.

Keep effort out of the risk calculation, then use it when planning. A one-line dependency update that removes a credible Medium issue may ship before a major migration for a High issue protected by two controls. That scheduling choice does not change either flaw’s risk. It acknowledges engineering reality without corrupting the measurement.

The owner should see why the item sits where it does. “Priority 87” invites argument about arithmetic. “Internet reachable, no authentication, code execution reproduced, service holds a production deploy token” tells the team what to verify and what can change the decision.

This context layer is also where the system learns from false urgency. When a maintainer downgrades a report because the model assumed an impossible caller, add that precondition pattern to future checks. When a supposedly unreachable library appears in a real request trace, feed that evidence back too. The goal is not a model that never changes its mind. The goal is a pipeline that records why it changed.

The human bottleneck is part of the architecture

Anthropic described human triage as the rate-limiting step in May and again on its August dashboard. This capacity constraint belongs in the product design rather than in a footnote.

Every candidate consumes attention before it consumes patching time. Someone has to identify duplicates, establish reachability, inspect the reproducer, interpret the impact, contact a maintainer, answer questions, review a fix, and decide when disclosure is safe. Volunteer maintainers carry much of that work for software used by profitable companies.

An AI scanner that increases finding volume without funding review transfers its cost to the recipient. The sender enjoys the discovery metric; the maintainer inherits the queue. This is a bad default, even when the findings are mostly real.

Set a disclosure budget. A team that can support 40 active maintainer conversations should not open 400 and hope. Rank the validated packages, disclose within the capacity to respond well, and hold the rest securely. If a candidate suggests immediate danger, escalate that case through a separate path rather than declaring the whole backlog urgent.

Bundle carefully. Five reports in the same parser may share one root cause and one patch. Sending five issue threads can hide that. Conversely, grouping unrelated bugs into a 70-page report makes each one harder to assign. The human reviewer should decide the unit of disclosure after deduplication and root-cause analysis.

Pay for external triage when the scanner is part of a commercial offering. The model bill is only one line item. Reviewer time, isolated compute, maintainer coordination, patch validation, and follow-up are part of cost per accepted finding. Excluding them produces a demo price rather than an operating cost.

Protect the reviewers as well. Reproduction workers should have no production credentials, limited outbound access, disposable storage, and strict resource caps. Candidate code and proof inputs are untrusted. A security scanner that executes generated reproducers on a shared CI runner creates a fresh route into the organisation.

Track reviewer disagreement without treating it as failure. A security firm and a maintainer may apply different threat models. Preserve both assessments, the assumptions behind them, and the final action. That record will improve routing far more than forcing one label into a canonical column.

Measure time at every handoff. Candidate to first triage, triage to reproduction, reproduction to maintainer report, acknowledgement to patch, patch to release, and release to observed deployment are separate clocks. The longest one tells you where another model call will help and where it will merely add inventory.

If candidate generation takes minutes while maintainer acknowledgement takes weeks, increasing scan parallelism inflates the queue.

A production pipeline that can say no

A useful implementation has gates. Each gate answers one question and can stop the candidate from moving forward. Passing a previous gate does not pre-approve the next one.

The intake gate checks provenance and scope. Was the code authorised for scanning? Is the commit immutable? Does the report contain the files and tool versions needed for replay? Candidates that fail stay inside the research system until repaired.

The reproduction gate runs the proposed test in an isolated worker. It captures the build, input, logs, resource use, and exit state. A reviewer should be able to rerun the same artefact. Flaky reproduction lowers evidence quality and blocks external disclosure unless a human documents why the issue remains credible.

The reachability gate joins the code path with package and runtime evidence. For your own systems, traces and deployment manifests can answer this. For open source, the report should state which common configuration exposes the path and leave project-specific deployment claims to maintainers.

The impact gate separates what happened from what might follow. It assigns a proposed CVSS vector, records attack preconditions, and marks every unknown. Human review is mandatory before a Critical label leaves the organisation. The September comparison makes that a reasonable minimum, not bureaucratic caution.

The novelty gate checks existing issues, advisories, commits, forks, and earlier internal candidates. Semantic similarity helps, but source location and root cause still matter. A confident duplicate can waste the same maintainer time as a false positive.

The disclosure gate checks recipient capacity, contact policy, embargo timing, report quality, and an internal owner. It should refuse to send if nobody is available to answer the maintainer. Scheduled disclosure without support is message delivery, not coordinated vulnerability disclosure.

The patch gate verifies that the proposed change removes the reproducer without breaking the affected feature. Run the regression test against the vulnerable revision and the candidate fix. If the test fails on both, it never proved the patch. If it passes on both, it never proved the vulnerability.

The deployment gate closes the loop. A merged patch is progress, but users remain exposed until fixed releases exist and affected installations adopt them. Anthropic’s August dashboard counted 421 upstream patches among 2,300 disclosed findings, and it explicitly warned that an upstream patch does not guarantee wide installation (Anthropic disclosure dashboard).

Every gate emits structured events. That gives you a funnel with defensible denominators: candidates received, reproducible candidates, reachable findings, accepted reports, patched findings, released fixes, and observed deployments. It also exposes where candidates die and why.

The pipeline must be allowed to reject volume. When review capacity is full, pause lower-value scans or store candidates without promoting them. A system that cannot stop discovery when downstream stages saturate has no backpressure. It is an alert generator attached to a growing database.

What to build this month

Most teams do not need a frontier cyber model to benefit from this design. The same controls improve a code-review agent, a static-analysis assistant, or a service that ranks dependency findings. Start with one repository and one accountable security engineer.

Do the work in sequence. Each step creates evidence for the next, and stopping early still leaves the system safer than a direct model-to-ticket integration.

1. Choose one bounded target and one decision. Scan a repository whose owner has agreed to participate. Define whether the output will open an internal ticket, propose a test, or prepare a draft report. Do not begin with automatic external disclosure.

2. Freeze the experiment. Pin the repository commit, dependency state, model version, policy prompt, tool container, and scan budget. Store them beside every candidate. If you cannot replay the run, you cannot compare the next one.

3. Define the evidence contract. Require a safe reproducer, observed result, source location, reachability statement, demonstrated impact, unknowns, and reviewer owner. Reject incomplete candidates before they touch the issue tracker.

4. Create a two-lane review sample. Route the best-ranked candidates into operational triage. Randomly sample a smaller set across confidence bands and bug classes for measurement. Publish precision for both lanes rather than blending them.

5. Separate finding, severity, and priority. Let the model propose each value, but store them in different fields. Require a person to approve external severity and join priority with current asset evidence.

6. Set hard queue limits. Cap open reproductions, active disclosures, and unanswered maintainer conversations. When a cap is reached, the scanner pauses or lowers its run rate. Backpressure is a feature.

7. Verify fixes twice. Run the regression against the vulnerable revision and the patched revision in clean workers. Record the expected change and keep the test with the code so the flaw cannot quietly return.

8. Review the funnel every week. Inspect confirmation by confidence band, severity disagreement, duplicate rate, reviewer minutes, maintainer acceptance, patch latency, and deployment evidence. Remove metrics that do not change a decision.

A four-week pilot should be able to answer practical questions. How many candidates can one reviewer process? Which bug classes reproduce cleanly? Where does the model overrate impact? What percentage of the top 20 findings become accepted work? How much does each accepted finding cost after human time?

If the pilot cannot answer those questions, buying more scans will not fix it. Add instrumentation before volume.

This is also the right moment to decide what never becomes automatic. I would keep public disclosure, final severity, production exploitation tests, and release approval behind named human owners. The model can prepare the work. Authority stays with people who can see the affected system and carry the consequences.

Discovery is cheap; decisions are the product

Claude Mythos appears to be finding real vulnerabilities at a useful rate. The public record contains accepted findings and shipped patches, not just benchmark claims. Dismissing the system because reviewers changed severity would ignore substantial evidence.

Treating 23,019 candidates as 23,019 actionable bugs would ignore the same evidence. Most of that early queue had not received outside review, and the reviewed public slice showed how much deployment context can move a rating. The model was often good at pointing to flawed code. The surrounding decision still belonged to a pipeline.

The mature response is pleasantly unglamorous. Preserve provenance. Reproduce in isolation. Measure a representative sample. Separate code truth from production context. Limit the queue to the review capacity behind it. Follow fixes through release and deployment.

A scanner produces findings. An engineering system produces justified actions, including the decision to wait. Build the second one.

For one practical security and AI engineering note each month, join the newsletter. One email per month, and the signup is on this site.

Sources

• Anthropic: Project Glasswing, an initial update, accessed 2026-09-05
• Anthropic: Coordinated vulnerability disclosure dashboard, accessed 2026-09-05
• Help Net Security: Most of the bugs Claude Mythos found have never been checked by a human, accessed 2026-09-05
• Forbes: Why Mythos Has a Vulnerability Assessment Problem, accessed 2026-09-05
• FIRST: CVSS v4.0 Specification Document, accessed 2026-09-05