Published
- 39 min read
LLMs Found 23 of 26 Known CVEs. What the Benchmark Does and Does Not Tell You.
Stay Safe Online Without Making It Your Second Job
The Digital Fortress (Second Edition)
A warm, plain-English guide for people with real lives and finite patience. Learn the handful of habits that genuinely protect your money, accounts, and family, and get honest permission to ignore the rest.
For People Who Cannot Afford to Get Privacy Wrong
The Anonymity Playbook (Second Edition)
A practitioner’s field manual for journalists protecting sources, whistleblowers, and activists. It explains how the surveillance actually works, what each technique costs you, and exactly where it fails.
Write, Ship, and Maintain Code Without Shipping Vulnerabilities
Secure Software Development
A hands-on security guide for developers and IT professionals who ship real software. Build, deploy, and maintain secure systems without slowing down or drowning in theory.
Use AI Coding Agents Without Losing Control of Your Codebase
The Secure Harness
A calm, practical guide to letting agents do useful work inside boundaries you set, enforce, and audit. Ships with 15 copy-pasteable artifacts: hook scripts, permission configs, release gates, and MCP templates.
Stop Shipping Demos. Start Shipping Systems.
The AI Native Engineer
Sixteen hands-on chapters, one real product. Grow it from a single model call into a retrieved, tool-using, observable, production-grade system, with evaluation treated as a habit from the first feature.
The benchmark, and its ceiling
A study circulated in late July 2026 tested thirteen models against twenty-six known vulnerabilities drawn from GitHub advisories. GPT-5.6 and Kimi K3 both rediscovered twenty-three of twenty-six, an 88.5% recall. Grok-4.5 landed at 77%. Claude Opus variants came in between 58% and 69%.
The finding that got less attention is more useful than the leaderboard. Running a cheaper model several times outperformed running an expensive model once, at comparable cost. Vulnerability discovery behaves like a search problem, and sampling the search space repeatedly beats sampling it once with a better sampler.
Before anyone reorganises a security programme around those numbers, the benchmark’s shape needs stating plainly. These were known vulnerabilities from public advisories, in code the models very likely encountered during training, evaluated in a setting where the tester knew a bug existed and roughly where. Every one of those conditions is absent when you point a model at your own codebase on a Tuesday morning.
Other evidence worth putting next to it
One benchmark makes a thin basis for a decision, and the past two years produced several other results that point in useful directions.
Google’s OSS-Fuzz team used models to write and improve fuzz targets rather than to judge code directly, and reported twenty-six vulnerabilities to maintainers on the back of it. One was CVE-2024-9143 in OpenSSL, a defect that had survived roughly two decades of human review and conventional fuzzing. The model contributed coverage; the fuzzer and its sanitisers decided what counted as a bug.
Big Sleep took the agentic route and found a stack buffer underflow in SQLite, tracked as CVE-2025-6965, which Google described as the first public case of an agent finding a previously unknown exploitable memory-safety issue in widely used software. By August 2025 the team had reported a further twenty issues across projects including FFmpeg and ImageMagick. Same shape: the agent proposed, a crash confirmed.
DARPA’s AI Cyber Challenge gave the cleanest measurement of the three, because the ground truth was known in advance and the systems ran without human help. In the 2025 finals, competing systems found eighty-six percent of the synthetic vulnerabilities, fifty-four of sixty-three, against thirty-seven percent at the semifinals, and produced correct patches for sixty-eight percent of what they found, up from twenty-five percent. They also turned up eighteen real vulnerabilities in the open-source projects used as targets, which went into responsible disclosure. Average time per successful task was about forty-five minutes at roughly $152 of compute. Trail of Bits’ Buttercup, which took second place and is open source, reported twenty-eight vulnerabilities across twenty CWEs at around ninety percent accuracy.
Notice what those three share and the July leaderboard lacks. Every finding passed through something that could not be talked into agreeing: a sanitiser, a crash, an input that demonstrably reaches the defect. The distance between eighty-eight percent recall on a text benchmark and ninety percent accuracy at AIxCC is largely that adjudication step, and the choice of whether to build one is the single decision that most affects what your own deployment produces.
Rediscovery is not discovery
The distance between “found the CVE in a repository where a CVE exists” and “found the bug nobody knows about yet” is where most of the difficulty lives.
Rediscovery benefits from an enormous prior. The model has probably seen the advisory text, the patch commit, and blog posts analysing it. Asking it to identify the flaw is closer to a recall task than an analysis task, and recall is what these systems are best at. There is no reliable way to strip that prior from a public benchmark, which is why benchmark results in this category should be read as an upper bound.
The base rate problem is the other half. In a benchmark run, the prior probability of a vulnerability in the file under examination is effectively one. In your repository, the overwhelming majority of files contain no exploitable vulnerability at all. A detector with excellent recall and a modest false positive rate produces an unusable stream of findings once the base rate drops, which is the same arithmetic that makes medical screening tests behave badly on healthy populations.
Work the numbers on a realistic repository. Ten thousand functions, of which perhaps five contain a genuine exploitable flaw. A model with 88% recall and a 5% false positive rate finds four of the five real bugs and reports roughly five hundred false ones. Triage cost swamps the value, and the team stops reading the output by week three.
Doing the base-rate arithmetic for your own repository
The paragraph above used round numbers to make a point quickly. Your numbers are different, and the calculation is short enough to run properly before you spend anyone’s week on triage.
Four quantities decide the outcome. N is the number of units you analyse, whether those are functions, files, or changed hunks. p is the fraction of units containing a real exploitable defect. r is recall on the classes you care about, and f is the false positive rate per clean unit. True positives come out as N * p * r, false positives as N * (1 - p) * f, and precision is the first divided by the sum of both.
Take a service with ten thousand functions and five genuine vulnerabilities sitting in it, so p is 0.0005. Hold recall at the benchmark’s 88% and vary only the false positive rate:
| False positive rate | Real findings | False findings | Precision |
|---|---|---|---|
| 5% | 4.4 | 500 | 0.9% |
| 1% | 4.4 | 100 | 4.2% |
| 0.1% | 4.4 | 10 | 30.6% |
| 0.01% | 4.4 | 1 | 81.5% |
Precision swings by a factor of ninety down that column while recall never moves, which is why the headline number on the leaderboard is the least interesting thing about it. Engineering effort belongs at f. A detector that gives up ten points of recall to cut its false positive rate by an order of magnitude is a straight win at these prevalences, and any vendor comparison that reports only recall has told you nothing about the variable that matters.
Diff scoping gets sold as a precision fix, which the arithmetic does not support. Scoping to changed code leaves f alone and lowers N, so precision improves only if freshly written code carries defects at a higher rate than the average line in your repository. It usually does, by a factor of two or three, not a hundred. What scoping reliably fixes is the volume per review event. Forty changed functions at f = 1% produce 0.4 spurious comments per pull request, roughly one every second or third PR, which a team will tolerate. The same detector across the full repository drops a hundred of them on somebody’s desk at once.
Consensus attacks f directly. If three runs were fully independent, a clean unit would need two bad draws to survive a 2-of-3 vote: 3 * f^2 * (1 - f) + f^3, which at f = 5% works out to 0.72%, about a sevenfold cut. Independence flatters the method, since runs share a model and a prompt and correlate their mistakes, so plan for two to four times rather than seven. Recall behaves differently. Bugs divide into ones the model can see, where it flags them on most runs and voting keeps them, and ones it cannot see under any sampling, where voting changes nothing either way. That split is why majority voting costs less recall than the per-run numbers suggest.
from math import comb
def triage_forecast(units, prevalence, recall, fp_rate, runs=1, votes=1):
"""Expected findings per scan, and what a human actually has to read.
Vote thresholds use the independent-draw approximation, which flatters
consensus. Replace fp_rate with a number you measured on your own code
before anyone quotes this in a planning document.
"""
def at_least(k, n, prob):
return sum(comb(n, i) * prob**i * (1 - prob) ** (n - i)
for i in range(k, n + 1))
eff_recall = at_least(votes, runs, recall)
eff_fp = at_least(votes, runs, fp_rate)
true_pos = units * prevalence * eff_recall
false_pos = units * (1 - prevalence) * eff_fp
total = true_pos + false_pos
return {
"true_positives": round(true_pos, 2),
"false_positives": round(false_pos, 1),
"precision": round(true_pos / total, 4) if total else 0.0,
"items_to_read": round(total),
}
if __name__ == "__main__":
print(triage_forecast(10_000, 0.0005, 0.88, 0.05))
print(triage_forecast(10_000, 0.0005, 0.88, 0.05, runs=3, votes=2))
print(triage_forecast(900, 0.005, 0.88, 0.05, runs=3, votes=2))
The third line of that output is the interesting one. It models a pre-filtered candidate set: nine hundred functions instead of ten thousand, with prevalence ten times higher because the filter selected for danger. Same model, same false positive rate, and precision lands somewhere a human can work with.
Why several cheap runs beat one expensive run
The multi-run finding points at how to fix that arithmetic, so it is worth understanding the mechanism rather than treating it as a pricing curiosity.
Model outputs vary between samples. Run the same analysis five times and you get five overlapping but non-identical sets of findings. Real vulnerabilities, which have actual evidence in the code supporting them, tend to appear across most runs. False positives, which arise from a particular chain of reasoning that happened to go wrong, tend to appear once and vanish.
That difference is directly exploitable. Require a finding to appear in a majority of runs before it reaches a human, and the false positive rate drops sharply while recall degrades only slightly.
from collections import Counter
def consensus_findings(runs, threshold=3):
"""Keep findings that independent runs agree on.
runs: list of lists of findings, one list per model invocation.
A finding is identified by (file, line, vulnerability_class), which
tolerates wording differences between runs while still requiring
the runs to point at the same defect.
"""
counts = Counter(
(f["file"], f["line"], f["cwe"])
for run in runs
for f in run
)
return [key for key, seen in counts.items() if seen >= threshold]
Two details make this work in practice. Match on structural identity rather than the description text, since models phrase the same bug differently every time. And vary something between runs, whether that is the model, the temperature, or the framing of the prompt, because five identical runs correlate their errors and the vote tells you nothing.
Consensus that survives contact with real output
The nine lines above state the idea. Running it against real model output breaks it inside an hour, for four reasons that each have a fix.
Line numbers drift. Asked about the same injection, one run points at the line building the query, another at the line executing it, a third at the function signature. Exact-line matching splits one finding into three singletons and the vote never fires. Bucket by enclosing function, or by a window of a few lines, and the votes land together.
Class labels drift too. The same defect arrives as CWE-89, CWE-943, “SQL injection”, and “unsanitised user input in query construction”. Normalise onto a short internal taxonomy of ten or fifteen buckets before counting, and accept that the mapping loses information at the edges.
Correlated runs vote as one. Five calls to the same model at temperature zero with the same prompt agree with themselves, and the tally then measures determinism rather than truth. Vary an axis on purpose: two different models, or one model with the prompt framed once as “review this code for defects” and once as “you are writing an exploit for this function”. Record which axis varied, because you will want to know later which pairing produced the disagreements.
Weight the evidence rather than counting heads. A run that returned a concrete input reaching the sink has produced something checkable; a run that returned prose has produced an opinion. Give the first more weight, run any reproducer you get, and promote a confirmed finding regardless of the tally.
import re
from collections import defaultdict
CWE_BUCKETS = {
"cwe-89": "injection.sql", "cwe-943": "injection.sql",
"cwe-78": "injection.command", "cwe-77": "injection.command",
"cwe-79": "xss", "cwe-22": "path-traversal",
"cwe-918": "ssrf", "cwe-502": "deserialisation",
"cwe-798": "hardcoded-credential", "cwe-862": "missing-authz",
"cwe-863": "missing-authz", "cwe-639": "idor",
}
TEXT_HINTS = [
(r"sql\s*inject", "injection.sql"),
(r"command\s*inject|shell\s*inject", "injection.command"),
(r"cross[- ]site\s*script", "xss"),
(r"path\s*traversal|directory\s*traversal", "path-traversal"),
(r"server[- ]side\s*request", "ssrf"),
(r"deserial", "deserialisation"),
(r"hardcoded|hard[- ]coded\s*(secret|key|password)", "hardcoded-credential"),
]
def bucket(finding):
"""Map a model's label onto one internal class, or 'unclassified'."""
raw = (finding.get("cwe") or "").strip().lower()
if raw in CWE_BUCKETS:
return CWE_BUCKETS[raw]
text = f"{finding.get('title', '')} {finding.get('description', '')}".lower()
for pattern, name in TEXT_HINTS:
if re.search(pattern, text):
return name
return "unclassified"
def key_for(finding, window=8):
"""Structural identity: file, coarse location, normalised class."""
line = int(finding.get("line") or 0)
return (
finding["file"],
finding.get("function") or f"lines:{line // window}",
bucket(finding),
)
def consensus(runs, threshold=2, reproducer_wins=True):
"""Aggregate findings across runs that differ in model, prompt, or both.
runs: list of {"axis": "model=A,frame=review", "findings": [...]}
Returns findings ordered by weight, with the evidence attached so a
triager can see who agreed and why.
"""
tally = defaultdict(lambda: {"weight": 0.0, "axes": set(), "evidence": []})
for run in runs:
seen_this_run = set()
for finding in run["findings"]:
key = key_for(finding)
if key in seen_this_run:
continue # one vote per run, however chatty the run
seen_this_run.add(key)
record = tally[key]
record["weight"] += 2.0 if finding.get("reproducer") else 1.0
record["axes"].add(run["axis"])
record["evidence"].append(finding)
promoted = []
for key, record in tally.items():
confirmed = reproducer_wins and any(
f.get("reproducer_verified") for f in record["evidence"]
)
distinct_axes = len(record["axes"])
if confirmed or (record["weight"] >= threshold and distinct_axes >= 2):
promoted.append({
"key": key,
"weight": record["weight"],
"axes": sorted(record["axes"]),
"verified": confirmed,
"evidence": record["evidence"],
})
return sorted(promoted, key=lambda item: (-item["verified"], -item["weight"]))
The distinct_axes >= 2 condition is the part teams skip and then wonder why consensus bought them nothing. Two votes from the same model with the same prompt are one vote wearing a disguise.
The pre-filter that decides what the model ever sees
Consensus cleans up what comes back. The larger lever sits on the other side of the call, in deciding what gets sent at all.
A deterministic pass over the repository can answer three cheap questions that a model should never be asked to answer: where are the dangerous sinks, which functions changed recently, and which of those functions sit behind a request handler. Semgrep, CodeQL, or a hand-rolled AST walk all work. The output is a candidate list, and the candidate list is what you pay frontier prices to reason about.
Three things improve at once when you do this. Cost drops in proportion to how aggressively the filter cuts. Reasoning quality improves, because a model given one function and its immediate callers reasons more carefully than one given a directory. And you gain a measurable blind spot: everything the filter rejected is, by construction, invisible to the model, so you can sample rejections and quantify what you are missing. A pipeline without that property has an unknown false negative rate and no way to estimate it.
import json
import subprocess
from pathlib import Path
DANGEROUS = {
"sql": ["execute(", "raw(", "cursor.execute", "text("],
"command": ["subprocess.", "os.system", "popen("],
"deserial": ["pickle.loads", "yaml.load(", "marshal.loads"],
"path": ["open(", "send_file", "os.path.join"],
}
SANITISER_HINTS = ["parameterize", "escape(", "bindparam", "quote(", "allowlist"]
def recently_changed(repo, days=90):
"""Functions in files touched recently carry higher prior risk."""
out = subprocess.run(
["git", "-C", repo, "log", f"--since={days} days ago", "--name-only",
"--pretty=format:"],
capture_output=True, text=True, check=True,
).stdout
return {line.strip() for line in out.splitlines() if line.strip()}
def score(source, path, changed_files, reachable_from_handler):
"""Cheap risk score. Deterministic, explainable, and easy to argue with."""
points = 0
for sinks in DANGEROUS.values():
if any(token in source for token in sinks):
points += 3
if not any(hint in source for hint in SANITISER_HINTS):
points += 2
if str(path) in changed_files:
points += 2
if path.as_posix() in reachable_from_handler:
points += 3
if len(source.splitlines()) > 200:
points -= 1 # huge functions blow the context budget
return points
def select_candidates(repo, reachable_from_handler, budget=900, floor=5):
"""Return the highest-scoring units, capped by what you can afford."""
changed = recently_changed(repo)
candidates = []
for path in Path(repo).rglob("*.py"):
if any(part in {"tests", "vendor", "node_modules", ".venv"}
for part in path.parts):
continue
source = path.read_text(errors="ignore")
points = score(source, path.relative_to(repo), changed,
reachable_from_handler)
if points >= floor:
candidates.append({"path": str(path), "score": points,
"bytes": len(source)})
candidates.sort(key=lambda c: -c["score"])
selected = candidates[:budget]
rejected = candidates[budget:]
print(json.dumps({
"selected": len(selected),
"rejected_over_budget": len(rejected),
"sample_for_blind_spot_audit": [c["path"] for c in rejected[:20]],
}, indent=2))
return selected
Keep that rejection sample. Once a quarter, have somebody read twenty rejected functions properly and record whether any contained a real defect. That single number tells you whether the filter is doing its job or quietly hiding half your attack surface.
The scaffolding is the product
A survey published by Semgrep in July 2026 sorted the emerging open-source security scaffolds into three families, and the taxonomy is a useful way to think about the design space.
The first family puts the model in the lead and asks it to generate working exploits. A finding is accepted when the exploit runs, which is the strongest possible evidence and also the most expensive to obtain. The second family grafts security methodology onto a general coding agent through skills or prompts, teaching it to think like a reviewer without changing the underlying loop. The third family constrains the model with deterministic tooling, using traditional static analysis to narrow the search space before the model reasons about what remains.
Named projects in the survey include Anthropic’s defending-code-harness, Cloudflare’s security-audit-skill, Trail of Bits’ skills collection, Visa’s VVAH, and Vercel’s deepsec. Google shipped Mantis, a modular toolkit that chains threat modelling, scanning, deduplication, crash reproduction, patching, and risk calibration into standardised reports. Cisco released Antares, a set of open-weight models for vulnerability localisation, alongside a 500-task benchmark and CLI tooling that keeps the code on-premises.
Semgrep’s own conclusion was that no dominant reference implementation has emerged, which is the correct read. The field is nine months old and every serious participant is still discovering which parts of the problem the model should own.
My read on the three families: the hybrid approach is where the near-term value sits. Deterministic tools have negligible false negative rates on the vulnerability classes they cover and terrible precision, since they flag every string concatenation near a query. Models have the opposite profile. Using the deterministic pass to generate candidates and the model to reason about exploitability plays to both, and the model’s context stays small enough that it can reason carefully.
Picking a family for your situation
Semgrep’s taxonomy describes what exists. Choosing among the three depends on what your code is written in, what test infrastructure you already own, and what you can afford to run every night.
| Family | Strongest on | What it costs | Failure mode | Pick it when |
|---|---|---|---|---|
| Exploit-first | Memory safety, parsers, anything with a crash oracle | Highest compute per finding; needs a runnable target and a seed corpus | Silent on defects that produce no observable crash | You ship C, C++, or a parser, and already run fuzzing |
| Methodology on a coding agent | Broad coverage, fastest to adopt | Cheap per run, expensive in triage | Unreproducible runs, so no stable precision number | You want a pilot this week and can absorb the noise |
| Deterministic filter plus model | Injection, authorisation, secrets, configuration | Engineering time to build and tune the filter | Blind to anything the filter never surfaces | You already run SAST and want its output made useful |
Exploit-first is the only family whose findings arrive pre-adjudicated, which is why both Big Sleep and the AIxCC systems sit there. The price is a build, a corpus, and an oracle that can tell a crash from a shrug. For a Rails or Django application, generating an exploit means driving a live instance with authentication and session state, which is a test-environment project rather than a security-tooling project, and teams routinely underestimate it by a quarter.
The methodology-on-an-agent family is the easiest to adopt and the hardest to defend in a review. You cannot rerun last month’s scan and get last month’s answer, so no trend line survives, and the question “did we get better?” has no evidence behind it. Useful as a research tool, weak as a control.
Hybrid asks for the most engineering up front and repays it as the only family with a blind spot you can measure. My recommendation for a team starting today: build the hybrid pipeline, borrow exploit-first for whatever memory-unsafe components you ship, and run the agent-with-skills family as a time-boxed experiment with a date on which you either measure it or switch it off.
The bug classes where models earn their cost
Aggregate recall numbers hide enormous variation by vulnerability class, and knowing the shape of that variation is what lets you point the tool somewhere useful.
Models do well on defects that are visible in a bounded region of code and have a recognisable textual signature. Injection flaws where untrusted data reaches a sink in the same function, missing authorisation checks on an endpoint that resembles its neighbours, hardcoded credentials, unsafe deserialisation of a known-dangerous format, and misuse of a cryptographic primitive in an obvious way. These are, roughly, the classes a good senior reviewer catches by reading, and the model is doing a competent imitation of reading.
They do notably worse on defects that live in the relationship between distant parts of a system. A race condition between two services, an authorisation model that is correct in every individual handler and wrong in aggregate, a state machine that permits an illegal transition through an unusual sequence of valid calls. Holding the whole system in view is expensive in context and the reasoning chains get long, which is precisely where accuracy degrades.
Business logic flaws are the hardest category and worth calling out separately, because they are also where the expensive breaches come from. A discount that stacks with itself, a refund path that does not check whether the order shipped, an API that lets you enumerate other users’ identifiers: none of these look wrong in the code. They look wrong against an understanding of what the business intended, which the model does not have and cannot infer from the source.
The practical consequence is to point these tools at the first category, keep human review on the third, and be honest in your reporting that a clean model scan says nothing about logic. Teams that present model output as general assurance create a false sense of coverage in exactly the area where coverage is weakest.
A scorecard by class, and the check that settles each one
The split above sorts classes by how hard they are to see. A more operational sort asks whether a cheap independent check exists, because that decides whether a finding reaches a human as a claim or as a fact.
| Class | Model performance | Independent check | Deploy as |
|---|---|---|---|
| SQL and command injection in one function | Strong | Taint query plus a crafted input against a test fixture | Automated PR comment |
| Reflected XSS in a template | Strong | Render the template with a probe payload | Automated PR comment |
| Hardcoded secrets and keys | Strong, and cheap tools already do it | Validate the credential against the provider | Automated, deduplicated against your secret scanner |
| Path traversal and SSRF | Good | Send the request in a sandbox and watch egress | Automated, with the request recorded |
| Memory safety in C and C++ | Good when paired with a fuzzer | Sanitiser crash with a reproducer | Automated, high trust |
| Unsafe deserialisation | Good | Gadget availability check against the classpath | Human review, model output as context |
| Crypto misuse | Strong on obvious misuse, weak on protocol errors | Rule-based check covers the obvious half | Automated for the obvious half only |
| Missing authorisation on an endpoint | Mixed, depends on neighbours being correct | Replay the request without the token | Automated where replay exists, human otherwise |
| IDOR and tenant isolation | Weak | Two-account differential test, expensive to build | Human review, model as lead generator |
| TOCTOU and races | Weak | Almost never cheap | Human review only |
| Cross-service authorisation | Weak | Nothing short of a model of the whole system | Human review only |
| Business logic | Weak by construction | Requires product intent | Human review only |
The right-hand column is the deployment decision, and it follows from the third rather than the second. Anything with a cheap check can go straight into a pull request, because the finding arrives with evidence attached and a reviewer spends seconds rather than minutes on it. Anything without one is a lead, and leads belong in a queue somebody works through deliberately. Posting both with the same visual weight teaches reviewers to treat all of it with the same weight, which is how the bottom four rows poison the top five.
One more column worth adding for your own repository: what your existing SAST already catches. The strong rows are exactly where a mature Semgrep or CodeQL ruleset already fires. If eighty percent of what the model flags in that class was already flagged for free, you are paying frontier prices for better ranking and fewer false positives, which may well be worth it, and should be a decision somebody made rather than an outcome nobody noticed.
Cost, and where it goes
The economics deserve a paragraph because they drive design decisions that get made implicitly.
Analysing a large repository in full costs real money at frontier model prices, and the multi-run consensus approach multiplies that by three to five. A codebase of a million lines analysed at any serious depth is a four-figure run, and running it nightly is a budget line somebody will question.
The cheap alternatives are all forms of narrowing. Diff-scoped analysis reduces the input by three orders of magnitude for the same coverage of new code. A deterministic pre-filter that surfaces only functions containing a dangerous sink cuts most of the remainder. Cheaper models for the first pass with escalation to an expensive model only for candidates that survive gives you most of the recall at a fraction of the spend, which is the same insight as the multi-run finding applied to cost rather than accuracy.
What does not work is silently reducing depth to fit a budget while continuing to describe the scan as comprehensive. If cost forced you to analyse only changed files, the report should say so, because the next person to read it will otherwise assume the untouched code was examined.
Token math before the budget conversation
Narrowing is the answer, and the sizing that justifies it takes ten minutes with a calculator rather than a quarter of pilot data.
Start with a conversion you should verify on your own code rather than accept from me: roughly ten tokens per line of source for mainstream languages, so a four-hundred-line file runs about four thousand tokens. Single-shot file review adds maybe fifteen hundred tokens of instructions and returns under a thousand. Agentic analysis costs far more, because the agent reads the function, then two or three callers, then a configuration file, then the test, and every turn resends the accumulated context. Budget twenty-five thousand input tokens per unit for an agentic pass and replace that figure with a measured one in week one.
At mid-2026 rates a mid-tier model sits near $3 per million input tokens and $15 per million output, with frontier tiers running a few times higher. The absolute numbers below will drift; the ratios between the rows will not.
| Mode | Units | Input tokens per unit | Runs | Total input | Cost at $3 per million |
|---|---|---|---|---|---|
| Whole repo, single-shot per file | 2,500 files | 6k | 1 | 15M | about $45 |
| Whole repo, agentic, 3-run consensus | 20,000 functions | 25k | 3 | 1,500M | about $4,500 |
| Pre-filtered agentic, 3-run consensus | 900 candidates | 25k | 3 | 68M | about $205 |
| Diff-scoped per PR, 3-run consensus | 12 units | 25k | 3 | 0.9M | about $2.70 |
At four hundred pull requests a month, the last row costs a little over $1,000, which is a line item somebody signs off in a meeting. The second row run nightly is roughly $135,000 a month, which is why nobody runs it nightly and why the phrase “full repository scan” in a vendor deck deserves a follow-up question about depth.
Output tokens cost several times input and consensus multiplies them, though they stay a minority of spend unless the prompt invites essays. Ask for structured findings with a two-sentence justification and cap the response length; a scan that returns three paragraphs per finding is burning money to produce something nobody reads.
Prompt caching changes the arithmetic when repeated calls share a large stable prefix, which is exactly the shape of a security scan: same system prompt, same taxonomy, same repository summary, different file at the end. A cache read typically costs around a tenth of the base input rate, against a write premium of roughly 1.25 times, so the break-even lands at two reads. Structure calls so the invariant material comes first and the file under review comes last, and check your hit rate rather than assuming it. A pipeline that shuffles context order or stamps a timestamp into the system prompt gets a hit rate near zero, and nobody notices until the invoice arrives.
Two costs the spreadsheet usually omits. Triage time, which at ten minutes per finding and a hundred findings a week is most of an engineer. And the rerun tax: every prompt change invalidates your historical precision measurements, so a team that tunes the prompt weekly never accumulates enough adjudicated findings to know whether it is improving.
Where this belongs in a pipeline
The deployment question that matters is which decisions these findings are allowed to influence, and the answer should start conservative.
Blocking a build on a model-generated finding is a bad first move. Precision is not yet good enough, and a security control that halts delivery on false positives gets disabled within a month, taking the true positives with it. Start with a channel where a wrong answer costs a few minutes: a comment on a pull request that a human reviewer can dismiss, or a nightly report the security team triages.
Diff-scoped analysis is a much better fit than whole-repository scanning. Reviewing the fifty lines that changed puts the base rate back in reasonable territory, keeps cost predictable, and matches the moment when a developer has the relevant context loaded. Whole-repository sweeps are worth running occasionally, with the expectation that most of the output is noise and the exercise is prospecting rather than assurance.
Measure precision on your own code before expanding scope. Take a hundred findings, have someone competent adjudicate them, and compute what fraction were real. That number, not the published benchmark, determines whether the tool earns a place in the workflow. Teams that skip this step end up arguing about vendor claims instead of their own data.
Wiring it into CI without slowing delivery
Diff-scoped review is the recommendation; the workflow that implements it has a few sharp edges worth naming before somebody discovers them in production.
Run on pull_request, never on pull_request_target combined with a checkout of the pull request head. That pairing hands repository secrets, including your model API key, to code an outside contributor controls, and it is the single most reliable way to turn a security tool into a security incident. Fork pull requests consequently get no key at all, so either skip them or route them through a separate workflow that a maintainer approves.
Time-box the job and let it fail open. A model-generated comment is advisory, and a scanner that blocks the merge queue for twenty minutes because a provider is having a bad afternoon gets removed by the first person with admin rights. Post one review with inline comments rather than a comment per finding, and replace the previous bot review on each push, otherwise a force-push-heavy branch accumulates duplicates until reviewers collapse the thread and stop reading.
Filter at the diff-parsing stage, not in the prompt. Generated files, vendored dependencies, lockfiles, minified bundles, and test fixtures produce a large share of the noise in a typical pilot, and they cost real money to analyse. Finally, record every run: commit SHA, model, prompt version, findings, and later whether a human agreed. Without that record you cannot compute precision in three months, and the argument about whether the tool is working becomes a matter of opinion.
name: security-review
on:
pull_request:
types: [opened, synchronize, reopened]
permissions:
contents: read
pull-requests: write
jobs:
review:
# Fork PRs have no access to secrets; run them through an approved workflow.
if: github.event.pull_request.head.repo.full_name == github.repository
runs-on: ubuntu-latest
timeout-minutes: 12
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Collect changed hunks
id: diff
run: |
git diff --unified=15 \
"origin/${{ github.base_ref }}...HEAD" \
-- . ':(exclude)**/vendor/**' \
':(exclude)**/*.lock' \
':(exclude)**/*.min.js' \
':(exclude)**/testdata/**' \
> changed.diff
echo "bytes=$(wc -c < changed.diff)" >> "$GITHUB_OUTPUT"
- name: Review changed code
if: steps.diff.outputs.bytes != '0'
continue-on-error: true # advisory control: never block the queue
env:
MODEL_API_KEY: ${{ secrets.MODEL_API_KEY }}
GH_TOKEN: ${{ github.token }}
PR_NUMBER: ${{ github.event.pull_request.number }}
COMMIT_SHA: ${{ github.event.pull_request.head.sha }}
PROMPT_VERSION: '2026-07-24'
run: python .ci/review_diff.py changed.diff
- name: Archive run record
if: always()
uses: actions/upload-artifact@v4
with:
name: security-review-${{ github.event.pull_request.number }}
path: findings.jsonl
retention-days: 90
#!/usr/bin/env python3
"""Diff-scoped review that posts one consolidated PR review and logs everything."""
import json
import os
import subprocess
import sys
BOT_MARKER = "<!-- security-review-bot -->"
MAX_INLINE_COMMENTS = 10
def dismiss_previous_reviews(pr):
"""Replace, do not accumulate. Force-push-heavy branches get unreadable fast."""
raw = subprocess.run(
["gh", "api", f"repos/{os.environ['GITHUB_REPOSITORY']}/pulls/{pr}/reviews"],
capture_output=True, text=True, check=True,
).stdout
for review in json.loads(raw):
if BOT_MARKER in (review.get("body") or ""):
subprocess.run(
["gh", "api", "-X", "PUT",
f"repos/{os.environ['GITHUB_REPOSITORY']}/pulls/{pr}"
f"/reviews/{review['id']}/dismissals",
"-f", "message=superseded by a newer scan",
"-f", "event=DISMISS"],
check=False,
)
def post_review(pr, sha, findings):
verified = [f for f in findings if f["verified"]]
body = [
BOT_MARKER,
f"**Security review** of `{sha[:8]}` "
f"({len(findings)} findings, {len(verified)} with a reproducer).",
"",
"Advisory only. Dismiss anything wrong and, if you have thirty seconds, "
"mark it wrong in the thread so it lands in the precision dataset.",
]
comments = [
{
"path": f["file"],
"line": f["line"],
"side": "RIGHT",
"body": (f"**{f['class']}** ({'verified' if f['verified'] else 'unverified'}, "
f"weight {f['weight']}, agreed by {', '.join(f['axes'])})\n\n"
f"{f['justification']}\n\n"
f"<sub>prompt {os.environ['PROMPT_VERSION']}</sub>"),
}
for f in sorted(findings, key=lambda f: (-f["verified"], -f["weight"]))[
:MAX_INLINE_COMMENTS
]
]
payload = {"commit_id": sha, "event": "COMMENT",
"body": "\n".join(body), "comments": comments}
subprocess.run(
["gh", "api", "-X", "POST",
f"repos/{os.environ['GITHUB_REPOSITORY']}/pulls/{pr}/reviews",
"--input", "-"],
input=json.dumps(payload), text=True, check=True,
)
def main(diff_path):
from review_lib import consensus, run_model # your pipeline
diff = open(diff_path).read()
runs = [
{"axis": "model=cheap,frame=review", "findings": run_model(diff, "cheap", "review")},
{"axis": "model=cheap,frame=exploit", "findings": run_model(diff, "cheap", "exploit")},
{"axis": "model=strong,frame=review", "findings": run_model(diff, "strong", "review")},
]
findings = consensus(runs, threshold=2)
with open("findings.jsonl", "w") as log:
for f in findings:
log.write(json.dumps({
"commit": os.environ["COMMIT_SHA"],
"prompt_version": os.environ["PROMPT_VERSION"],
"adjudication": None, # filled in by a human, later
**f,
}) + "\n")
if findings:
dismiss_previous_reviews(os.environ["PR_NUMBER"])
post_review(os.environ["PR_NUMBER"], os.environ["COMMIT_SHA"], findings)
if __name__ == "__main__":
main(sys.argv[1])
Deployment modes ranked by how fast they burn trust
The workflow above sits deliberately at the low-consequence end of a spectrum. Knowing where the other modes sit helps when somebody proposes promoting it after a good week.
| Mode | Feedback latency | Volume a team sees | Precision needed | First thing that breaks |
|---|---|---|---|---|
| Blocking merge gate | Immediate | Every PR | Above 90% | Someone adds a skip label and it becomes permanent |
| Required reviewer sign-off | Immediate | Every PR | Above 70% | Rubber-stamping within three weeks |
| Advisory PR comment | Immediate | Every PR | Above 25% | Comments get collapsed by default |
| Nightly digest to the security team | Under a day | One inbox | Above 15% | The digest becomes an unread folder |
| Weekly triage queue | Days | One rota | Above 10% | Backlog grows faster than the rota clears it |
| Quarterly prospecting sweep | Weeks | One analyst | Any | Nothing, which is why it is the safe place to start |
Those precision thresholds are judgement rather than measurement, and yours will differ, but the ordering holds: the tighter the loop and the wider the audience, the more precision a mode consumes before it earns its place. Start at the bottom of that table with the mode whose failure costs an analyst an afternoon, measure, and promote one row at a time.
Note where the loop returns. Promotion is per class, not per tool. Hardcoded credentials with a validated key might reasonably block a merge while the same pipeline’s authorisation findings stay advisory forever, and a tool that promotes wholesale drags its weakest class into its strongest deployment mode.
Building a ground-truth set and measuring against it
Every threshold above needs a number you measured yourself, and measuring recall and precision are different projects with different data requirements.
Recall needs known bugs, and the cheapest source is your own history. Mine the git log for commits referencing a CVE, a security ticket prefix, or the words that show up in your remediation workflow, then reconstruct the vulnerable tree by reverting each fix. Thirty reconstructed defects is a usable corpus and most teams with three years of history can assemble that in a day. The reconstructed set has one property no public benchmark can offer: if your code is private, the model did not train on the fix, so recall measured this way is genuinely predictive rather than an upper bound. Synthetic injection is the fallback when history is thin, and AIxCC’s own results are the caveat worth remembering, since systems that found eighty-six percent of the synthetic defects surfaced a much smaller absolute number of real ones.
Precision needs adjudication, and the sampling matters more than the count. Adjudicate a random sample of output rather than the top of the ranked list, because the top is where the tool is strongest and grading it there measures your sort order rather than your detector. Use two reviewers, blind to each other, and record disagreements as a first-class number: if competent people disagree on fifteen percent of findings, your precision figure carries at least that much slop and no amount of extra sampling removes it.
import json
from math import sqrt
def wilson_interval(successes, trials, z=1.96):
"""Confidence interval for a proportion. The normal approximation lies
badly at the small counts and low rates typical of this work."""
if trials == 0:
return (0.0, 0.0)
p = successes / trials
denom = 1 + z**2 / trials
centre = (p + z**2 / (2 * trials)) / denom
margin = (z / denom) * sqrt(p * (1 - p) / trials + z**2 / (4 * trials**2))
return (max(0.0, centre - margin), min(1.0, centre + margin))
def measure(adjudicated_path, ground_truth_path):
"""adjudicated.jsonl: findings a human labelled true or false.
ground_truth.jsonl: known defects, from reverted security fixes."""
adjudicated = [json.loads(line) for line in open(adjudicated_path)]
truth = [json.loads(line) for line in open(ground_truth_path)]
labelled = [f for f in adjudicated if f["adjudication"] in ("true", "false")]
true_positives = sum(1 for f in labelled if f["adjudication"] == "true")
precision = true_positives / len(labelled) if labelled else 0.0
lo, hi = wilson_interval(true_positives, len(labelled))
found = {(f["file"], f["class"]) for f in labelled
if f["adjudication"] == "true"}
recalled = sum(1 for d in truth if (d["file"], d["class"]) in found)
recall = recalled / len(truth) if truth else 0.0
disagreements = sum(1 for f in adjudicated if f.get("reviewers_disagreed"))
reviewed = sum(1 for f in adjudicated if f.get("reviewer_count", 0) >= 2)
by_class = {}
for f in labelled:
entry = by_class.setdefault(f["class"], [0, 0])
entry[1] += 1
if f["adjudication"] == "true":
entry[0] += 1
return {
"sample_size": len(labelled),
"precision": round(precision, 3),
"precision_95_interval": (round(lo, 3), round(hi, 3)),
"recall_on_ground_truth": round(recall, 3),
"ground_truth_size": len(truth),
"reviewer_disagreement": round(disagreements / reviewed, 3) if reviewed else None,
"precision_by_class": {
name: round(hits / total, 3) for name, (hits, total) in by_class.items()
},
}
if __name__ == "__main__":
print(json.dumps(measure("adjudicated.jsonl", "ground_truth.jsonl"), indent=2))
Read that interval before anyone declares a trend. A hundred adjudicated findings with twelve real ones gives a point estimate of 12% and a ninety-five percent interval of roughly 7% to 20%, which cannot distinguish this quarter’s 12% from last quarter’s 18%. Version the ground-truth set, freeze it between prompt changes, and add to it rather than replacing it, because the whole value of the exercise is comparability across time.
Anti-patterns that show up in every early deployment
The failures below are not exotic. They recur across teams with different tooling, different languages, and different levels of skill, which suggests they come from the shape of the problem rather than from anyone’s incompetence.
| Symptom | Underlying cause | Fix |
|---|---|---|
| Findings count reported as the metric | No adjudicated denominator | Report precision with its interval, and the sample size |
| Precision looks great in the pilot, terrible in production | Graded the top of the ranked list | Sample randomly from all output |
| Consensus buys nothing | All runs share model, prompt, and temperature | Require agreement across at least two varied axes |
| Everything is High severity | The model wrote the severity | Derive severity from class and reachability, deterministically |
| Reviewers stopped reading the comments | Weak classes posted alongside strong ones | Route classes by whether a cheap check exists |
| Every scan finds the same thing again | No suppression state | Persist decisions keyed on structural identity, not line number |
A few deserve more than a table row. Auto-filing tickets from model output is the fastest way to destroy a security backlog, because a queue that grows faster than anyone clears it loses its signalling value and the genuinely urgent item sits at position four hundred. Keep a human between the detector and the tracker until precision justifies otherwise.
Trusting the model’s own confidence score is another. Self-reported confidence correlates weakly with correctness and it correlates strongly with how fluent the explanation was, which is precisely backwards for triage. Vote counts across varied runs and the presence of a reproducer are both better predictors, and both are things you compute rather than things you are told.
Reporting diff-scoped coverage as full coverage deserves its own warning, because it is the one that gets people hurt. The report goes into a compliance folder, somebody reads it eighteen months later, and concludes the untouched authentication module was examined. Write the scope into the artefact.
The last one is the most consequential and comes with real numbers attached. Do not send unverified model output to somebody else’s maintainers. The curl project watched its confirmed-vulnerability rate fall from above fifteen percent to below five as low-effort submissions arrived, with roughly one in five 2025 reports being AI-generated noise, each one consuming hours from a seven-person volunteer team. Daniel Stenberg compared the triage burden to a denial-of-service against the project, and curl ended bounty payouts over it. If your pipeline produces something you want to disclose upstream, verify it first, attach the reproducer, and send one report rather than ten.
AI-assisted exploitation and the shrinking n-day window
Everything above assumes the pipeline is yours to tune. The same capability is available to people attacking you, and their version of the problem has different economics and a different clock.
The clearest change is on the n-day side. A published patch is a precise description of a vulnerability, written by someone who understood it, and turning a patch commit into a working trigger has always been the slowest part of weaponisation. Models are good at exactly that task: bounded scope, clear signal, an oracle in the form of the patched and unpatched builds. The practical consequence is that the interval between a security release and exploitation attempts against unpatched installs is compressing, and any patching SLA calibrated on the old interval is quietly out of date.
On the zero-day side, XBOW’s autonomous system reaching the top of HackerOne’s United States leaderboard in June 2025 is the data point worth sitting with. Whatever the debates about scope and target selection, a system produced enough validated, paid-out reports to outrank human researchers, and it did so with a low false positive rate because it validated before submitting. The same design decision that makes a defensive pipeline usable makes an offensive one credible.
The curl case is the second-order effect, and it cuts the other way. Volume without validation is a denial-of-service on human attention, and defenders absorb that cost whether the volume is malicious or merely careless. Triage capacity is now a security resource with a budget, and treating it as free is how a programme ends up unable to process the one report that mattered.
Four things follow for a defensive programme:
- Patch velocity moves from a hygiene metric to a control with a measurable window. Track time from vendor release to fleet coverage, per component, and set the target against how quickly a competent attacker can build a trigger rather than against last year’s average.
- Inventory becomes the binding constraint. Fast patching is worthless for software you did not know you were running, and the components most likely to get a model-assisted exploit are the widely deployed ones you inherited rather than chose.
- Spend detection effort on exploitation behaviour rather than on more scanning. A pipeline that reliably alerts when deserialisation runs with attacker-controlled input is worth more than one that lists every place deserialisation appears.
- Budget the triage queue explicitly, with a named owner and a stated capacity, so that saturation shows up as a visible metric rather than as a backlog nobody admits to.
What changes for defenders
The strategic picture is less comfortable than the tactical one. Whatever recall these systems have against your code, attackers have the same tools pointed at the same targets, and they benefit from a structurally easier problem.
An attacker needs one exploitable finding and can discard every false positive without cost. A defender needs to find all of them and pays triage cost for each mistake. That asymmetry means equal capability on both sides favours the attacker, and it argues for spending the capability budget on the places where a single miss is fatal: authentication, authorisation, deserialisation, and anything parsing untrusted input.
The WordPress case from the same week illustrates the direction of travel. A researcher used a frontier model to find a pre-authentication SQL injection in the batch API through a validation desync, then chained cache poisoning and embed handling into temporary admin privileges and, from there, plugin-based remote code execution on a default installation. That is a real chain in widely deployed software, found with model assistance.
Which suggests the practical posture. Adopt these tools for the throughput they add to review, hold them to a precision number you measured yourself, and assume anything you can find this way is already being found by people who do not file responsible disclosures. The benchmark number is a starting point for that conversation rather than a conclusion.