Using Coding Agents to Validate Detections End to End

The rule that never fired

Every security team has detection rules that have never produced a true positive, and no way to tell the good ones from the broken ones. A rule that has been silent for a year might be watching for something that has not happened, or might be referencing a log field that a vendor renamed in an update eight months ago. Both look identical from the console.

Work published in July 2026 described closing this loop with automation: run an emulated attack, ingest the resulting telemetry, and verify that the detection actually fired. The author built agent skills that write and execute TTPForge attack simulations, then refine the detection iteratively when it fails to catch the technique.

The interesting part has little to do with a model being able to write a detection rule. What changes is that the expensive, tedious, repeatedly-deferred step of proving a rule works becomes cheap enough to run continuously.

Why detection validation gets skipped

Understanding the failure mode explains why automation helps here specifically.

Writing a detection is quick. You read a threat report, identify an observable, write a query, and deploy. Validating it means reproducing the attack in an environment where the relevant telemetry is being collected, waiting for the pipeline to ingest, and confirming the alert appears. That is an afternoon of work per rule, it requires an environment safe to attack, and it competes against the next incident.

So teams ship rules on the strength of the query looking correct. The query does look correct. It also frequently depends on a field that only exists when a particular agent policy is enabled, or matches a process name that varies by Windows version, or is scoped to a log source that stopped forwarding in June. None of that shows up until an incident, when the post-mortem includes the sentence “we had a rule for that”.

Coverage maps make this worse by being reassuring. A grid showing detections mapped to techniques counts rules, not working rules, and every unvalidated rule in that grid is a claim without evidence.

What the loop looks like

The automation has four stages, and each one is mechanical enough to delegate while the judgement stays with the engineer.

Emulation runs a specific technique in a controlled environment. Frameworks in this space express a technique as a declarative file with a setup step, the action itself, and a cleanup step, which makes the run repeatable and reversible. The agent’s job is translating “credential dumping via LSASS access” into a concrete, runnable simulation.

Collection waits for telemetry to arrive and pulls it back. This stage is where most homegrown attempts break, because ingestion latency varies and a check that runs too early reports a false failure. Polling with a generous timeout beats a fixed sleep.

Verification queries the detection platform for an alert matching the technique within the time window of the run. The output is binary and unambiguous, which is what makes the whole thing worth building.

Refinement is the stage where the agent earns its cost. When no alert fired, it has the emulation definition, the raw telemetry that was generated, and the current rule. Comparing what the attack produced against what the rule looks for is exactly the kind of bounded, evidence-rich task these systems handle well, and the output is a concrete proposed change rather than a suggestion.

# The shape of the loop. Each stage is small; the value is in running
# it for every rule, every week, without a human in the middle.
def validate(technique_id: str, rule_id: str) -> Result:
    started = now_utc()
    run = emulate(technique_id)              # execute, capture host + time
    telemetry = collect(
        host=run.host,
        since=started,
        timeout=timedelta(minutes=10),       # generous: ingestion is slow
    )
    alerts = query_alerts(rule_id=rule_id, since=started, host=run.host)

    if alerts:
        return Result(ok=True, technique=technique_id, evidence=alerts)
    return Result(
        ok=False,
        technique=technique_id,
        telemetry=telemetry,                 # what the attack actually made
        rule=get_rule(rule_id),              # what we were looking for
    )

A failing result carries everything needed to diagnose it. That payload is what you hand to the agent, and the quality of the proposed fix tracks the quality of that context far more than it tracks the model.

Picking a framework for the emulation stage

That context starts at the emulation stage, which means the framework you point the agent at shapes how honest the whole loop can be. Four open-source projects cover most of what a detection team needs, and they disagree about scope and about how a technique gets expressed.

Framework	Maintainer	Scope	How a technique is defined	Agent model
Atomic Red Team	Red Canary	Windows, macOS, Linux endpoints	YAML atomic tests, one technique per file, often several procedures each	Agentless, run on the host
MITRE Caldera	MITRE	Endpoints and network, chained operations	Abilities and adversary profiles mapped to ATT&CK, several hundred procedures by default	Agent-based implants
Stratus Red Team	Datadog	Cloud control planes across AWS, GCP, Azure, and Kubernetes	Techniques compiled into one CLI, each with warmup, detonate, and revert	Agentless, calls cloud APIs
TTPForge	Meta	Cross-platform endpoints	YAML with ordered steps, arguments, and per-step inline cleanup	Agentless, local runner

Atomic Red Team is the easiest first target because each test is small, portable, and maps to a single technique, which is the granularity the loop wants. Caldera goes further by chaining steps into a full operation, useful once you want to validate a detection that only fires on a sequence, though its implant model adds a moving part you now have to keep healthy in the lab. Stratus Red Team earns its place the moment your detections cover cloud control-plane activity, since it detonates against real cloud APIs and produces the CloudTrail or activity-log records your rules read. TTPForge is worth a look when your procedures are custom, because cleanup attaches to each step rather than getting bolted on at the end, which makes a partial run reversible.

Match the framework to where your alerts actually live. A team whose detections are ninety percent endpoint gets more from Atomic Red Team and TTPForge than from a cloud-first tool, and a team drowning in CloudTrail rules gets the opposite. Nothing stops you running two, and mature programs usually do.

Whatever you choose, the agent’s contribution is turning a written technique into a definition the framework can run and then undo. A TTPForge-style file for local account creation makes the shape concrete.

name: create-local-account
description: Emulates T1136.001, local account creation for persistence
args:
  - name: username
    default: svc-diagnostics
steps:
  - name: baseline
    inline: whoami /user
  - name: create-account
    inline: |
      net user {{ .Args.username }} T3mp-Passw0rd! /add
      net localgroup administrators {{ .Args.username }} /add
    cleanup:
      inline: |
        net localgroup administrators {{ .Args.username }} /delete
        net user {{ .Args.username }} /delete

The cleanup block is not an afterthought in this model; it runs even when the act step fails, which is the property that lets you schedule the technique unattended. With a definition in hand, the four stages fall into a fixed shape that repeats for every rule.

The controls this needs

Running attack simulations under automation is a capability worth being careful with, and a few controls should be non-negotiable before it runs unattended.

Scope it to an environment built for the purpose. Detection validation belongs in a lab that mirrors production telemetry collection, not in production. The telemetry pipeline needs to be identical, since validating against a lab with different agent configuration proves nothing about production, and that requirement is the main engineering cost of doing this properly.

Restrict the technique catalogue to a reviewed set. An agent that can author arbitrary attack simulations and execute them is, in a meaningful sense, malware with a change-management process. Keep the catalogue in version control, require review for additions, and have the runner refuse anything not in it.

Ensure cleanup runs and verify it. Simulations create accounts, modify registry keys, and drop files. A cleanup step that fails silently leaves an environment that drifts further from production with every run, which degrades the value of the whole exercise.

Separate the credentials. The runner needs to execute techniques and read alerts. It does not need permission to modify or disable detection rules, and it should not have it. Proposed rule changes go through the same review as any other change, with a human approving the merge. An automation that can both fail a detection and edit the detection has an obvious degenerate solution available to it.

Building the lab so the telemetry actually matches

The first control, keeping this out of production and inside a purpose-built environment, is the one that quietly sinks most programs, so it deserves a concrete walkthrough. A lab that fails to reproduce production telemetry produces validations that feel reassuring and prove nothing.

The common mistake is standing up fresh Windows and Linux hosts, installing the sensor with its default policy, and calling it a lab. Production endpoints almost never run default policy. They run a specific agent build, a tuned audit configuration, a Sysmon config with particular include and exclude rules, PowerShell script-block and module logging, command-line process auditing, and a forwarding path that lands events in a specific SIEM index or table. Miss any of those and the emulation generates events that either never reach the platform or arrive with different fields than the ones your rules read.

Parity is the requirement, and it is worth writing down as a checklist the lab gets built against:

• the same sensor build and policy as the production endpoint baseline
• the same audit configuration: Sysmon config, PowerShell logging, command-line capture, and object-access auditing
• the same forwarding path, so events land in the same index or table through the same collector
• the same normalization layer, whether that is ECS, OCSF, or a house schema, applied identically
• the same detection content deployed, so you validate the rule that is actually running

Build the lab from infrastructure as code rather than by hand. A golden image derived from the production endpoint baseline, rebuilt with Terraform and Ansible from the same configuration source the fleet uses, keeps the lab from diverging by accident. Snapshot every host before a run and revert after, so a cleanup step that fails does not leave residue that compounds across weeks of runs.

Before you trust a single result, run a parity test. Pick one well-understood technique, detonate it in the lab, and diff the fields, values, and event volume it produces against a stored sample of the same technique from production or a past incident. If the field names differ, if a value is formatted differently, or if the lab emits a fraction of the events production would, fix the lab before validating anything else. A rule that passes in a lab that under-reports is a rule that will stay silent in production.

Isolation is the last piece. The lab lives on its own segment with no route to production, its own cloud subscription or project for cloud techniques, and its own credentials. That containment is what lets you detonate genuinely malicious procedures on a schedule without turning the exercise into the incident it was meant to catch.

Wiring the loop into CI

With a lab that mirrors production, the loop can hang off the same pipeline that ships the detections rather than living in a separate script nobody maintains. Detection-as-code treats rules as source: store them as Sigma in a repository, and make a pull request the only way a rule reaches production. Two tiers of checking then make sense, split by how much they cost to run.

The fast tier runs on every pull request and finishes in seconds. sigma check from sigma-cli validates syntax, required fields, and logsource blocks, and a malformed condition or a missing logsource exits non-zero and blocks the merge. The SigmaHQ Sigma Rules Validator action wraps this for GitHub. Conversion runs next: pySigma backends translate the rule into the query languages you actually run, so one Sigma source becomes Splunk SPL and Microsoft Defender or Sentinel KQL, and a rule that fails to convert for a target platform fails the build. A unit test then runs the converted query against recorded fixtures, EVTX or JSON captures of a known-positive event, so a rule that stops matching a sample it used to catch breaks loudly and cheaply.

name: detection-ci
on:
  pull_request:
    paths: ['detections/**.yml']
jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Install sigma tooling
        run: pip install sigma-cli pysigma-backend-splunk pysigma-backend-microsoft365defender
      - name: Lint rules
        run: sigma check detections/
      - name: Convert to SPL and KQL
        run: |
          sigma convert -t splunk -p ecs_windows detections/ -o out/spl.txt
          sigma convert -t microsoft365defender detections/ -o out/kql.txt
      - name: Unit test against recorded fixtures
        run: python tests/run_fixtures.py --rules detections/ --samples tests/samples/

The slow tier is the emulation-backed validation, and it runs on a schedule rather than per pull request because it needs the lab, minutes of ingestion latency, and real detonation. A nightly job takes the techniques mapped to changed rules, runs them, waits for telemetry, and confirms the alert fired. A failure opens a ticket carrying the telemetry payload rather than blocking a merge that already passed the fast checks. Keeping the two tiers apart is what keeps the pipeline usable: engineers get a syntax answer in seconds and an evidence-backed answer overnight.

Where the agent helps and where it does not

Being specific about the division of labour keeps expectations calibrated.

Agents are good at translation work: turning a written technique description into a runnable simulation, mapping a log schema onto a rule’s field references, and drafting the diff that makes a rule match observed telemetry. These are tasks with clear inputs, checkable outputs, and a fast feedback signal, which is the profile where current models are most reliable.

They are weak at deciding what to detect. Choosing which techniques matter for your organisation depends on your architecture, your threat model, and your tolerance for noise, and a model asked to prioritise will produce a plausible generic ranking that reflects its training data rather than your environment.

They are also weak at judging false positive impact. A rule that catches the emulated attack perfectly may also fire two hundred times a day on a legitimate administrative tool that your operations team runs hourly. Validation proves the rule catches the attack; it says nothing about whether the rule is deployable. That judgement needs someone who knows what normal looks like in your estate, and it should gate every proposed change.

Feeding the agent a payload it can act on

The refinement stage lives or dies on what you hand it, and the point made earlier bears repeating as a practice: the quality of the proposed fix tracks the quality of the context far more than it tracks the model. A raw event dump is the wrong context. Ten thousand lines of unfiltered telemetry bury the three fields that matter, cost tokens you did not need to spend, and invite the agent to pattern-match on noise.

Structure the payload instead. A failing result should carry five things, each small and each load-bearing:

• the normalized event the emulation produced, in the same schema the rule reads
• a field-level diff of what the rule referenced against what actually appeared
• the rule’s current source, in Sigma, rather than the converted query
• the technique definition that was run, so the agent knows what behaviour to expect
• the platform’s field schema for the log source, so a rename is visible rather than guessed

The field-level diff does most of the work.

{
	"rule_id": "lsass-memory-access",
	"outcome": "silent_with_data",
	"rule_matches": { "SourceImage|endswith": "\\mimikatz.exe" },
	"observed": {
		"SourceImage": "svc-host.exe",
		"GrantedAccess": "0x1410",
		"TargetImage": "C:\\Windows\\System32\\lsass.exe"
	},
	"technique": "T1003.001",
	"note": "rule keyed on tool name; event carries the access mask"
}

When the runner can say the rule matched a SourceImage ending in mimikatz.exe while the event carried a different SourceImage and a GrantedAccess of 0x1410, the agent has the whole diagnosis in two lines and the diff writes itself. Compare that with dropping the full event and hoping the model finds the discrepancy; the structured version is cheaper and steadier at the same time.

Ask for a structured answer back. The agent should return a unified diff against the Sigma source, a one-paragraph rationale a human can check, and its own classification of which of the four outcomes it believes it is looking at. That classification is a cheap sanity gate: if the agent proposes a rule edit while claiming a collection gap, the two disagree, and a person should look before anything merges.

Redact before you send. Real telemetry from a lab that mirrors production carries hostnames, usernames, and internal paths, and a payload bound for a model should be scrubbed of anything that is not needed for the diagnosis. The field-level diff makes this easy, because you are already selecting a handful of fields rather than shipping the raw record, and the selection step is the natural place to drop the sensitive ones.

Keeping the payload in the normalized schema, rather than raw vendor format, is what lets one refinement path serve every platform. The rule is written once in Sigma, the telemetry is normalized to ECS or OCSF once on ingestion, and the agent reasons over one representation instead of learning the quirks of every product’s event format. That single choice, made back at the lab-parity stage, pays off again here, because the agent never has to translate between a Splunk field name and a Defender one to understand what went wrong.

Drawing the automation boundary

That division of labour is easier to hold to when it is written down as a table the runner enforces in code, not a principle everyone remembers differently.

Task in the loop	Who owns it	Why
Running the emulation	Automation	Mechanical, repeatable, reversible
Polling for telemetry	Automation	A timing problem, not a judgement
Checking whether the alert fired	Automation	Binary and unambiguous
Diffing telemetry against the rule	Automation, as a draft	Bounded and evidence-rich
Choosing which techniques matter	Human	Depends on your architecture and threat model
Judging false-positive blast radius	Human	Needs to know what normal looks like in your estate
Approving a rule change	Human	Every diff merges through review
Retiring stale emulations	Human	Requires reading current threat reporting

The boundary is more than documentation; encode it in permissions. The runner holds read access to alerts and execute access to the reviewed technique catalogue, and nothing else. It cannot write to rules, disable detections, or add techniques. The agent’s refinement output arrives as a pull request against the detection repository, which means it inherits the same review, the same fast-tier checks, and the same human approval as a change an engineer typed by hand. A proposed diff that a person never looks at should never reach production, and the merge gate is what guarantees that.

Two rows deserve emphasis. Choosing what to detect stays human because a model asked to prioritise will return a generic ranking drawn from its training data, confident and wrong for your environment. Judging false-positive impact stays human because a rule can catch the emulated attack perfectly and still fire two hundred times a day on a backup agent your operations team runs on a timer. The loop tells you the rule works against the attack; a person tells you the rule is safe to deploy.

A rule that fails, then passes

A single worked example shows what that failing payload actually buys, and how the agent turns it into a diff. Take a rule for LSASS credential access, technique T1003.001. A first draft keys on the tool that made the memory read.

title: LSASS memory access
status: experimental
logsource:
  product: windows
  category: process_access
detection:
  selection:
    TargetImage|endswith: '\lsass.exe'
    SourceImage|endswith: '\mimikatz.exe'
  condition: selection

The emulation runs the same technique with a renamed binary, and no alert fires. The failing result carries the Sysmon process-access telemetry the run produced, and that telemetry shows a GrantedAccess value of 0x1410 against lsass.exe from a SourceImage that has nothing to do with the tool name the rule expected. The diagnosis is immediate: the rule matched an artifact of one procedure, the binary name, instead of the behaviour, the access mask. The agent’s proposed diff keys on the mask and filters the handful of legitimate readers.

detection:
  selection:
    TargetImage|endswith: '\lsass.exe'
    GrantedAccess:
      - '0x1010'
      - '0x1410'
      - '0x1438'
  filter_legitimate:
    SourceImage|endswith:
      - '\wmiprvse.exe'
      - '\taskmgr.exe'
  condition: selection and not filter_legitimate

pySigma converts the corrected rule for each platform the team runs, so the same source ships to Defender and Splunk without a second author.

DeviceEvents
| where ActionType == "OpenProcessApiCall"
| where FileName =~ "lsass.exe"
| where AdditionalFields has_any ("0x1010", "0x1410", "0x1438")
| where InitiatingProcessFileName !in~ ("wmiprvse.exe", "taskmgr.exe")

index=sysmon EventCode=10 TargetImage="*\\lsass.exe"
  (GrantedAccess="0x1010" OR GrantedAccess="0x1410" OR GrantedAccess="0x1438")
  SourceImage!="*\\wmiprvse.exe" SourceImage!="*\\taskmgr.exe"
| stats count by SourceImage, GrantedAccess, Computer

The runner that produced the failing payload is small, and most of its length is timing and evidence handling. The polling loop and the cleanup in finally are the two parts worth copying exactly.

from dataclasses import dataclass, field
from datetime import datetime, timedelta, timezone
import time


@dataclass
class Result:
    ok: bool
    technique: str
    evidence: list = field(default_factory=list)
    telemetry: list = field(default_factory=list)
    rule: dict | None = None


def now_utc() -> datetime:
    return datetime.now(timezone.utc)


def collect(host: str, since: datetime, timeout: timedelta) -> list:
    """Poll for telemetry until it lands or the timeout expires."""
    deadline = now_utc() + timeout
    while now_utc() < deadline:
        events = fetch_events(host=host, since=since)
        if events:
            return events
        time.sleep(15)                      # ingestion is slow; poll, never sleep once
    return []


def validate(technique_id: str, rule_id: str) -> Result:
    started = now_utc()
    run = emulate(technique_id)             # detonate, capture host + time
    try:
        telemetry = collect(
            host=run.host,
            since=started,
            timeout=timedelta(minutes=10),
        )
        alerts = query_alerts(rule_id=rule_id, since=started, host=run.host)
        if alerts:
            return Result(ok=True, technique=technique_id, evidence=alerts)
        return Result(
            ok=False,
            technique=technique_id,
            telemetry=telemetry,            # what the attack actually made
            rule=get_rule(rule_id),         # what we were looking for
        )
    finally:
        run.cleanup()                       # runs even if verification throws

When the alert still refuses to fire after a change, a fixed triage order keeps diagnosis from turning into guesswork.

The corrected rule is not finished when it passes once. Validate it against two more procedures, a direct-syscall reader and a tool that opens the handle with a different mask, before recording it as validated. A rule that passes one procedure has earned evidence about that procedure and no more.

Reading the results honestly

A validation programme produces four outcomes, and conflating them is how teams end up with dashboards that overstate their position.

A rule that fires on the emulated technique is the good case, with a caveat: it proves the rule catches this implementation of the technique. Attackers have many implementations, and a rule matching a specific command line will miss a variant that achieves the same thing differently. Validation against one procedure is evidence about that procedure rather than about the technique as a whole, and reporting it as technique coverage is the most common overstatement in this discipline.

A rule that does not fire is useful and unambiguous, which is the entire value of the exercise. The telemetry either lacked the necessary data, meaning a collection problem, or contained it and the rule looked in the wrong place, meaning a rule problem. Those are different fixes and the raw telemetry distinguishes them immediately.

A rule that fires on something unrelated during the run is a finding worth investigating rather than a pass. It usually means the rule is broader than intended, which is the same thing as saying it will be noisy in production.

Telemetry that never arrived is the fourth case and it is a data pipeline incident, not a detection failure. It is also the most valuable early finding, because a silent collection gap invalidates every rule that depends on that source, and nobody was monitoring for it.

Track these separately. A single “coverage” percentage that mixes validated rules with rules that merely exist tells you less than the raw count of each category.

The four outcomes, at a glance

Those four cases fit in a table you can pin next to the dashboard, so nobody collapses them back into a single number.

Outcome	Alert	Telemetry	What it means	Who fixes it
Detection on the technique	fires	present	The rule catches this procedure	Detection engineer varies the procedure
Silent with data	none	present	Rule logic wrong or field mismatched	Detection engineer
Silent with no data	none	missing	Collection gap	Platform or pipeline team
Fires on the unrelated	fires (wrong rule)	present	Rule broader than intended	Detection engineer tightens scope

The middle two rows look similar on a status board and could not be further apart in cause. Silent with data is a rule problem you fix by editing the query; silent with no data is a pipeline incident that no rule change will ever solve. The raw telemetry in the failing payload separates them in a glance, which is the argument for carrying that payload rather than a pass or fail flag.

The sneaky row is the last one. A run that trips some other rule reads as activity on the dashboard and gets mistaken for a pass, when it actually signals that a rule is broader than its author believed and will be noisy in production. Encode the outcome as an enumerated status on every run, not a boolean, so the pipeline can route a collection gap to the platform team and a logic error to the detection engineer without a human triaging each one. Each of the four has a different owner and a different fix, and a green tile that hides which one you got is worse than an honest red.

Keeping the corpus honest over time

The failure mode of any test suite is drift, and detection validation has a specific version of it worth guarding against.

Rules get tuned until they pass. That is the point, and it becomes a problem when the tuning fits the emulation rather than the behaviour. A rule adjusted to match the exact command line your simulation generates will pass forever and catch nothing else. The guard is to vary the emulation deliberately: implement the same technique two or three ways, and require the rule to catch all of them before considering it validated.

Emulations also decay. A simulation written against one operating system version stops being representative when the fleet moves on, and a technique that relied on a now-patched behaviour becomes untestable. Review the corpus periodically against what current threat reporting describes, and retire simulations that no longer reflect anything real rather than keeping them for the passing count.

Finally, resist letting the agent own both sides. A system where the same automation writes the emulation, writes the rule, and judges whether the rule caught the emulation will converge on a self-consistent pair that proves nothing. Keep the emulation corpus under human review, keep rule changes behind human approval, and let the automation do the running and the diagnosis in between.

Purple teaming, made continuous

Traditional purple team exercises put red and blue in a room for a week, run a scenario, and write findings that decay the moment the environment changes. The loop described here is the same activity stripped of the calendar, run every night against a fixed corpus. Framing it that way clarifies what the automation takes over and what it leaves alone.

What it takes over is regression. A manual purple team exercise proved a set of detections worked on the day it ran; the scheduled loop proves they still work this week, and it catches the silent breakage, a renamed field, a forwarder that stopped, a rule scoped to a host group that got decommissioned, within days rather than at the next exercise a year out. Regression checking is repetitive, evidence-driven, and dull, which is the profile that suits automation and exhausts people.

What it leaves alone is discovery. A human red teamer invents a procedure the catalogue has never seen, chains three techniques in an order nobody modelled, or notices that a detection folds to a timing trick no YAML file encodes. That creativity is the reason to keep running live exercises, and the loop makes them more valuable by clearing the regression work off the table so the humans spend their week on the novel attack rather than re-checking last year’s rules.

The two feed each other. A live exercise that finds a gap should end by adding the procedure to the emulation catalogue, so the loop guards that ground from then on. Every manual finding becomes a permanent regression test, and the catalogue grows into an institutional memory of every attack the team has ever reasoned about. Over a few years that corpus is worth more than any single tool, because it records what your organisation has learned to care about, in a form a machine can re-run on demand.

There is a governance dividend too. Auditors and frameworks such as NIST CSF, ISO 27001, and DORA increasingly ask for evidence of continuous control validation rather than an annual test. A scheduled loop that records, per rule, when it last proved itself against a named technique produces exactly that evidence as a byproduct, with timestamps and telemetry attached. The compliance artifact falls out of doing the engineering well, which is the rare case where the audit trail and the useful work are the same activity rather than two competing claims on the team’s time.

Common mistakes and anti-patterns

Some of the ways this goes wrong are worth naming directly, because each one produces a program that looks healthy while proving less than it claims.

• Tuning to the emulation. A rule adjusted until it matches the exact command line your simulation emits will pass forever and catch nothing else. The tell is a rule whose match string mirrors a hardcoded argument in the emulation YAML. Vary the procedure and require the rule to catch every variant before you call it validated.
• Letting the agent own both sides. If the same automation writes the emulation, writes the rule, and judges the match, it converges on a self-consistent pair that proves nothing about a real attacker. Keep the technique catalogue under human review and rule merges behind human approval.
• Running it in production. Detonating credential-dumping tools against live hosts to test a rule is how a validation program becomes an outage. Scope every run to the isolated lab, and gate the runner so it refuses to execute against production asset tags.
• Skipping cleanup verification. A cleanup step that fails silently leaves accounts, keys, and files behind, and the lab drifts further from production with every run. Verify cleanup succeeded, and revert to a snapshot when it did not.
• Reading a green board as coverage. A grid full of green counts rules that passed against one procedure, not techniques an attacker cannot get past. The board measures what you tested, and it says nothing about the variants you did not.
• Validating one procedure and reporting the technique. This is the most common overstatement in the discipline. One passing procedure is evidence about that procedure; claim the technique only after several implementations pass.
• Sleeping instead of polling. A fixed sleep before checking for the alert reports false failures whenever ingestion runs slow. Poll with a generous timeout so the answer reflects the rule, not the queue depth.
• Giving the runner write access to rules. An automation that can both fail a detection and edit the detection has an obvious degenerate solution available. Read on alerts, execute on the catalogue, and write on nothing.
• Letting the lab drift. Config parity decays the moment the fleet updates and the lab does not. Pin the lab to the same configuration source as production endpoints and re-image on a schedule.
• No cap on refinement. An agent allowed to iterate without limit will overfit the rule to the run and burn budget doing it. Cap iterations, and send anything still failing to a human instead of a fourth automated attempt.

What to tell leadership instead of a coverage number

The honesty that separates the four outcomes should also govern what leaves the team as a metric. A single coverage percentage counts rules that exist, mixes validated and unvalidated, and reassures a board into believing the estate is safer than the evidence supports. Better numbers answer questions leadership actually has.

Instead of	Report	It answers
Detection coverage percentage	Share of production rules with a passing validation in the last 30 days	Are our rules proven, or merely present
Number of rules deployed	Median time from a rule change to a validated rule	How fast does a new detection become trustworthy
Alerts generated this month	Collection gaps found and their mean time to close	Are we blind somewhere we did not know about
Techniques on the matrix	Procedures validated per technique	How deep the coverage runs, not only how wide
A static grid	Trend of rules failing validation, week over week	Is the estate getting more reliable or less

Freshness is the headline. A rule validated eighteen months ago against a field a vendor has since renamed is an unvalidated rule wearing a green badge, so the metric that matters is how many detections have passed recently, not how many have ever passed. Pair it with the count of collection gaps found, because every silent gap invalidated a set of rules nobody was watching, and finding them early is the clearest return this program produces.

Keep one slide that leadership will remember: the list of load-bearing but unvalidated detections, the rules the organisation is relying on with no evidence behind them. That list shrinks as the program runs, and its length over time is a more honest picture of progress than any percentage. The MITRE ATT&CK Evaluations set the precedent here by refusing to publish a single winner and reporting detection categories such as Technique, Tactic, Telemetry, and None instead. The reason is the same one that argues against a coverage number: collapsing distinct outcomes into one score destroys the information a decision needs.

Running this across hundreds of rules

Everything so far assumes ten rules; the shape changes once the corpus reaches the hundreds, and both runtime and cost need managing.

Deduplicate by technique first. Many rules read the same telemetry, so four hundred rules often map to fewer than two hundred unique techniques. Emulate each technique once, capture the run’s telemetry, and evaluate every rule that maps to it against that single run. A fan-out from one detonation to a dozen rule checks is the difference between a nightly job that finishes and one that never does.

Run the emulations in parallel on ephemeral lab hosts, one per technique batch, torn down after cleanup verification. Cache each run’s telemetry keyed by technique and timestamp, so re-checking a rule after a fix reads the stored events instead of detonating again. Then stagger the schedule by importance rather than treating every rule as nightly.

Tier	What it holds	Cadence	Why
Crown jewels	Credential access, persistence, and whatever the last incident involved	Nightly	Highest cost if silent
Standard	The bulk of the corpus	Weekly	Catches drift within a sprint
Long tail	Rare or low-severity techniques	Monthly	Keeps the count honest without burning the lab

Cost lands almost entirely on the refinement stage, so bound it deliberately. Only failures call the agent, which means a corpus that mostly passes barely touches the model budget. Cap each failing rule at three refinement iterations, route the cheap classification step, deciding which of the four outcomes you got, to a small fast model, and reserve a stronger model for drafting the actual diff. Set a per-run token budget and stop when it is spent rather than letting a stubborn rule consume the night.

The arithmetic stays friendly. If four hundred rules collapse to a hundred and eighty techniques, and a fifth of the runs fail, you detonate a hundred and eighty times and invoke the agent roughly thirty-six times, not four hundred. That is a job that fits in a scheduled window on modest lab capacity, and its cost tracks the number of broken rules rather than the size of the corpus, which is exactly the property you want as the corpus grows.

Watch the platform side of the bill as well. Every verification queries the SIEM, and a hundred and eighty verification queries fired in a tight window can trip API rate limits or spike a consumption-based search bill on their own, before a single agent call. Space the queries, batch alert lookups by time window where the platform allows it, and prefer a single query that returns alerts for many rules over a query per rule. The same fan-out that saves detonations saves search quota, because one run’s telemetry answers a dozen rule checks with one pull. Treat both the lab and the SIEM as shared capacity the loop borrows on a schedule, and the program stays cheap enough that nobody argues to switch it off.

Starting small

The version of this worth building first covers ten techniques rather than the full matrix.

Pick the techniques you would be most embarrassed to miss, which for most organisations means credential access, persistence mechanisms, and whatever the last incident involved. Validate the existing rules for those manually, once, to establish the pipeline works and to calibrate the ingestion timeout. Then automate the run and schedule it weekly.

The first execution is usually uncomfortable. Teams commonly find that a third of tested rules do not fire, and the reasons are mundane: a renamed field, a log source that stopped forwarding, a rule scoped to a host group that no longer exists. Finding three broken rules out of ten justifies the project on its own, before any agent has proposed a single fix.

From there the value compounds, because the check runs on schedule and catches the next silent breakage within a week rather than during the next incident. Detection engineering has always suffered from having no test suite. This is the closest the discipline has come to one, and the agent is the component that makes writing the tests cheap enough to be worth doing.