Back to blog
August 21, 2026·6 min read·Metronix Team

Agent Failure Detection: Checks Beat Monitors

A deterministic recompute-and-coverage check caught agent failures at zero false positives where a learned monitor hit 17%. What to ship first.

Your agent's failure detector doesn't need to be a model

If your agent reports a number, recompute that number from the tool results the run actually received. If the run was supposed to make three calls, confirm it made three. Those two checks — no model, no threshold, no calibration — outperform the learned monitor and the LLM judge that most teams reach for first.

That's the uncomfortable finding when you put all three approaches on the same episodes with the same labels. And it changes what you should build first.

The three layers teams actually consider

There are broadly three ways to catch an agent going wrong mid-run.

LLM-as-judge. A second model reads each step and rules on it. Expensive — you're paying roughly a forward pass per step — but assumed to be accurate.

Learned anomaly monitors. Train a one-class model on healthy runs only, score each step's telemetry, alarm when the score drifts. Cheap at inference. No failure labels required, which is the main selling point.

Deterministic verification. Recompute the stated answer from the inputs the run received. Confirm required calls happened. Validate that each tool result matches the shape that tool can return.

Most architecture diagrams I've reviewed put layer one or two at the center and skip layer three entirely. The measured numbers argue for the reverse ordering.

What the head-to-head actually showed

Across 2,823 committed agent episodes over 25 datasets — three frameworks (bespoke, LangGraph, AutoGen), qwen2.5 7b/3b, llama3.1:8b, and gemini-2.5-flash — scored on identical episodes and identical labels:

Deterministic checks. On a frozen held-out arm of 120 episodes at disjoint task seeds: 54% of failures caught by the totals check alone, 93% with the coverage check added, at 0 false positives out of 64. Arithmetic errors specifically: 36 of 36.

The learned monitor. Same episodes: 54% caught at 17% false positives (11 of 63).

The judge. A real gemini-2.5-flash judge measured over 161 distinct prompts detected 54.8% of failures (95% CI 0.44–0.65) at a 5.2% false-positive rate. The escalation design it was feeding had assumed 0.90 detection. Correcting the assumption dropped claimed detection recovery from roughly 82% to 43%.

That last number is the one worth pausing on. A great many agent reliability architectures are built on an unmeasured premise that the judge is nearly perfect. When someone measures it, the judge lands near a coin flip on failures.

Transfer is where the gap becomes structural

Recall parity is interesting. Transfer behavior is decisive.

Calibrate the one-class monitor on qwen2.5:7b, then deploy it on llama3.1:8b over a matched task, tool and injector plan without recalibrating. It sits at chance — AUROC 0.527, with a healthy false-alarm rate of 0.75. Recalibrated on itself, the same target reaches 0.885. So the monitor isn't wrong about the target; it's coupled to the serving distribution. Change temperature, toolset, probe pattern or latency regime and the healthy null you spent time collecting is stale.

The deterministic checks were run on the same 120 task seeds with a llama3.1:8b arm and nothing retuned: 110 of 110 failures caught at 0 of 10 false positives.

A third check — validating tool results against the shapes their tool can return — tripped on 0 of 1,825 healthy episodes while flagging 46% of injected context corruption, and 215 of 218 of those within one step of onset.

Checks that need no null, no threshold and no calibration are portable by construction. Monitors that need all three are an operational commitment you re-pay on every deployment change.

The repair result nobody expects

Detection only matters if it closes into a fix. The interesting part is what you tell the agent when a check trips.

Roll the run back to its last fact-gathering step and re-run it live. Then vary the hint:

  • Say nothing, just resample: 16% recovery (the control).
  • Name which check failed, no values attached: 45% recovery (p=0.0005), 6.4 model calls per recovery.
  • Hand over the actual finding, values included: 36% recovery, 8.1 calls per recovery.

Handing the agent more information recovered less. And 26 of those 55 "specific" hints contained the correct total outright. Give a model the conclusion and it stops re-deriving; point at the failing step and it re-does the work. Net over 120 episodes, the "located" policy lifted task success from 52% to 73% and broke zero already-correct runs, for about one extra model call per run.

What this means for your teams

Engineering. Inventory your agent tasks by whether their output is independently recomputable. Every task that terminates in a number, a count, a diff or a set of records can carry a verifier written in an afternoon. Ship those before you evaluate a monitoring vendor. Then instrument the coverage check — did every required call actually happen — because that's where most of the recall lives.

Security. A tool-contract check that trips on 0 of 1,825 healthy runs is a precision profile you can actually alert on. A 17% false-positive monitor is a rotation that stops reading alarms in a fortnight. Precision is a security property, not a nicety.

Product. The repair result reframes what an error message is for. Surfacing "the totals check failed" is more useful to an autonomous retry than surfacing the corrected value. Design your failure payloads to point, not to answer.

My opinion

The industry has spent two years building statistical detectors for a problem where a large slice is deterministic. Not all of it — plausible-value corruption, a wrong-but-well-formed number, is undetectable from telemetry by construction and needs an external reference. Slow goal drift evades every per-step monitor tested. The judge, for its part, is perfect on goal drift (21 of 21) and nearly blind on context corruption (0.18), where the monitor is the reverse. These layers are complementary, not competing.

But the ordering matters, and most teams have it backwards. Write the check. Measure your judge before you build an escalation policy on top of it. Treat the learned monitor as the layer you add for the classes the checks structurally can't see, not as the foundation.

The honest boundary: this evidence comes from mock-tool and research-loop tasks with recomputable answers, from a single-author preprint, and the 60%/96% figure often quoted is the arm the checks were written on — 54%/93% is the frozen held-out number. Treat it as a strong prior, not a settled result.

At MTRNIX we build the memory and context substrate agents run on, and this shapes how we think about it: the more of an agent's inputs are addressable and replayable, the more of its output becomes recomputable — which is exactly what makes cheap verification possible in the first place. Verification is downstream of context you can actually retrieve.

If you're auditing your agent stack for where reliability actually comes from, we're happy to compare notes.

References

  • Sunny Dubey, "Real-Time Detection and Repair of LLM Agent Failures," arXiv:2608.02464v1 [cs.AI], 3 Aug 2026 — https://arxiv.org/abs/2608.02464
  • Code and traces: https://github.com/sunnydubey1111/agent-trajectory-sentinel
  • Cemri et al., "Why Do Multi-Agent LLM Systems Fail?" (MAST taxonomy) — https://arxiv.org/abs/2503.13657