Skip to content
Agent Engineering Lab

FIELD NOTE · FN-002

Your eval rubric needs failure buckets, not just scores

Two agents can score identically on a benchmark and fail differently in production. The aggregate is not a rubric. It is what falls out of one.

buckets of failure, not a single tidy score
buckets of failure, not a single tidy score

Quality is what teams say keeps their agents out of production. LangChain’s State of Agent Engineering report, built from 1,340 survey responses collected over two weeks in late 2025, puts quality first at 32 percent of respondents, ahead of latency at 20 percent. The same report finds evals in place at 52 percent of teams while observability sits near 89 percent. The thing people name as the blocker is the thing they measure least.

Stanford’s 2026 AI Index supplies the other half, and it is worth being precise about what it does and does not say. It puts no number on the drop from benchmark score to deployed behavior. It makes the point in words instead: even when benchmark scores are technically valid, strong benchmark performance does not always translate to real-world utility. On structured benchmarks it finds agents still failing roughly one attempt in three. And the benchmarks are themselves shakier than their scores suggest. Truong et al. count 88 of GSM8K’s 997 questions as invalid: ambiguous wording, answer keys that are simply wrong, and a grader that takes the last number in a response and calls it the answer.

So the benchmarks measure what they were designed to measure, and not much past it. The rubrics were never designed to catch the failure modes that get agents pulled from production.

A rubric that produces a single accuracy number tells you the average of your failure modes. It does not tell you which failure modes you have, or how much each one is costing you. For an agent that runs in production, that distinction is the entire game.

The pattern

Failure buckets are the rubric. The aggregate score is downstream of it. Chapter 6 of the book frames a working eval harness around three architectural moves.

  1. Define the buckets before you write the eval. Not after. A failure bucket is a category of mistake with three properties: a concrete example, a detection rule that does not require human review, and a severity tier. If you cannot write all three, the bucket is not ready for the rubric.
  2. Score by bucket distribution, not by aggregate accuracy. A system at 0.85 accuracy with failures distributed evenly across eight buckets is a different system from one at 0.85 with all the failures concentrated in hallucination and cost runaway. The aggregate hides the difference. The distribution exposes it.
  3. Re-bucket on every model upgrade. Buckets ossify. New models surface failure modes that did not exist on the prior model, and silence ones that did. If your bucket list has not changed in six months and you have changed models, the rubric is measuring a system you no longer run.

Two agents at 0.85 accuracy with different failure bucket distributions, showing why the aggregate hides what matters

Same aggregate. Different consequences. The agent on the right is unshippable, and the aggregate hides it.

A working starting set of buckets, pulled from the book and refined across three deployments:

  • Hallucination. Fabrication of facts, citations, or tool capabilities. Severity tier 1.
  • Tool selection error. Wrong tool for the task. Tier 1 if the tool has side effects, tier 2 otherwise.
  • Tool invocation error. Right tool, malformed arguments. Tier 2.
  • Scope creep. Answering more than the task asked for. Tier 2.
  • Refusal. Will not do something it should. Tier 2.
  • Cost runaway. Right answer, infeasible spend. Tier 1.
  • Latency violation. Right answer, too slow for the operating envelope. Tier 2.
  • Format violation. Right answer, wrong shape. Tier 3.

Eight buckets is a starting point, not a target. A clinical agent needs an “unsafe recommendation” bucket. A code agent needs a “fabricated symbol” bucket. A retrieval agent needs a “stale citation” bucket. The general principle is the same. Name the failures. Write the detection rules. Score the distribution.

Illustration of a building cross-section with a calm thermostat in the central hallway while rooms behind the walls show very different conditions, including an overheated room with a red radiator and a frozen room with a leaking pipe

The dial reads “average.” Each room is failing in a different direction.

Where this fails

Failure buckets work cleanly for tasks with a bounded failure space. Open-ended generative work resists tidy bucketing because the failure modes are emergent. A research-synthesis agent can fail in ways the rubric designer did not anticipate. The honest answer is to ship the rubric anyway, mark a residual “other” bucket, and graduate failures out of “other” as patterns emerge. Buckets are a living artifact, not a one-time spec.

The pattern also fails when teams confuse precision for value. A rubric with thirty buckets and one example per bucket has lower signal than a rubric with eight buckets and twenty examples each. Bucket inflation looks rigorous and runs shallow. The discipline is fewer buckets, more examples.

The third failure mode is more political than technical. Failure buckets reveal which failure mode dominates. Sometimes the dominant bucket is the one a senior stakeholder shipped, or the one their roadmap depends on. The rubric will land harder than the aggregate score did. That is a feature, not a bug. It changes the conversation from “is the agent good enough” to “good enough at what.”

In code

The harness this pattern runs on is in src/ch06/ in the companion repo:

  • eval_harness.py — Rubric, FailureCategory, and an EvalReport that carries a failure_distribution next to the aggregate score
  • cost_profiler.py — the cost and latency measurements the cost runaway and latency buckets are scored against
  • reliability.py — repeat-run checks, for buckets that only show up intermittently
  • tracer.py — structured traces, which is where a detection rule gets the evidence to fire on

The rubric for the document-intelligence agent is checked in at project/doc-intelligence-agent/evals/rubric.yaml as weighted criteria with a pass threshold. There is no command-line entry point yet, so tests/unit/test_eval_harness.py is the shortest way to see the harness run end to end. The categories in FailureCategory are the document-intelligence set rather than the eight general buckets above; the eight are the starting list, not the code.

An aggregate score is the average of the failure modes you did not catalog.

Sources

  1. LangChain (2026) State of Agent Engineering
  2. Stanford HAI (2026) 2026 AI Index Report, Chapter 2: Technical Performance
  3. Truong et al. (2025) Fantastic Bugs and Where to Find Them in AI Benchmarks