Quality is what teams say keeps their agents out of production. LangChain’s State of Agent Engineering report, built from 1,340 survey responses collected over two weeks in late 2025, puts quality first at 32 percent of respondents, ahead of latency at 20 percent. The same report finds evals in place at 52 percent of teams while observability sits near 89 percent. The thing people name as the blocker is the thing they measure least.
Stanford’s 2026 AI Index supplies the other half, and it is worth being precise about what it does and does not say. It puts no number on the drop from benchmark score to deployed behavior. It makes the point in words instead: even when benchmark scores are technically valid, strong benchmark performance does not always translate to real-world utility. On structured benchmarks it finds agents still failing roughly one attempt in three. And the benchmarks are themselves shakier than their scores suggest. Truong et al. count 88 of GSM8K’s 997 questions as invalid: ambiguous wording, answer keys that are simply wrong, and a grader that takes the last number in a response and calls it the answer.
So the benchmarks measure what they were designed to measure, and not much past it. The rubrics were never designed to catch the failure modes that get agents pulled from production.
A rubric that produces a single accuracy number tells you the average of your failure modes. It does not tell you which failure modes you have, or how much each one is costing you. For an agent that runs in production, that distinction is the entire game.
The pattern
Failure buckets are the rubric. The aggregate score is downstream of it. Chapter 6 of the book frames a working eval harness around three architectural moves.
- Define the buckets before you write the eval. Not after. A failure bucket is a category of mistake with three properties: a concrete example, a detection rule that does not require human review, and a severity tier. If you cannot write all three, the bucket is not ready for the rubric.
- Score by bucket distribution, not by aggregate accuracy. A system at 0.85 accuracy with failures distributed evenly across eight buckets is a different system from one at 0.85 with all the failures concentrated in hallucination and cost runaway. The aggregate hides the difference. The distribution exposes it.
- Re-bucket on every model upgrade. Buckets ossify. New models surface failure modes that did not exist on the prior model, and silence ones that did. If your bucket list has not changed in six months and you have changed models, the rubric is measuring a system you no longer run.
Same aggregate. Different consequences. The agent on the right is unshippable, and the aggregate hides it.
A working starting set of buckets, pulled from the book and refined across three deployments:
- Hallucination. Fabrication of facts, citations, or tool capabilities. Severity tier 1.
- Tool selection error. Wrong tool for the task. Tier 1 if the tool has side effects, tier 2 otherwise.
- Tool invocation error. Right tool, malformed arguments. Tier 2.
- Scope creep. Answering more than the task asked for. Tier 2.
- Refusal. Will not do something it should. Tier 2.
- Cost runaway. Right answer, infeasible spend. Tier 1.
- Latency violation. Right answer, too slow for the operating envelope. Tier 2.
- Format violation. Right answer, wrong shape. Tier 3.
Eight buckets is a starting point, not a target. A clinical agent needs an “unsafe recommendation” bucket. A code agent needs a “fabricated symbol” bucket. A retrieval agent needs a “stale citation” bucket. The general principle is the same. Name the failures. Write the detection rules. Score the distribution.

The dial reads “average.” Each room is failing in a different direction.
Where this fails
Failure buckets work cleanly for tasks with a bounded failure space. Open-ended generative work resists tidy bucketing because the failure modes are emergent. A research-synthesis agent can fail in ways the rubric designer did not anticipate. The honest answer is to ship the rubric anyway, mark a residual “other” bucket, and graduate failures out of “other” as patterns emerge. Buckets are a living artifact, not a one-time spec.
The pattern also fails when teams confuse precision for value. A rubric with thirty buckets and one example per bucket has lower signal than a rubric with eight buckets and twenty examples each. Bucket inflation looks rigorous and runs shallow. The discipline is fewer buckets, more examples.
The third failure mode is more political than technical. Failure buckets reveal which failure mode dominates. Sometimes the dominant bucket is the one a senior stakeholder shipped, or the one their roadmap depends on. The rubric will land harder than the aggregate score did. That is a feature, not a bug. It changes the conversation from “is the agent good enough” to “good enough at what.”
In code
The harness this pattern runs on is in src/ch06/ in the companion repo:
eval_harness.py—Rubric,FailureCategory, and anEvalReportthat carries afailure_distributionnext to the aggregate scorecost_profiler.py— the cost and latency measurements the cost runaway and latency buckets are scored againstreliability.py— repeat-run checks, for buckets that only show up intermittentlytracer.py— structured traces, which is where a detection rule gets the evidence to fire on
The rubric for the document-intelligence agent is checked in at project/doc-intelligence-agent/evals/rubric.yaml as weighted criteria with a pass threshold. There is no command-line entry point yet, so tests/unit/test_eval_harness.py is the shortest way to see the harness run end to end. The categories in FailureCategory are the document-intelligence set rather than the eight general buckets above; the eight are the starting list, not the code.
An aggregate score is the average of the failure modes you did not catalog.