The problem
Ask a team why their guardrail pipeline is slow and the answer is usually the same: the expensive detector, an LLM judge weighing a policy against the actual intent of a call, runs on every request. It is also the design ARCH-008 rules out early: detection is added latency on top of retrieval and inference, and a judge costing a few hundred milliseconds, run on every call, consumes a share of the budget nothing else can spare.
The instinctive fix is to make the judge faster: a smaller model, a shorter prompt, more aggressive caching. All of that helps a little and misses the real question, which is not how fast the judge runs but how often it has to. Most calls a coding assistant makes are reads. Most requests through a retrieval pipeline carry nothing adversarial. Running the same expensive check on all of it treats every call as equally worth the judge’s attention, when the calls that actually need it are a minority the cheap, deterministic checks can already point at.
The other instinctive fix, dropping the judge and keeping only the cheap checks, trades the problem for a worse one. The judge exists because some cases are genuinely ambiguous to a rule: a registry lookup, an authority check, an argument scan, none of which reason about intent. Somewhere between run it on everything and never run it at all is the real design question, and it is a question about which calls reach the judge, not how fast any one of them goes once they do.
Forces
Only the expensive detector reliably resolves the cases a rule cannot, but running it on every request spends a latency budget nothing else in the pipeline can spare. A judge that only ever sees the easy cases is not doing the work it exists for. A judge that sees everything is a judge nobody can afford to wait on. Both failure modes are real, and a pipeline has to avoid both at once, not pick one.
The width of the escalation band is a choice with no natural resting point. Narrow it and the judge answers fewer calls, some of which needed it. Widen it and the fraction of traffic reaching the judge grows, and with it the exact cost tiering was meant to avoid. Nothing about the traffic tells you where to draw that line on its own; it is set, and has to be defended, by whoever owns the budget.
Percentiles do not amortise the way averages do. A judge that runs on a tenth of traffic costs a tenth as much, on average. It does not cost a tenth as much at the tail, because the calls that escalate are, by construction, the slower ones, and a slow minority does not spread itself evenly across a sorted distribution. It sits at one end of it. How large that minority is decides which percentile it occupies. How fast the cheap tier ran decides nothing about that.
The pattern
Run the cheap, deterministic detectors, a registry lookup, an authority check, an argument scan, on every request. Reserve the expensive detector, the one that actually reasons about intent, for the band where the cheap tier cannot resolve the call on its own: the near-threshold, ambiguous middle. Everything outside that band gets a verdict from the cheap tier alone and never reaches the judge at all.
The mechanism is not new, and saying so plainly is the point of calling this pattern restated rather than proposed. Viola and Jones described the same shape in 2001, for face detection rather than agent guardrails: a cascade of classifiers ordered from cheapest to most expensive, where each stage rejects a majority of the negative windows still in play and passes only the harder remainder to the next, costlier stage. No single stage does the whole job; the cumulative effect across stages leaves only a small fraction of candidates for the expensive final stages. Carrying that cascade into a detector pipeline for tool calls and prompts is a naming act, not a discovery: the underlying arithmetic, reject cheaply and early, spend the expensive step only on what survives, is the same.
That arithmetic is where teams get it wrong, and it is worth making mechanical rather than hand-waved. Call f the fraction of requests that escalate to the expensive tier. If every escalated request is slower than every request the cheap tier resolved alone, which is the whole point of only escalating the ambiguous cases, then the escalated calls occupy the slow end of the sorted latency distribution, and the size of that slow end is exactly f.
# constructed illustration, ARCH-008's own escalation fractions, not measured traffic.
# "marginal" at f = 0.05 because the exact cut depends on the percentile
# method (nearest-rank vs. interpolated); ARCH-008 itself calls this "the boundary".
escalation_fractions:
- f: 0.10 # one call in ten
sets_p95: true # the judge is your p95, however fast the cheap tier is
sets_p99: true
- f: 0.05 # one call in twenty, ARCH-008's own stated boundary
sets_p95: marginal
sets_p99: true
- f: 0.02 # one call in fifty
sets_p95: false # the cheap tier still sets p95
sets_p99: true # two percent still exceeds the one percent p99 allowsEscalate a tenth of traffic and the judge is your p95, whatever the cheap tier’s own speed. Escalate one call in fifty and the cheap tier still sets p95, but the judge has already claimed p99, because two percent exceeds the one percent that position allows. Reporting p95 alone, in a tiered pipeline, hides whichever tier actually decided it. The number worth watching is not how fast the judge runs. It is what fraction of calls reach it, measured against the percentile the budget actually promised.
Worked example
ARCH-008 works this exact arithmetic through a coding assistant that reads a repository, proposes changes, and calls a small set of tools. The chokepoint is tool execution, the last agent-side chokepoint that can still stop an action before it happens. The budget, a constructed target rather than a measurement, not a report of a specific deployment: 120ms p95 for the whole policy decision, generous because most of the pipeline is deterministic rather than inference.
The cheap tier runs on every call: a tool registry lookup at 5ms, an authority check against actor, action and resource at 20ms, an argument scan for secrets and paths outside the workspace at 15ms, each answering an authorization question rather than a classification one. The judge, at 400ms, runs only on calls flagged reversal_cost: impossible that the cheap tier also places in the near band: an assistant proposing a force-push or a database write it cannot undo, where the cheap checks alone cannot tell whether this specific call is one the policy permits.
Read the arithmetic against that pipeline. ARCH-008 frames this assistant’s traffic as dominated by reads, so the share of calls that are both irreversible and ambiguous is the fraction to watch. If that share sits under roughly five percent of tool traffic, the deterministic checks set the p95 and the judge shows up only once you look at p99. That is tiering working as intended: the expensive part of the design is also the smallest part of it.
It stops working the moment the traffic mix changes and nobody re-measures. Extend the same assistant to propose deploys or direct database writes, and the fraction of calls landing in the irreversible, ambiguous band can climb well past five percent without any change to the judge itself. At that point the fix is not a faster judge. ARCH-008 names two real ones: route the ambiguous, irreversible band to require_approval instead of waiting on the judge synchronously, which turns a latency problem into a human queue, or narrow the band so fewer calls qualify as ambiguous in the first place. What does not work is treating the judge’s own latency as the lever, because the lever was never there. The escalation fraction is.
When not to use it
Tiering earns its cost only when the cheap tier is actually a proxy for what the expensive one would decide. In a genuinely novel or adversarial domain, one where nothing simple, a registry entry, an argument shape, a keyword, correlates with the judge’s eventual verdict, the cheap tier cannot narrow anything. Every call looks equally ambiguous to it, so every call escalates anyway, and tiering has added a hop without moving the fraction that matters. The deterministic tier is only worth building once it demonstrably narrows what reaches the judge, and that has to be measured before the machinery is built, not after.
A single action whose reversal cost is high enough should not be triaged at all. The pattern’s whole bet is that the cheap tier’s confident verdict, this call is not ambiguous, is one you can trust without a second opinion. That bet holds for the overwhelming majority of calls a system makes, but it is the wrong bet for the rare call where a missed escalation is the entire incident: a payment above a threshold that makes little sense to reverse, a production database migration, an approval that cannot be undone once the tool fires. For that class of call, run the expensive check every time and treat the extra latency as the cost of the decision, not as a design defect. Deciding which calls qualify for that exemption belongs to whoever owns the threshold, not to whoever is trying to hit a budget.
And the escalation fraction itself needs enough traffic to mean anything. A tool called only a handful of times a day does not have a stable f; a small sample can look like one fraction this week and a different one next week for reasons that have nothing to do with risk. Run the judge on everything until the volume exists to measure f honestly, then tier once the fraction is a real property of the traffic rather than a guess dressed up as one.
Related patterns
Chokepoint Placement answers a prior question: which stage carries a control at all. Tiered detection starts once that stage is chosen, deciding within it which calls get the expensive check and which get the cheap one, the shape ARCH-008’s own coding-assistant judge shows, seeing only the ambiguous band tool execution hands it. Verdict Composition picks up once a tier has more than one verdict to reconcile: tiering decides who reaches the judge, composition decides how the cheap tier’s checks and the judge’s own verdict combine into one decision, without averaging a rule’s pass or fail against a probability the judge assigned. Fail Closed, Degrade on Exposure governs what happens when the tier an escalated call needs is itself unavailable: a call that cannot reach the judge does not quietly fall back to the cheap tier’s verdict, it fails closed or waits on a human, because a timed-out escalation is not a cleared one. Shadow Before Enforce is how the escalation band earns the right to gate anything: run it recording what it would have sent to the judge, before it is allowed to actually route traffic there. Baseline and Floor keeps that boundary from drifting once it is live, catching the case where the escalation fraction creeps upward release over release until the judge is quietly back to being everyone’s p95.