Production Execution

Every detector in ARCH-002 is correct in isolation and expensive in aggregate. A pipeline assembled by adding one well-justified control at a time, each approved on its merits, arrives at a system where detection costs more than inference and the product team is asking to turn it off.

This blueprint is the runtime half of the control layer: what the pipeline costs, how it is ordered, and what it does when a component is slow rather than wrong. It is where guardrails stop being a safety topic and become an ordinary distributed systems problem.

Start from the latency budget

The budget is the design input. Decide it before choosing detectors, because it is what makes the choice decidable.

Two properties matter. Detection latency is added latency, on top of retrieval and inference, so users experience it directly. And it lands differently at each chokepoint: work at upload is asynchronous and effectively free, work at the prompt layer is fully on the critical path, and work at the response layer is on the path but can sometimes overlap with streaming.

budget for a synchronous assistant turn      p95 2,500 ms
(illustrative allocation, not a benchmark)

  retrieval                                   250 ms
  inference (streamed)                      1,700 ms
  ─────────────────────────────────────────────────
  remaining for all detection                 550 ms   (22%)

  allocated
    prompt-layer detectors, parallel           120 ms
    response-layer detectors, parallel         180 ms
    tool-call policy decision, per call         40 ms
    reserve for retries and jitter             210 ms

The reserve is the part teams omit. A budget spent to the last millisecond at p95 has no room for the day a dependency is slow, and the failure mode is timeouts across the whole pipeline at once.

Detection you cannot afford at p95 is detection you will turn off during the first incident. Budget it like capacity, not like correctness.

Parallel first, then tiered

Two independent techniques, and the order in which you apply them matters.

Parallel execution is the cheap one, in latency. Detectors at the same chokepoint are independent by construction, per ARCH-002. Run them concurrently and the chokepoint costs the slowest detector rather than the sum. Any pipeline still running detectors in sequence is paying a bill it does not owe.

It is not free in any other currency. The compute, the connections, the rate limit against a managed service and the load on every shared dependency are unchanged, and they now all arrive at once instead of spread out. Two second-order effects follow. Concurrency raises peak resource demand at exactly the moment traffic peaks. And the latency of a maximum over several distributions is worse than any single one of them: with enough parallel detectors, the chance that at least one is in its own tail approaches certainty, so a chokepoint of six detectors has a worse tail than its slowest detector measured alone. Parallelism buys you the mean and costs you a little of the tail.

The composition rule also needs all the verdicts before it decides, so the chokepoint cost is bounded below by your slowest detector. That is what makes a single LLM judge in the parallel set so expensive: it sets the floor for everything.

Tiering is how you afford the expensive one. Rather than running the judge on every request, run the cheap deterministic detectors first and reserve the judge for the band where the cheap ones are ambiguous.

The economics are worth doing properly, because the usual hand-wave is wrong. Amortising cost is straightforward: a judge that runs on a tenth of traffic costs a tenth as much. Amortising latency is not, and this is where budgets get broken.

A percentile is a position in a sorted distribution. If a fraction f of requests escalate to the judge, and every escalated request is slower than every non-escalated one, then the judge shows up in your p95 whenever f is greater than 5 percent. Escalate a tenth of traffic and your p95 is a judge call, no matter how fast the cheap tier is.

  escalation rate f      what sets p95        what sets p99
  ─────────────────────────────────────────────────────────
  f = 10%                the judge            the judge
  f =  5%                the boundary         the judge
  f =  2%                the cheap tier       the judge

Three consequences. The fraction reaching the judge has to be smaller than the tail you are protecting, or the judge is your p95. Reporting p95 alone hides an expensive tier entirely, so report the percentile above your escalation rate too. And if the ambiguous band is genuinely 10 percent of traffic, the answer is not a faster judge; it is to stop waiting on it, either by routing that band to an action that does not need the judge’s answer now, such as require_approval, or by narrowing the band.

Timeout isolation

Every detector gets its own timeout, and no detector’s timeout is the request’s timeout. This sounds obvious and is routinely violated by pipelines that await the whole detector set with one deadline.

Three rules.

Per-detector deadlines, sized to the detector. A regex has a 5ms deadline; a judge has 400ms. A shared deadline means the fast detectors inherit the slow one’s tail.

A pipeline deadline that is less than the remaining request budget. When it expires, the pipeline returns what it has, with the missing detectors marked execution_status: timeout rather than absent. ARCH-002 keeps execution_status separate from decision precisely so this case is representable.

Circuit breakers per detector, not per pipeline. A detector failing 80 percent of calls should be taken out of the parallel set entirely for a cool-down, marked degraded, and reported. Continuing to call it costs the whole pipeline its timeout on every request for no verdicts.

Degradation that does not fail open

ARCH-002 said a timeout is not a verdict. Here is what that means at runtime. The tiers are the ones from ARCH-006, in one line: Tier 3 is read-only, internal, no restricted data; Tier 2 handles restricted data or proposes changes; Tier 1 can write to a system of record or act externally.

The row that decides whether this design is honest is the last one. A system that proceeds normally with its entire detection pipeline down has not degraded gracefully; it has turned the controls off and kept serving.

There is a tempting shortcut here, and it is wrong: that information-harm chokepoints can fail open because only actions are irreversible. You cannot un-disclose. A response that emits credentials, personal data or a restricted document is as irreversible as a payment, and in a regulated setting it is reportable. Reversibility is a property of the specific harm, not of the harm family, which is the same correction ARCH-004 applies to the action grid.

The defensible rule is by exposure, sensitivity, and whether the call writes, never by family or by chokepoint identity. Fail open with a record where the call is read-only, low sensitivity, and the exposure is contained, which is most Tier 3 internal traffic. Fail closed wherever a call writes to a system of record, is externally visible, carries an impossible reversal cost, or runs under high privilege, whatever chokepoint it runs through, including tool execution, which fails closed here for the ordinary reason that most tool calls write or are irreversible, not because its name is a match key on its own. It is the last agent-side point where the intended action, its arguments, and the delegated identity behind the call still coexist, before the agent’s own control decides whether it fires; downstream layers, an API gateway, a service’s own authorization check, database permissions, cloud IAM, can still refuse the call after that. A system whose only fail-closed chokepoint is tool execution has decided, usually without noticing, that disclosure is recoverable.

Two invariants regardless of tier. Every degraded request is marked degraded in its decision record, so the period is reconstructable afterwards. And degradation is time-boxed with an alert, because a system that has been quietly degraded for four hours is one that will describe itself as healthy in the incident review.

Rollout is a config change

The lifecycle in ARCH-006 needs shadow, canary and enforce to be states the running system can be moved between without a deploy. If changing enforcement requires shipping code, then shadow is a branch nobody maintains and rollback during an incident is a release.

# per control, per system: the runtime state machine
control: prompt_injection.retrieval
system: rag-assistant-eu
mode: canary                 # off | shadow | canary | enforce
canary:
  slice: business_unit=markets
  percentage: 100
timeouts:
  cheap_tier_ms: 60
  judge_ms: 400
  pipeline_ms: 500
degradation:
  on_timeout: proceed_marked_degraded
  on_pipeline_down: fail_closed_at_tool_execution
rollback:
  to: shadow
  authorised: [platform.oncall, platform.guardrails]
  requires_approval: false      # deliberately

requires_approval: false on rollback is the important line and the one that will attract governance objections. The argument for it: moving a control from enforce back to shadow reduces user impact and is fully recorded. Requiring an approval to do it means the real rollback during an incident will be someone disabling detection entirely, which is strictly worse and less visible.

Worked example: tool-use control on a coding assistant

A coding assistant that can read a repository, propose changes, and run a small set of tools. The control in question is the tool-execution policy decision from ARCH-004, the last agent-side chokepoint that can prevent action harm.

Budget. Tool calls are not on the streaming path; the user is waiting on a discrete action. Budget: 120ms p95 for the policy decision, which is generous because it is deterministic work.

Pipeline, per tool call:

Reading it from the bottom: the judge only ever sees calls that are both irreversible and ambiguous. In an assistant whose traffic is dominated by reads, that is a small share of calls, and the rule from the previous section applies directly. Measure that share. If it sits below the tail you are budgeting, the deterministic tiers set your p95 and the judge shows up only in p99. If it does not, the judge is your p95 and the design has to change rather than the number being explained away.

Degradation. If the registry or authority check is unavailable, fail closed: no tool call proceeds without an authority decision, because there is no safe default for “may this actor do this”. If only the judge is unavailable, calls with reversal_cost: impossible in the near band route to require_approval instead of proceeding, which converts a detection outage into a human wait rather than an unchecked action.

What the rollout looks like. This is a constructed example rather than a report of a specific deployment. Shadow, for the period the tier requires, recording what would have been blocked. The finding worth anticipating: a widely used internal tool registered with the wrong exposure, which would have blocked a normal workflow on day one of enforcement. Registry data quality, not detector quality, is the usual reason a tool-use control cannot be switched on. Then canary on one team, then enforce, with rollback to shadow available to on-call without approval.

The point of this example. The expensive, model-based part of the control is the smallest part of the design. The load is carried by a registry lookup and a deterministic authority check, which is the same conclusion ARCH-001 reached from a different direction: the highest-value controls are usually authorization questions rather than classification ones.

Where this blueprint stops working

Budgets assume a synchronous request. Long-running agents that work for minutes have no user waiting on a p95, and their constraint is cumulative cost and the blast radius of many actions rather than added latency. The tiering logic still applies; the budget framing does not.

Tiering leaks information about the tiers. An attacker who can tell which requests took 400ms has learned which inputs reach the judge, and that is a usable signal for probing the cheap tier’s boundary. Where that matters, add jitter or pad the fast path, and accept the cost.

Fail closed is not free, and sometimes not right. A system that fails closed at tool execution during a detector outage stops doing useful work, and for some workloads that outage is more damaging than the risk it prevents. That is a legitimate business decision, and it belongs to the threshold owner from ARCH-005 with an explicit acceptance, not to the engineer choosing a default in a config file.