The problem
Chokepoint Placement decides where a detector sits. It says nothing about what happens once a chokepoint earns more than one, the ordinary result of running a guardrail programme for any length of time: a rules engine first because the risk is structured and cheap, a trained classifier once rules start missing unstructured cases, a judge last for whatever the first two leave ambiguous. ARCH-002 names the resulting problem directly: writing a detector is not the hard part; making a dozen, built differently and on different schedules, behave like one system a policy layer can reason about, is.
Here is where that goes wrong: it looks, from the outside, like careful engineering. Each of the three detectors returns a number between 0 and 1, because the shared interface requires one regardless of technique, and on a given input each happens to return 0.90. A team reads that agreement as a strong signal and averages the three, or takes the maximum, or sums them with hand tuned weights, then treats the result like each input to it: a probability, tunable at one global cutoff.
None of the three 0.90s was ever that. A regex returns 1.0, or near it, when it matches, because it either matched or it did not; the number describes the match, not whether it mattered, and a sixteen digit reference number trips a card pattern at the same confidence as an actual card number. A classifier’s 0.90 is an approximately calibrated probability, but only against the population it trained on, rarely the population now hitting production. A judge asked to rate severity out of ten and returning a 9 is reporting a sampled token, not a probability; ask it twice and it can report an 8. Three different kinds of fact, combined as one. The output means nothing, and because averaging looks like more rigor than any single detector, nobody questions it until an incident review asks why the cutoff that “works” was ever supposed to.
Forces
Combining more detectors feels like more rigor, and averaging their scores feels like the natural way to spend it. A number built by arithmetic over inputs that were never on a common scale carries the appearance of statistical support without the substance, which makes it hard to challenge later. A single detector’s score is at least honest about being one opinion. An averaged score pretends to be a synthesis of three, when a synthesis of incomparable things does not exist.
Calibrating every detector onto one shared scale is the statistically stronger fix, and it does not stay fixed. Mapping a detector’s raw output onto a common scale, so a calibrated 0.90 means the same empirical fact wherever it came from, roughly nine of ten items flagged this way are true positives, takes held out labelled data matching the population the detector runs against right now. It has to be redone whenever the detector, threshold, or traffic mix moves, and in a live programme all three move often. The fix is real; keeping it current is a standing cost, not a one time project.
Weighted and majority composition both assume the detectors fail for different reasons, and that assumption is invisible until somebody tests it. Three regex variants over the same keyword list agree with each other constantly, and that agreement is not independent confirmation of anything; it is one detector voting three times under three names. A composition rule is only as sound as the diversity of what it composes, and that has to be checked, not assumed from a headcount of detectors in the pipeline.
The pattern
State the rule the way ARCH-002 states it: do not combine scores, by default. Give each detector its own threshold, tuned against its own labelled data, and let it emit a verdict: clear, flagged, or abstain when it lacks enough signal. Combine verdicts with logic that is written down and reviewable. The reason this works is narrower than it sounds: a threshold answers one question entirely inside its own detector’s frame, did this input cross the line this detector’s own data calls meaningful, so the layer above never asks what the number underneath meant. A verdict never travels; only the decision does.
This is the specific failure The Uncalibrated Ensemble names: an ensemble that averages, maxes, or sums scores built on different scales, mistaking the result for evidence. The two are a matched pair, one naming what not to build, the other what to build instead.
None of this is a wholly new mechanism. Liu et al.’s catalogue of agent design patterns names Voting-based Cooperation, several agents’ votes composed toward one task decision, and credits it with accountability and identity traceability in multi-agent coordination. That is a different shape of problem: agents voting on a shared outcome, not one control point’s detectors sitting on scales never meant to compare. Proposed status here means this specific argument, only verdicts against each detector’s own threshold may be combined, never the raw scores underneath, did not turn up named in the prior art surveyed for this catalogue. Not that nobody had built a multi-detector pipeline before. Only that no surveyed source stated the score versus verdict distinction as the rule an ensemble must obey.
ARCH-002’s interface contract keeps this checkable: the fields below are what keep a score from being mistaken for a verdict:
interface DetectorResult {
detector_id: string;
execution_status: 'ok' | 'error' | 'timeout' | 'skipped';
decision: 'clear' | 'flagged' | 'abstain' | null; // null unless status is ok
score: number; // 0..1, meaningful only here
score_kind: 'raw' | 'calibrated'; // raw: meaningful only within this detector
threshold_version: string; // which threshold produced `decision`
category_schema: string;
}
// WRONG: three scales, averaged into one meaningless number
const fused = results.reduce((sum, r) => sum + r.score, 0) / results.length; // 0.90
// RIGHT: each already decided, against its own threshold_version
function compose(results: DetectorResult[]): 'flagged' | 'clear' | 'abstain' {
const live = results.filter(r => r.execution_status === 'ok'); // a timeout is not a verdict
const flagged = live.filter(r => r.decision === 'flagged');
const highPrecision = flagged.some(r => r.detector_id === 'regex_pii_v3');
if (highPrecision || flagged.length >= 2) return 'flagged'; // a live flag outranks a dead detector
return live.length < results.length ? 'abstain' : 'clear';
}The first line is the whole anti-pattern in one expression: arithmetic cannot know score means three different things per detector. The function below never touches score. It reads decision, already computed against each threshold_version, composed with logic a reviewer can read top to bottom. WEIGHTED, a high precision detector flags or two lower precision ones agree, is what the function above does; ARCH-002 also names ANY, MAJORITY, and ALL, each right for a different balance of recall against cost. A judge is expensive enough that pipelines reserve it for the band the cheaper detectors could not decide.
Worked example
Consider a document retrieval system inside a regulated institution, checking PII disclosure risk before a response leaves the system. The pipeline grew the way these usually do: a rules engine catching account and card numbers at effectively no cost, a classifier added once rules missed unstructured disclosures no pattern list anticipated, a judge added last for cases the classifier could not call with confidence.
On one flagged response, the rules engine returns 0.90 because a sixteen digit reference number matched its card pattern, an ordinary business identifier and not a card number. The classifier returns 0.90 because the surrounding text resembles disclosures it trained on, a real signal. The judge, asked to rate disclosure risk out of ten, returns a 9, a sampled judgement about context neither of the other two can see. Weighted and summed, the old ensemble lands the response at 0.90, over the cutoff a team once found empirically and froze, exactly the folklore ARCH-002 warns about. The response is blocked on grounds tracing back to a false positive, beside two numbers that never meant what the weighting formula assumed.
Composed as verdicts, the rules engine’s 0.90 never reaches the policy layer. Its own threshold, tuned against real card numbers and known false positives like this exact pattern, returns clear once that pattern is excluded, the actual fix, made inside its own frame rather than papered over by a global cutoff. Say the rules engine still flags a genuinely ambiguous match elsewhere in the response, on a different span than the reference number. The classifier, against its own labelled disclosures, returns flagged on the surrounding text, a signal the rules engine cannot see. The judge returns abstain, tuned to treat this shape of case as inconclusive. Under WEIGHTED, a high precision flag is enough on its own: the response is blocked, but now for a reason that survives a question, traceable to a specific detector, threshold, and evidence span, exactly what Decision Record exists to carry forward, because a verdict nobody can reconstruct months later was never evidence, however defensible it looked the day it fired.
When not to use it
The pattern has a real cost, and it is not owed to a single detector. One rules engine at one chokepoint has nothing to compose; it emits its own verdict against its own threshold, and building a WEIGHTED rule and an independence check over an ensemble of one is machinery with nothing to govern. Add the second detector when it earns its place, not before.
Verdict composition is also a deliberate trade of statistical power for auditability, and that trade is not free everywhere. A programme with the labelled volume to calibrate every detector onto one shared scale, and a population stable enough that calibration will not rot before anyone checks it, has a legitimate reason to run score fusion instead. This pattern is the default because most programmes never reach that condition, not because the alternative is wrong.
And the auditability this pattern buys is only as real as the thresholds underneath it. A prototype running three detectors against default, untuned cutoffs and composing their verdicts anyway has wrapped the same arbitrariness the score path had in a format that merely looks more disciplined. The logic is sound the moment each detector’s own threshold is; before that, it is theatre with better field names.
Related patterns
Chokepoint Placement decides which stage in the request path a detector occupies; once that stage carries more than one, this pattern decides how their outputs combine. The Uncalibrated Ensemble is the failure this pattern prevents, named from the other side: averaging, maxing, or summing scores that were never on the same scale, mistaking the result for evidence. Tiered Detection decides which calls even reach an expensive detector in the ensemble; this pattern decides what happens to its verdict once it arrives. Baseline and Floor keeps a composed detector’s per-detector threshold honest release over release, so the tuning this pattern depends on does not quietly erode. Decision Record is where the composed verdict, its evidence, and threshold version have to live, because a verdict nobody can reconstruct later was never evidence at all.