Asking a model to write a probability throws away what it already knows

ARCH-002 says not to threshold on a model’s self-reported confidence. This Lab set out to test that, and the interesting part turned out not to be that the number is untrustworthy. It is that the model has a better answer available and writing it down as text is what loses it.

The instrument comes first

An instrument that has not been tested against a known answer is not evidence. Before any real detector was measured, synthetic detectors with known calibration went through it: one perfectly calibrated, one overstating every claim by a fixed amount. The instrument separates them, 0.01 against 0.21.

It also established a baseline that matters more than anything else here. Expected calibration error is biased upward on small samples, because each bin’s observed rate carries noise and the metric takes the absolute value of every gap, so noise cannot cancel. A perfectly calibrated detector still reads:

| Rows | Mean ECE for a perfect detector | Observed range over 30 trials | A corpus at this scale | | ---: | ---: | --- | | 50 | 0.104 | 0.036 to 0.224 | the Lab-004 corpus | | 200 | 0.053 | 0.026 to 0.083 | JBB-Behaviors | | 569 | 0.031 | 0.012 to 0.048 | R-Judge | | 1,000 | 0.021 | 0.013 to 0.043 | deepset/prompt-injections | | 10,000 | 0.007 | 0.003 to 0.015 | wildguardmix |

These are means over thirty trials at each size on a six-value score grid with ten equal-width bins, and the spread is wide: at fifty rows a perfectly calibrated detector read anything from 0.036 to 0.224. This is an expected finite-sample value, not a lower bound and not a threshold. The bias depends on the score distribution and the binning as well as the sample size, so simulate your own detector’s distribution rather than reusing these numbers.

A detector that overstates by 0.35 reads about 0.21, which fifty rows can just about distinguish from a calibrated one on average and not reliably on any single run. A detector off by 0.05, which is the interesting case, cannot be told from a perfect one below roughly a thousand rows. Most published agent-safety corpora sit in the top three rows of that table.

The ablation

Comparing different models cannot say what caused a difference between them. So this arm changes exactly one thing, and it changes it after the model has already answered.

The model is asked one question: does this ticket contain text trying to make an agent call a tool it is not allowed to call, answer Yes or No. At temperature 0 the token it emits is the argmax of its own distribution, so two readouts are available from a single forward pass. One takes the word. The other takes the probability mass on Yes against Yes plus No. Same prompt, same call, same weights. Nothing else can differ, because nothing else is run.

AUC is the measure because the two readouts put their scores in different places and ranking is the only comparison that survives that. It measures ordering and says nothing about whether a number is a calibrated probability.

llama3.2:3b said “No” to all fifty tickets. One distinct value, AUC 0.500, which is no information whatsoever. The distribution behind that same token ranks all fifty correctly. The model had a graded belief the entire time and the act of reading it as a word threw every bit of it away.

The scale moved. The logprob readout averages 0.106 on injected tickets and 0.007 on clean ones. Ranking is perfect and a 0.5 threshold catches nothing. A readout change is a detector change: every threshold tuned against the old one is invalid, and more granularity bought more candidate operating points rather than a calibrated number.

What the typed models do not prove

Asked the original question, the one that requests a number in text, the three general models used three, four and six distinct values with 88 to 90 percent of answers at exactly 0.0 or 1.0, ranking at 0.733 to 0.800. kev-0.8b used twenty values with none at the endpoints and ranked at 1.000, and jev-1.13.0 used ten and also ranked at 1.000. kev-0.8b is the smallest model in that set.

It is tempting to read that as the typed output head being the cause. It is not established here. Those systems differ from the general models in base model, training and output head all at once, and no same-model ablation of the head was run. What the comparison supports is narrower: granularity did not track model size in these configurations, which is the specific thing you would need if reaching for a bigger general model were the answer. It does not rule out gains from scaling within one family.

The precision column is a trap

Every detector produced zero false positives on twenty clean tickets, so measured precision reads 1.00 at every base rate. That is not a finding, it is the corpus. Zero events in twenty trials bounds the false-positive rate at roughly 3/20, not at zero.

A detector that looks perfect on the corpus is one whose false-positive rate the corpus never measured.

What this does not measure

• Calibration. Granularity is not calibration. A coarse detector can be calibrated and a granular one can be badly wrong. Whether a stated 0.93 is right 93 percent of the time is unmeasured here for every detector, because at fifty rows a perfectly calibrated detector reads 0.104 on average and up to 0.224 on a single run.
• Latency as a vendor claim. Jev was reached from Pune through a Cloudflare gateway; its vendor measured 70 to 500 ms from their own laptops. Not comparable, and nothing here refutes theirs.
• Generalisation. Fifty tickets from six injection texts. AUC 1.000 is a statement about this corpus and no other.
• The best possible prompt. One prompt pair. A better text prompt might narrow the gap.

What we got wrong

A claim that outran its evidence, twice, caught in review both times. The first version of this page said the mechanism produced the probability rather than the model, on the strength of comparing kev-0.8b against three general models. Review pointed out that those systems differ in base, training and output head at once, so the causal claim was not earned.

The fix was to run an ablation. The first ablation was also wrong: it compared a numeric-JSON prompt read as text against a Yes/No prompt read from logits, which changes the question as well as the readout. Review caught that too. The version above changes neither: one prompt, one forward pass, two ways of reading the same token. That is the third attempt and the first one that isolates anything.

Review caught three more. Near-binary output was described as impossible to calibrate, which is false; a map can attach an empirical rate to three values, and the real limit is that calibration cannot recover a distinction the detector never made. Recall at a single threshold was read as evidence about uncertainty expression, which it does not support. And the ECE figures were called a floor, implying a bound, when they are an expected finite-sample value with a wide spread.

A token cap read as a model refusing to answer. The cap was first set to 400. gemma4:e2b thinks before answering, so twelve of fifty replies were truncated and scored as unparseable, which looked like refusal. This is the same mistake Lab-004 records, repeated by someone who had read it that day. At 700 it parses fifty of fifty.

An expired token discarded a four minute run. A bare HTTPError 401 threw away three models already measured. A failed call is now recorded as an unparsed verdict rather than raised. On the final run jev parsed 49 of 50 and the run survived, which is the fix doing its job.

Reproduce

Everything except the last row runs on a laptop.

python -m labs.lab_007.run                  # the instrument and the floor, no models
python -m labs.lab_007.baseline             # three local models, text readout
python -m labs.lab_007.baseline --logprob   # the ablation
python -m labs.lab_007.baseline --kev       # an open-weight typed decision model

kev-0.8b is Apache-2.0 and serves the same contract Jev does. Only the jev row needs an account.