# The additive fleet test, with Agent Unl as shipped

A second measurement of the with-arm, run 25 to 26 Sep 2026 on Neil's word (stop 47f03acc, option 1: "re run", momentum 6733749d). **It does not replace [RESULT.md](RESULT.md).** That is the pre-registered result and stays the result of record. This page reports the same test with the curator working as it now ships, beside the original, as that option asked.

## Why it was re-run

In the pre-registered runs the curator used the judge's old default (Haiku with thinking on). Under fleet load that setting hit its 45-second ceiling on 66 of 180 calls (37%), and each of those turns got the whole uncurated field instead of a curated serve. The limits audit (a999e2bb, #1375) measured the setting timing out 8 times in 8, and changed the default to thinking off (9c32a972). So the +0.10 measured a curator that no live path uses.

## What ran

- **Code:** the frozen harness, unchanged. Both runs carry freeze sha256 `d58f11fe089fd62896bf2d40302cdfa3d85df92c14fed9a938ed48e635ab3d52`, the same as the pre-registration. `runner.ts` refused to start otherwise.
- **Only the with-arm was re-run.** Each run directory (`fleet-additive-s1-with-as-shipped-v1`, `fleet-additive-s2-with-as-shipped-v1`) was seeded with the original runs' without-arm episode, task and call rows, marked `seeded_from`. Then `runner.ts --resume` ran the 18 with-arm episodes it did not find. The without-arm is therefore the same 18 episodes in both comparisons.
- **The curator as shipped:** the judge default is now thinking off, from `scripts/judged-serve/judge.ts` at current main.
- **Totals:** 18 with-arm episodes, 90 judged tasks, 0 infrastructure faults and 0 unjudged tasks. The curator fell back to the full serve on **1 of its 180 calls** (it was 66 of 180).

## The judge under fleet load, before and after

This answers cargo bd26c743 on f2bcbe8b ("re-run the fleet test's load and report timeouts, median and p95 judge latency, and the share of turns served curated, before and after"). All 180 curator calls on each side, from the `call` rows:

| | Pre-registered | As shipped |
|---|---|---|
| Timeouts at the 45 s ceiling | 66 | 0 |
| Median judge latency | 33.7 s | 2.9 s |
| p95 judge latency | 45.0 s (the ceiling) | 3.4 s |
| Slowest call | 45.0 s | 7.1 s |
| Turns served curated | 114 of 180 (63%) | 179 of 180 (99%) |

The one uncurated turn as shipped was not a timeout. The judge answered in 7.1 s (syn-gamestudio, 418 output tokens), but no kept set was recorded, which is consistent with an answer the strict parser refused. It is noted here as a finding for the judge, not investigated in this run.

## Results, pooled over both stages (18 pairs)

Each measure is the mean paired difference (with minus without) and its 95% bootstrap interval over projects, from the frozen `analyse()`.

| Measure | Pre-registered (RESULT.md) | As shipped |
|---|---|---|
| M2 honoured share (higher is better) | +0.100 [+0.044, +0.156] | +0.067 [+0.022, +0.111] |
| M1 crossings per task (lower is better) | −0.044 [−0.100, +0.011] | **−0.044 [−0.078, −0.011]** |
| M4 drift score, 1–5 (higher is better) | +0.56 [+0.22, +1.00] | +0.56 [+0.33, +0.78] |
| M3 rework (lower is better) | −0.008 [−0.025, +0.001] | −0.002 [−0.008, +0.001] |
| M5 seconds per episode | +170 [+96, +223] | **−1 [−67, +36]** |
| M6 collisions per episode | −0.11 [−0.33, 0.00] | 0.00 [−0.17, +0.17] |
| H3 cost ratio, with ÷ without (limit 1.25) | 1.150 | 1.186 |

**Raw counts:**
- Honoured: 61 of 90 without; 70 of 90 with (pre-registered); 67 of 90 with (as shipped).
- Crossed: 9 of 90 without; 5 of 90 with, in both.

## What changed, read plainly

1. **The time cost is gone.** The +170 seconds per episode was the curator waiting out its ceiling. With the judge that answers, Agent Unl adds no measurable time.
2. **Fewer crossings now clears zero.** H1 was "not shown" in the pre-registered result; as shipped, the interval is [−0.078, −0.011]. The point estimate did not move, but the curated serve made it consistent. By task type, detect_cross is −0.167 [−0.333, −0.056].
3. **The honoured-share gain is smaller, and still clear of zero.** +0.067 against +0.10. The re-run does not say the working curator is worse at this, only that 3 fewer tasks were honoured out of 90. The two intervals overlap almost entirely.
4. **Drift is the same, with a tighter interval.**
5. **Cost rose a little and stays under the limit.** 1.186 against 1.150, because every curated call now completes and is counted.

## What this does not show

- **It is not pre-registered.** The hypotheses were fixed before the first run, not this one, so read it as a second measurement of the shipped product, not a confirmation.
- **The without-arm was not re-run.** Both comparisons share one baseline, so the difference between them is only the with-arm. The with-arm ran a day later, on the same models, and agents vary run to run. Part of any gap between the two columns is that variation.
- **One setting changed, not one variable.** Thinking-off answers faster and changes which blocks are kept (see the comment on `judgeThinking` in `judge.ts`). This measures the product as shipped, not thinking on against off in isolation.

## Reproduce it

Seed a run directory with a finished run's without-arm rows, then run `npx ts-node scripts/fleet-additive/runner.ts --stage <1|2> --run-id <id> --resume`. Present it with `report.ts <id>`. Pool the two stages by passing both runs' rows to `analyse()` from `analysis.ts`.