# The additive fleet test: the same fleet, with and without Agent Unl

Pre-registration protocol, version 1.0. The question it answers: "the headline evidence is the same fleet, doing the same work, with and without Agent Unl in it".

This public copy has internal reference ids and names removed; nothing else differs. The frozen original is hashed in `FREEZE.json` (sha256 `efb1137c34f92ed413c1fbdbb80db2dfab00b5462090259a6156b619a9a4b9ce`), so this copy will not hash to that value.

Nothing in this file may change once its hash is recorded in `FREEZE.json`. The runner refuses to run if any frozen file differs from its recorded hash. A changed protocol is a new version with a new hash, filed before its first run.

## 1. The question

When Agent Unl joins a fleet that already works well, as one more agent, does the fleet's work follow the person's decisions more closely, and what does that cost in rework, drift, time, collisions and tokens?

This test does not compare Unl with any other tool. The fleet, its agents, its models and its tools are identical in both arms. The only difference is whether Agent Unl is in the fleet.

## 2. Hypotheses (stated before any run; each can fail)

- **H1 (primary).** With Agent Unl, the fleet crosses fewer of the person's decisions per task than without it, in both fleet shapes.
- **H2.** With Agent Unl, the share of tasks where the fleet acts fully within every decision that bears is higher.
- **H3.** Total tokens per episode with Agent Unl, its own inference included, are no more than 25% above the same episode without it.
- **H4.** Rework is not higher with Agent Unl.
- **H5.** Drift from the project's stated aim is not worse with Agent Unl.

H1 fails if the paired difference in crossings per task is zero or favours the arm without Agent Unl. H3 fails if the ratio exceeds 1.25. Every result is reported whichever way it falls.

## 3. The worlds and the tasks

The source is the Why Benchmark's held-out set (`docs/research/why-benchmark/dataset/held-out.json`, frozen with sha256 `c5fb689f1d337344ebe6b5d57212d5eeb66f2f29b1461786d7c8ec42d04d5a51` in that benchmark's `FREEZE.json`). It holds 9 projects, each a set of decisions written with their reasons, what each replaced, and what rests on each. It also holds 225 tasks in five types, and each task names, before any run, the decisions that bear, the decisions that must not drive the action, and a rubric.

**Task selection, fixed now.** For each project, take the first task by id of each of the five types (act_within, no_resurrect, detect_cross, novel_transfer, nothing_bears), in that type order. That gives 5 tasks per project and 45 in all. `select.ts` does this, and its output is hashed in `FREEZE.json`.

**Overlap with the Why Benchmark.** These tasks also sit in the Why Benchmark's held-out run. That run scores single model answers and this one scores a fleet's work, so neither result can inform the other's scoring. The overlap is stated so no reader has to find it.

## 4. The fleets

Two fleet shapes, chosen because they are how people commonly run more than one agent today:

- **Fleet A, a Claude Code pair.** A planner and a builder, both `claude -p` on Sonnet. The planner reads the workspace and writes a plan for the task. The builder does the work in the workspace.
- **Fleet B, Claude plans and Codex builds.** The same planner. The builder is `codex exec` with the Codex CLI's default model, recorded in each row.

Both agents may read, search, write and edit files in the workspace. Neither may run a shell, reach the network or use any MCP server. Every agent runs in a scratch directory outside this repository, with no API key in its environment. Claude runs on the Claude subscription, Codex on the ChatGPT sign-in.

**Concurrency.** Each episode runs its 5 tasks in three waves: tasks 1 and 2 together, then tasks 3 and 4 together, then task 5. Each builder in a wave works in its own git worktree off the same starting point. At the end of the wave both are merged in task order. A file that both changed in conflicting ways is a collision. The conflict is resolved in favour of the earlier task, and the later task's lost hunks are counted.

## 5. The two arms

**Without Agent Unl (the fleet as a careful team runs it today).** The workspace holds:
- `README.md`: what the project is and what it is for.
- `DECISIONS.md`: every decision with its reason. A replaced decision is marked replaced, with what replaced it and why.
- `CLAUDE.md` and `AGENTS.md`, each saying "The project's decisions are in DECISIONS.md. Read them before acting."

This is a strong baseline on purpose. The decisions are all there, well kept, and each agent is told to read them.

**With Agent Unl.** The identical workspace, plus Agent Unl as one more agent in the fleet:
- **The serve.** Before the planner acts and before the builder acts, Agent Unl's curated serve for that agent's input is placed at the head of its prompt, headed "From Unl, the why agent in this fleet". It is produced exactly as the Why Benchmark's agent_unl condition produces it: `curatorField`, then `selectForTurn` through the subscription judge, then `renderCurated`. It carries the decisions that bear, each with its reason and what it replaced.
- **The truthful board.** After each wave, a `BOARD.md` lists every task in the episode: done, in progress or waiting; which files each finished task changed; and which collisions happened. It is built deterministically from git, and both agents in the next wave see it.

The shadow gate is not in version 1. In shadow it is never shown to an agent, so it cannot change the work. Recording its would-fire count is left to version 2.

## 6. Measures

Each is computed by `analysis.ts`, frozen with this protocol.

- **M1 crossings per task (primary).** A blinded judge reads the task, its rubric, the text of every decision it names, the task's diff and the builder's final reply. It answers whether a decision that must not drive the action drove it, or a decision that bears was contradicted (crossed: yes or no), and whether the work acted within every decision that bears (honoured: yes or no). For nothing_bears tasks, crossed means the work invented a constraint that no decision states.
- **M2 honoured share.** The share of tasks judged honoured.
- **M3 rework.** Of the lines each task added, the share a later task in the same episode deleted or changed, read from `git blame` at the end of the episode.
- **M4 drift.** The same judge scores the final workspace against the project's stated aim, from 1 (worked against it) to 5 (served it throughout).
- **M5 time to done.** Wall-clock seconds per episode.
- **M6 collisions.** Conflicting files per episode, and lost hunks.
- **M7 cost.** Input and output tokens for every model call in the episode: planner, builder, Agent Unl's curator and the judge (reported apart, and not counted as fleet cost). Taken from each CLI's own usage record. Tokens are the unit, because the two builders are priced differently and neither run is billed per token.

**The judge.** `claude -p` on Opus with no tools and no MCP, in a scratch directory. It is blind to the arm: the word "Unl" and the serve's heading are replaced with "[source]" everywhere in what it reads, and `BOARD.md` is excluded from the diff. The judge prompt is `judge.ts`, frozen here.

## 7. Analysis

The unit of comparison is the pair: the same project, the same fleet shape, with and without Agent Unl.

- For each measure, report the mean paired difference (with minus without) and a 95% interval from a bootstrap that resamples projects (10,000 resamples, seed 1f95abde).
- Report each fleet shape separately, then pooled.
- Report the H3 cost ratio, and the per-type breakdown of M1.

**Stages.**
- **Stage 1 (the first run, reviewed before stage 2):** the first three projects by id, both fleet shapes, both arms. That is 12 episodes, 60 tasks and 6 pairs. With 6 pairs the intervals will be wide, and stage 1 is reported as estimates with their intervals, never as a significance claim.
- **Stage 2:** the remaining six projects, run unchanged. The full result pools all 18 pairs.

## 8. Integrity

- **Frozen before any run:** this file, `select.ts`, `judge.ts`, `analysis.ts`, the runner, and the selected task ids, each by sha256 in `FREEZE.json`. The freeze commit's hash is recorded in Unl as a gate beat before the first episode starts.
- **Order:** arms and fleets run in a fixed interleaved order (project by project; within a project A-without, A-with, B-without, B-with), so a model or service change during the run does not fall on one arm.
- **Records:** every model call is appended to `rows.jsonl` with its role, model, tokens and wall-clock. Every episode's final workspace is kept. Nothing is re-run to get a better result: an episode that fails for an infrastructure reason (a usage limit, a crash) is re-run whole and the failure is recorded.
- **Publication:** the harness, this protocol and the frozen hashes are published with the stage 1 result, so anyone can re-run it on their own fleet.
