# The first-pass study, version 2 result

Version 2 of the pre-registered first-pass study ([PROTOCOL-V2.md](PROTOCOL-V2.md), approved 4 Oct 2026, frozen as `FREEZE-V2.json` sha256 `7114fe3b746358ad0d9bcef846055e1e71c44cf544dba64973bae2a6fa3cb38d`, chained to version 1.0's hash). The freeze was committed to the project's repository before the first run (commit `1052aa90741df478ef907847d643ac13adbd942b`, 4 Oct 2026 06:04 UTC), but unlike version 1.0's it was not posted publicly before the run, which section 8 asks for. Only the draft protocol was public. The frozen protocol and `FREEZE-V2.json` are posted here with this result, and that departure is disclosed rather than repaired. Run on 5 Oct 2026, 04:48 to 05:54 UTC. The runner refused to start unless the version 2 freeze held and the agent's shell was proven confined on the machine, and the freeze was re-checked after the run with no faults. Version 1.0's stages stand as published in [RESULT-STAGE1.md](RESULT-STAGE1.md) and [RESULT-STAGE2.md](RESULT-STAGE2.md).

**In one line: when a decision changed partway through the work, the agent with Unl did as well as an agent reading a perfectly kept decisions file, and needed far fewer corrections than one reading a file nobody updated or no file at all, at the same tokens as the kept file.** Every task was accepted in every arm. This is nine projects, so it is reported as estimates with intervals, never as a significance claim.

## Why version 2 exists, disclosed

Version 1.0 tested one baseline on purpose: the best case for a team without Unl, every decision written in `DECISIONS.md` and `CLAUDE.md` telling the agent to read it. Against that baseline Unl made no measurable difference. That null is why version 2 adds the baselines teams actually live in. Version 1.0's result shaped this design, and that is stated here beside it.

## What ran

- **Projects:** the nine held-out projects, each in four arms, interleaved project by project in the fixed order KEPT, STALE, NONE, UNL. 36 chains, 144 tasks, 36 pairs per comparison.
- **The arms:** KEPT, a decisions file kept perfectly up to date; STALE, the same file written at the start and not updated when a decision changed; NONE, no decisions file; UNL, no file and Unl's serve over the project's decision record.
- **The change:** one message at the end of each chain's second task replaces a decision. The fourth task tests the changed decision.
- **Agent:** Claude Sonnet (`claude-sonnet-5-5`) through Claude Code, with a shell confined by Claude Code's sandbox (no network, no writes outside the workspace). Before the run, the confinement check's red arm, with the sandbox off, reached the network and wrote outside; with the sandbox on, neither happened. **Reviewer and labeller:** Claude Opus (`claude-opus-5-5`), blinded. **Curator:** Claude Haiku (`claude-haiku-4-5-20251001`), producing the serve as it ships.
- **The serve code** ran from commit `73e01f49537ed487dd53348970f809f92d3ccc94`. The serve is unfrozen on purpose, because the study measures Unl as it ships.
- **Infrastructure faults:** 0. **Curator calls:** 48; 17 of them kept no decision for the turn.
- **Spend:** 12,500,266 tokens in all (agent 10,780,525; reviewer 1,267,621; labeller 330,941; curator 121,179), on the subscription. The protocol's estimate was about 12.5M.

## Results

Each comparison is the mean paired difference (UNL minus the baseline) with its 95% bootstrap interval, resampling projects, from the frozen `analysis-v2.ts` with the seed taken from the version 2 freeze. Lower is better for corrections.

| Corrections per task (M1) | Difference | 95% interval | Reading, in the protocol's own terms |
|---|---|---|---|
| UNL minus KEPT (H1, margin +0.25) | -0.03 | -0.11 to +0.06 | **Shown:** no worse than a perfectly kept file, within the margin |
| UNL minus STALE (H2) | -0.31 | -0.47 to -0.14 | **Supported:** fewer corrections than a stale file |
| UNL minus NONE (H3) | -0.42 | -0.53 to -0.31 | **Supported:** fewer corrections than no file |

**The changed decision (H4).** The share of projects whose changed-decision task first acted on the decision that had been replaced:

| Arm | Acted on the replaced decision |
|---|---|
| KEPT | 1 of 9 |
| STALE | 7 of 9 |
| NONE | 4 of 9 |
| UNL | 1 of 9 |

UNL minus STALE is -0.67 (-1.00 to -0.33). UNL minus NONE is -0.33 (-0.67 to 0.00). UNL and KEPT are identical. The one project where both KEPT and UNL acted on the old rule is the same one, `syn-school`.

**Where the effect sits (H5).** On authored ground, the tasks that depend on decided choices, UNL minus STALE is -0.67 (-0.94 to -0.39) and UNL minus NONE is -0.89 (-1.11 to -0.67). On free ground, discoverable facts, the arms are within 0.06 of each other. The difference is all on authored ground, as the study predicted.

**Tokens (H6, reported, not tested).** Agent and curator tokens per arm: KEPT 2,414,127; STALE 2,876,295; NONE 3,191,910; UNL 2,419,372. UNL over KEPT is 1.00, over STALE 0.84 and over NONE 0.76. Without a correct record, the agent spends more turns being corrected.

**Reversals of a decided choice (M2, secondary).** UNL minus KEPT 0.00; UNL minus STALE -0.25 (-0.36 to -0.14); UNL minus NONE -0.33 (-0.50 to -0.19).

**The raw counts.**

| Arm | Accepted | Corrections | Citing a decision | Turns |
|---|---|---|---|---|
| KEPT | 36 of 36 | 4 | 1 | 40 |
| STALE | 36 of 36 | 14 | 10 | 50 |
| NONE | 36 of 36 | 18 | 13 | 54 |
| UNL | 36 of 36 | 3 | 1 | 39 |

## What this does and does not say

It says that Unl matched a decisions file that someone kept perfectly up to date, without anyone keeping it, and beat a file that went stale after one change, and having no file. That is the claim the version 2 design can support.

It does not say Unl beats a perfectly kept file: H1 is an equivalence reading, and it shows "no worse", not "better". It is nine projects, with synthetic projects alongside one drawn from the house's own work, and one change per chain. A team whose decisions change more often, or whose file drifts further, is not measured here. The operator was not blinded to the arms, as declared in the protocol.

## The raw record

- [v2/rows.jsonl](v2/rows.jsonl): every model call, task, change and chain, as the runner wrote it.
- [v2/manifest.json](v2/manifest.json): the run's conditions, the freeze hash, the serve commit, the projects and the confinement proof. The one local path in it, the confinement record's, is written with the home directory as `~`; nothing else is changed.
- [v2/report.json](v2/report.json): the frozen analysis's output, unedited.