# The first-pass study, version 2: does Unl hold the right world when nobody keeps the file

**Approved by Neil on 4 Oct 2026, as version 2.0.** His answer to the three questions in section 11 was "yes", recorded as the recommended option: version 2 with **four arms** (the fifth, STALE+UNL, is **out**) and the **+0.25** equivalence margin, run after the Why Benchmark's r0 score finishes. Version 1.0 (`PROTOCOL.md`, frozen as `FREEZE.json`) is not changed and its stages stand as published. Version 2.0 has its own freeze (`FREEZE-V2.json`), chained to version 1.0's hash and disclosed in every result.

## 0. Why a version 2

Stage 1 of version 1.0 (`RESULT-STAGE1.md`) found no difference: when every decision is written in `DECISIONS.md` and `CLAUDE.md` tells the agent to read it, the agent gets every decided choice right without Unl, and Unl costs about a fifth more tokens. Version 1.0 tested one baseline on purpose, the best case for the team without Unl. It did not test the cases teams actually live in: a decisions file nobody updated after a decision changed, or no file at all. It also gave the agent no shell, and the one task where the arms differed was corrected only because the agent could not run its own tests.

Version 2 tests those cases. The claim it can support, and the only one it is built to support, is this: **Unl matches a perfectly kept decisions file without anyone keeping it, and beats a stale file or no file.** Each half can fail, and both are published whichever way they fall.

## 1. The question

When a decision changes partway through a run of work, does an agent with Unl act on the current decision as reliably as an agent reading a perfectly maintained file, and more reliably than one reading a file nobody updated, or no file?

## 2. Hypotheses (stated before any run; each can fail)

Arms are named in section 4: KEPT, STALE, NONE and UNL.

- **H1 (primary, equivalence).** UNL is no worse than KEPT on corrective turns per task. It fails if the upper end of the 95% interval for UNL minus KEPT is above the margin of **+0.25 corrections per task**. The margin is fixed here, before any run. With 9 pairs the interval may be too wide to show equivalence even if the arms are the same; that outcome is reported as "not shown", never as "equivalent".
- **H2 (primary).** UNL needs fewer corrective turns per task than STALE. It fails if the paired difference is zero or favours STALE.
- **H3.** UNL needs fewer corrective turns per task than NONE. It fails the same way.
- **H4 (the changed decision).** On the task that tests the changed decision, the share acted on the replaced decision is lower with UNL than with STALE and with NONE, and no higher than with KEPT.
- **H5 (the essay's prediction, carried from version 1.0).** The differences sit on authored-ground tasks and are near zero on free-ground tasks.
- **H6 (cost).** Reported, not tested: tokens per chain in every arm, and the ratio of UNL to each baseline.

## 3. Worlds and chains

**Worlds.** The same 9 held-out projects as version 1.0 (sha256 `c5fb689f1d337344ebe6b5d57212d5eeb66f2f29b1461786d7c8ec42d04d5a51`), built by the fleet study's `workspace.ts`, imported unchanged.

**The changed decision comes from the dataset, not from new writing.** Every held-out project holds two decisions that replace an earlier one, each with the reason it changed, and five `no_resurrect` tasks that test them. Version 2 uses this directly. For each project the harness picks, by a rule fixed before the freeze, one replacing decision `R` that replaces `D`, and the first `no_resurrect` task by id that names `R` as bearing and `D` as must-not.

**The chain: the 4 scored tasks of version 1.0, with the change between the second and the third.** Steps are numbered so that "task 4" below means step 4.
1. **Authored**, the project's first `act_within` task by id that does not name `D` or `R` (unchanged from version 1.0 where possible).
2. **Free**, the project's first free-ground task from the frozen pool (unchanged).
3. **The change (not scored).** After step 2, the person tells the agent the decision changed, in the dataset's own words: `R`'s statement and the reason it replaced `D`. This is one message at the end of task 2's conversation, identical in every arm, and the agent acknowledges it. It is how a decision actually changes in a team: someone says so, once, in a conversation.
4. **Authored, after the change**, the chosen `no_resurrect` task. It starts a fresh conversation, as every task does, so the agent only knows about the change if something carried it.
5. **Free**, the project's second free-ground task (unchanged).

That is five agent conversations per chain: four scored tasks and the change message, which is not scored.

## 4. The arms

All four arms get the identical workspace except for the decisions file, the identical agent, and the identical change message.

- **KEPT (a team that keeps its file perfectly).** `DECISIONS.md` holds every decision live at the start, with `D` live and `R` absent. At the change, the harness rewrites `DECISIONS.md` to mark `D` replaced and add `R` with its reason, and commits it as `harness: decisions updated`. `CLAUDE.md` says "The project's decisions are in DECISIONS.md. Read them before acting." This is version 1.0's baseline, with the upkeep done for the team.
- **STALE (a team that wrote the file once).** The same `DECISIONS.md` and `CLAUDE.md` as KEPT at the start. At the change, nobody updates it. `D` still reads as live.
- **NONE (a team with no file).** No `DECISIONS.md`. `CLAUDE.md` holds only what `README.md` holds. The agent knows what the task, the files and the change message tell it.
- **UNL.** The NONE workspace, plus Unl's curated serve before every agent turn, produced as it ships (version 1.0 section 9). At the change, the change is recorded in Unl the way a person records one, by a save with its reason, through the same write the product uses. Nobody edits a file.

**A fifth arm, STALE+UNL, was offered and is out** on Neil's answer. It would have asked whether Unl corrects a file that has gone wrong rather than replacing one. It is not run in version 2.0.

## 5. The agent, the shell and the reviewer

- **The agent.** One Claude Code builder (`claude -p`, Sonnet, the same model in every arm), with Read, Glob, Grep, Write, Edit **and a shell** (Bash), confined to the workspace with no network. Stage 1 showed that a task asking for tests drew corrections only because the agent could not run them; the shell removes a correction that belongs to neither arm. How the shell is confined is settled and tested before the freeze (section 9), and if it cannot be confined the run does not start.
- **The reviewer.** Unchanged from version 1.0: blinded, Opus, no tools, accepts or returns one correction, cap of 4. It sees the decisions **as they stand at that task**, so after the change it judges against `R`. A correction on task 4 that the output follows `D` cites `R` and counts as a reversal.
- **Blinding.** As version 1.0. The reviewer and labeller never see the arm, the serve, or the decisions file the agent had.

## 6. Measures

Version 1.0's M1 to M7, computed by its analysis unchanged, for each pair of arms. One measure is added:

- **M8 changed-decision conflict.** On task 4, whether the agent's first output acted on `D` (a first review that cites `R` against the output). Reported per arm as a share of 9, with the paired difference for each comparison in section 2.

## 7. Analysis

- Pairs are the same project in two arms. Every comparison in section 2 is a paired difference with a 95% bootstrap interval resampling projects (10,000 resamples, seed from this protocol's frozen hash), as version 1.0.
- Primary: H1 and H2 on M1, pooled over the four scored tasks. Everything else is secondary and reported with its interval, never a significance claim.
- Split by authored and free ground (H5), and task 4 alone (H4).
- **No stages.** All 9 projects run, interleaved project by project in a fixed arm order (KEPT, STALE, NONE, UNL), so a model or service change mid-run falls on every arm.

## 8. Integrity

As version 1.0 section 8: frozen and hashed before run 1 (this protocol, the selection rule and the chosen `R`, `D` and tasks per project, the change message, the confinement, the runner, the analysis); amendments are new versions; infrastructure failures re-run the whole chain and are recorded; publication whichever way it falls.

One disclosure is new. Version 1.0 stage 1 was run before this design existed and its result shaped it: a null at the strong baseline is why the weaker baselines are added. That is stated in the result, beside the version 1.0 numbers, which stand on their own.

## 9. Settled before the freeze (implementation, not design)

Each item below was open in the draft and is settled as built:

- **The shell is confined, and proven on the machine before every run.** The builder's Bash runs in Claude Code's own sandbox (`scripts/first-pass/agents-v2.ts`): sandbox enabled, unsandboxed retries off, an empty network allowlist with the strict allowlist on, and Bash allowed only because it is sandboxed. The macOS sandbox profile around the whole agent process, the plan in the draft, would also have cut the agent off from its own model, so the sandbox confines the shell rather than the agent. `scripts/first-pass/confinement-v2.ts` asks the agent to fetch a public page, write outside its workspace, and write and run a script inside it, and reads the verdict from disk. Its red arm (the same request, through the same agent and spawn site, with the sandbox off) must show the fetch and the outside write landing, so the check can see a leak. Measured on 4 Oct: sandbox off, both leaks landed; sandbox on, neither did, and the script inside ran and printed 42. The runner repeats this proof at the start of the run and refuses to start if it fails.
- **How UNL records the change.** Version 1.0's serve does not read the live product store: it curates over the project's own decision record (`curatorField` and `renderCurated`, version 1.0 section 9). In UNL, the change is applied to that record exactly as a save with its reason leaves it: R is added with its reason and D is marked replaced by R. The serve after the change is therefore the product's curated serve over the record a person's save would have produced. Nobody edits a file.
- **What waits for the change.** Anything the dataset says rests on R (`built_on_by`, followed through) is held back with R and arrives with it, in every arm. Measured on the held-out set: one decision is held back (in syn-warehouse), none in the other eight projects.
- **The selection rule** is `scripts/first-pass/select-v2.ts`, and its output is recorded in `FREEZE-V2.json` per project: R, D, what is held back, the four task ids and the sha256 of the change message.
- **The harness** is `runner-v2.ts`, `agents-v2.ts`, `confinement-v2.ts`, `analysis-v2.ts` and `freeze-v2.ts`, beside version 1.0's files. It imports version 1.0's agent prompt, rework accounting, reviewer, labeller, pool and bootstrap unchanged, and the freeze hashes every one of them, so version 1.0's freeze keeps holding.

The draft's items, as they were written, for the record:

- **Confining the shell.** The agent's Bash must reach no network and nothing outside its workspace. The plan is the macOS sandbox profile around the agent process with network denied, proven by a red test (a `curl` that must fail and a write outside the workspace that must fail) before the freeze.
- **How UNL records the change.** Through the product's own save path against the study's per-project workspace, so the serve after the change is what a person would get. The write path is chosen and tested in a dry run before the freeze.
- **What else waits for the change.** Any decision the dataset says rests on `R` is held back with it and arrives with it, in every arm, so the start state is a world that could really have existed before the change. Checked at source on 4 Oct: every held-out project has 2 or 3 `no_resurrect` tasks that name each replacing decision as bearing and the decision it replaced as must-not, so the rule always has a task to pick.
- **The selection rule for `R`, `D` and the tasks**, written as code in a new `select-v2.ts` and hashed. Version 1.0's `select.ts` is not edited.
- **The harness changes** go in new files beside version 1.0's (a v2 runner that imports version 1.0's reviewer, pool and analysis unchanged), so version 1.0's freeze keeps holding.

## 10. Cost and time (an estimate, for the approval)

Version 1.0 stage 1 cost 1.63M tokens for 6 chains, about 0.27M per chain, in 35 minutes. Version 2 has 5 conversations per chain instead of 4 and a shell that will lengthen some turns, so allow about 0.35M per chain.

- **Four arms:** 36 chains, about **12.5M tokens**, about 4 to 5 hours.
- **Five arms (with STALE+UNL):** 45 chains, about **16M tokens**, about 5 to 6 hours.

All on the subscription, through `claude -p`. It should not run while the Why Benchmark's r0 score is drawing on the same allowance.

## 11. What Neil was asked, and his answer

1. Yes or no to this design as version 2.0. **Yes.**
2. The fifth arm, STALE+UNL: in or out. **Out** (his "yes" was recorded as the recommended option, which excluded it).
3. The equivalence margin for H1, +0.25 corrections per task: keep it or set another. **Kept at +0.25.**
