stack/agents/dewey/work/chat-02/BACKEND.md

# CHAT-02 backend: review packet, revision 2 (#1507, row 5)

Author: Dewey, 2026-09-26. Brief: `BRIEF.md` R4 (`636b0fac…`), §2.1 and D3.
Base: `1c5f6bc3` on `refactor`. Nothing here is committed; the candidate is
the working tree, pinned by the hashes below.

Revision 1 was `286c3ad5`, kept as `BACKEND-r1-286c3ad5.md`. Filbert
reviewed it and asked for a revision
(`agents/filbert/work/chat-02-backend-review-2026-09-26.md`, `27d64e14…`).
Darkwing approved the routes in it
(`agents/darkwing/work/chat-02-routes-review-2026-09-26.md`, `07b10ad1…`)
with two nonblocking notes. §0 answers both reviews.

Reviewers (Sage's order):
- Filbert: the code, all nine files. He asked to review the delta.
- Darkwing: the two board routes. `serve.mjs` and `serve.test.mjs` changed
  for his notes, so the route delta needs his look.

The Console (`packages/webui`) is the next step and is not in this packet.
The four WebUI browser failures Filbert saw in the shared tree come from my
unpinned Console edits to `app.js` and `index.html`. They are Console work.

## 0. Revision 2 changes

### Filbert, required

1. **Branch ids are stable.**
   - Roots are entries whose parent is null or absent, in file order. The
     first root's line is `main`. `main` exists even in a file with no
     entries, as an empty branch with a null leaf.
   - At a fork the earliest child in file order continues its parent's
     branch. Each later child starts a branch named `b.<its id>`. Each later
     root (Pi's `resetLeaf`) starts `b.<its id>` too.
   - Growth never renames a branch. `page.branch` always equals the cursor's
     branch, and `open({branch})` with a name taken before growth still
     works.
   - The default branch is the one ending at Pi's default leaf, the last
     entry in file order. `view.defaultBranch` names it; `view.branches`
     flags it with `isDefault`.
   - Follow rebuilds the cursor's own branch from a fresh snapshot. It
     refuses `source-replaced` with reconcile if the branch is gone, is
     shorter than the parts served, or the ids of those parts differ.
   - Tests: F12 "two leaves" (branches `main` and `b.o4`, one branch value
     across follows, the pre-growth name still opens, the `main` view picks
     up its growth), F12 "a second root", F12 "a follow refuses when an
     appended duplicate id changes the branch's earlier parts", F1 "a follow
     stays on its branch".
   - Your `branch.mjs`: open and follow both give `main`, every entry is
     labelled `main`, and reopening with the first branch id works.
2. **The bridge is gone.**
   - A missing parent stops the history with a `missing-parent` notice. When
     malformed lines sit just before the entry, the notice names them: "Line
     N could not be read and may have held it." Those lines are not repeated
     as separate notices.
   - The history before the gap is its own leaf branch and reads normally.
   - Tests: F1 "a missing parent stops the history with a notice that names
     the unreadable lines", F1 "an unreadable fork is never merged into
     another branch's history" (your case).
   - Your `bridge.mjs`: `b.y` shows only the notice and `leaf`.
3. **`EACCES` and `EPERM` on a directory component become `unreadable`.**
   - `lstatOrNull` maps them through `denied()`. The root is refused on its
     own row; the catalogue and opens under other roots are unaffected.
   - Test: "a seat directory without search permission refuses that root,
     not the catalogue", with the bad root first and then last.
   - Your `perm.mjs`: the catalogue returns the good root's row and lists
     `bad` in `refusedRoots` as `unreadable`. The open of an unknown id walks
     both roots and returns an `unknown-conversation` refusal, not a throw.

### Filbert, small

- Redacted thinking (`redacted: true`) is `unavailable` with empty text,
  whatever the `thinking` field says. Test: the mapping test.
- A relative header `cwd` is refused as `foreign-project`. Test: F10
  `relative.jsonl`.
- Malformed notices appear when a file has no valid entry. Each malformed
  line keeps its own notice at its file position, on every branch, including
  after the leaf. Test: F1 "a file whose entries are all unreadable".

### Darkwing, nonblocking

1. A cursor call without `branch` is now 400 ("a cursor call repeats the
   page's branch"), and the header comment says so.
2. `REFUSAL_STATUS` is exported. A new test scans the reader's three source
   files for every `new Refusal("<code>"` plus `UNSUPPORTED_HARNESS` and
   fails on a missing or stale entry. Its first run found `unavailable`
   falling to 422; it is now 404. `unknown-actor` (403) and
   `unsupported-purpose` (422) are now explicit.

Route delta against revision 1: `git diff 34777c56 -- packages/control-board`
shows the whole change; against Darkwing's pins it is the header comment,
`export` on `REFUSAL_STATUS` with three new entries, one line in
`conversationQuery`, the new test and branch names in expectations
(`main`, `b.e5`, `b.e1`).

## 1. Candidate hashes

```
f9008c01c608ea9aacdd15459f03a4ca8f8b4fe4d5225f3f8337bc5f9282f955  packages/conversation/package.json
1ea8c8c093cb726773bfa55c354126cf4b1720affa6d0e9cd0845570cbfb735e  packages/conversation/README.md
da336ed2a2a149ff56ac12af2440a8353d4573c33e9069381be60f1a0ca70151  packages/conversation/src/parts.mjs
20e781240846298fa65f9a8226b5e1ee1e81338b81229334666d845c17c99ac1  packages/conversation/src/pi.mjs
72c3255bca7a894f6484b3224aabf054d40a0db78cbe46a6c492ed385333b8ad  packages/conversation/src/reader.mjs
10a1ff9e91c1f1e4684fc38c5bca83c50d79bb1f459d269517396545a269203a  packages/conversation/src/safe-fs.mjs
78b7719b0c183cc1d9fedf009fb8a5e4dae34a30d1b39f931824016cc07fa86b  packages/conversation/tests/reader.test.mjs
d62720dcf1bd43ce9412356a04b5248a85aec59dd78194887e3c38010c0aa2f3  packages/control-board/src/serve.mjs
d38aa2b209e0fb22ab3c52c3cb26a8f05e9e269e567d40b7e6ad88430c593f4a  packages/control-board/tests/serve.test.mjs
```

Unchanged from revision 1: `package.json` and `parts.mjs`. The first seven
files are new. `serve.mjs` and `serve.test.mjs` are diffs against the base.
`scan.mjs` is unchanged. D3 allowed edits there, but none were needed.

## 2. What it does

`packages/conversation/README.md` is the reference: API, sources, safe open,
parser rules, branch names, page limits, snapshot, epoch and cursor rules,
follow, the refusal table and costs. In short:

- `rootsFromSpecs` turns the board's repository specs into roots, using
  registrations only as hints. `createReader` serves `catalogue`, `open` and
  `next`.
- Pages and cursors are CHAT-01 `page` and `cursor` records. Everything the
  Console needs beyond CHAT-01 goes in `view`.
- The board serves `GET /api/conversations` (no parameters) and
  `GET /api/conversation?id=&branch=&cursor=`. Both run after the existing
  Host/Origin guard (`d1629d61`) and its GET/HEAD check.
- Responses are `application/json` with `no-store` and `nosniff`, and no CORS
  headers.
- Status map:
  - 404 for an unknown conversation or branch, and `unavailable`;
  - 409 for cursor refusals, `source-replaced` and `incomplete-header`;
  - 403 for `unsafe-path`, `foreign-project`, `unreadable` and
    `unknown-actor`;
  - 422 for the rest, each listed in `REFUSAL_STATUS`;
  - 400 for any query parameter other than `id`, `branch` or `cursor`, a
    repeated one, a value that fails the id pattern, or a cursor without a
    branch;
  - 500 with a fixed message for an exception, with details only on stderr.

## 3. Choices and deviations to review

1. **Catalogue rows are a summary shape, not CHAT-01 `catalogueItem`.** The
   row is: conversation, seat, project, harness, history, title, readOnly,
   `controlMode: "unavailable"`, the three separate time fields,
   availability, `unsupportedReason` and refusal. `catalogueItem` carries
   binding and control fields that belong to CHAT-03. I did not want to fill
   them with placeholders that look authoritative.
2. **Registrations match on `sessionsDir`, not `sessionFile`.** Registrations
   have no `sessionFile` field. The brief §2.1 rule is applied to the
   directory instead: seat, layout `repo`, project and `samePath(sessionsDir)`
   must all match. A registration never adds a root or names a file (F9).
3. **Placeholder rows for non-Pi seats.** A seat on another harness has no Pi
   directory, so the board has no spec for it. A registration adds one
   `unsupported-harness` row when its `sessionsDir` is the standard
   directory under a project root that is already approved. Its directory is
   never read. Rocko (`claude-code`) is the live case (F15).
4. **An epoch id is comparable only within one cursor chain.**
   `sourceEpoch = "e-" + sha256(dev:ino:digest)`, taken at open and carried
   forward while the prefix verifies. Two opens of an unchanged file give the
   same value. After growth, a new open gets a new value for the same epoch,
   while the old chain keeps its own. Only a new inode, a shorter file or a
   changed prefix is a new epoch, and that is refused.
5. **Follow stays on its branch.** On the last page, `follow` replaces
   `cursor`.
   - `next` with it re-verifies the pinned prefix, takes a fresh snapshot and
     rebuilds the cursor's branch by name.
   - It returns the parts after the ones already served. The branch must
     still exist and the ids of the served parts must be unchanged;
     otherwise `source-replaced` with reconcile.
   - If Pi's default moved to another branch, the page stays on the old
     branch and `view.defaultBranch` names the new one. The Console decides
     what to show; nothing switches silently.
6. **A missing parent stops the history.** There is no bridge. The notice
   names any malformed lines just before the entry, since one of them may
   have held the parent. The history before the gap is its own branch. A
   malformed line keeps its notice at its file position on every branch, so
   a branch's served prefix does not change as the file grows.
7. **`unreadable` refusal.** `EACCES` or `EPERM` on a root listing, a
   directory component or a file open becomes a per-row 403, so one
   unreadable path cannot fail the whole catalogue.
8. **256 MiB file cap** (`too-large`, 422). The largest real file today is
   18.6 MB.
9. **Every page re-hashes the whole pinned prefix.** This makes detection
   simple and total, at a cost measured in §4.3.
10. **Refusal code names** not already in CHAT-01 are CHAT-02 values, like
    `unsupported-harness` (D2): `unsafe-path`, `foreign-project`,
    `unreadable`, `not-a-pi-session`, `incomplete-header`, `too-large`,
    `unknown-branch`, `unavailable`, `unknown-actor` and
    `unsupported-purpose`.

## 4. Evidence

All runs are on the candidate hashes above.

### 4.1 Suites and contract checks

- In `/tmp/dewey-chat02/overlay`, a `git archive` of `1c5f6bc3` with only
  the nine files overlaid: `node --test --test-concurrency=1` over the
  conversation, control-board, webui and seat suites gives 181 tests, 181
  pass (29, 124, 9 and 19).
- `node docs/plans/chat-00/check.mjs`, `chat-01/check.mjs` and
  `chat-01c/check.mjs` all exit 0.

### 4.2 Fixture map (brief §2.1)

| # | Test |
|---|---|
| F1 | five tests in `reader.test.mjs`: malformed notice in place (with a hostile id); missing parent names the unreadable lines; an unreadable fork is never merged; a follow stays on its branch; an all-malformed file shows a notice per line |
| F12 | three tests: two leaves with stable names across growth; a follow that refuses when an appended duplicate id changes served parts; a second root |
| F2–F11, F13–F15 | one named test each in `reader.test.mjs` |
| F16 | `serve.test.mjs` §12: foreign Host and cross-origin Origin on both routes give 403; a spy reader records zero calls, and no `access-control-*` header is sent |
| F17 | every reader call in `reader.test.mjs` runs inside a fingerprint of the whole fixture tree: size, SHA-256, mtime (ns), (dev, ino), mode and every directory listing. The route test in `serve.test.mjs` does the same. Files no read may touch are mode 000. |

Additional coverage: mapping of every Pi entry type and role (including
redacted thinking), pagination at 100 parts and at the byte cap,
surrogate-safe fragments, unknown and empty and non-Pi files, an unreadable
file or root, a seat directory at mode 000, every refusal code having an
HTTP status, and CHAT-01 schema validation of every page and cursor the
tests produce (Python `jsonschema`).

### 4.3 Real data (read-only, this checkout)

- Catalogue: 19 rows in 75 ms, no refused roots. There are 18 available Pi
  conversations across darkwing, dewey, filbert, researcher and sage, plus
  rocko as `unsupported-harness`.
- Reading every page of every conversation gave 102 pages and 8693 entries,
  with no refusals. Every page is on branch `main`. The slowest conversation
  took 798 ms (662 ms in revision 1, on files that have grown since; I did
  not isolate the difference). The largest file is 18.6 MB.
- 186 pages and cursors sampled from that run: 0 schema-invalid.
- Over HTTP, through a temporary board on port 0 with a temporary `boardDir`
  (the live board was not touched): 19 rows and 102 pages in 1487 ms. Rocko
  returns 422 `unsupported-harness`.
- Scripts and output: `evidence/backend/smoke.mjs`, `smoke-stats.txt`,
  `http-real.mjs`, `http-out.txt`.

### 4.4 Mutation testing

Mutations were run in scratch copies under `/tmp/dewey-chat02/`, never in
the served tree. The route scratch is a full copy of the checkout (without
`.git`, `agents`, `v1` and `skills/aws-*`), so each mutant's failures are
real test failures, not import errors. Scripts and full output:
`evidence/backend/mutate.py`, `mutate-board.py`, `mutate-backend.txt` and
`mutate-routes.txt`.

Reader: 38 of 39 caught.

| Mutant | Caught by |
|---|---|
| no O_NOFOLLOW and no lstat symlink check | F7, F8 |
| directory components unchecked | F7 |
| listing follows symlinks | F7, F8 |
| no prefix digest check | F4 |
| no dev/ino check | F3 |
| no shorter check | **not caught, equivalent** (below) |
| cwd always in project | F10 |
| relative cwd accepted | F10 |
| no character cap | F14, schema |
| no page byte cap | F14 |
| no block cap | F14, schema |
| no foreign check | F6 |
| no branch binding | F6 |
| no expiry | F6 |
| no parentSession notice | F11 |
| branch parameter ignored | F12 two leaves, F12 second root, F1 missing parent |
| unknown branch served | F12 two leaves |
| registration harness ignored | F15 |
| no placeholder roots | F15 |
| malformed notices dropped | four F1 tests |
| bridge restored | F1 missing parent, F1 unreadable fork, F1 follow |
| missing-parent lines not named | F1 missing parent, F1 unreadable fork, F1 follow |
| trailing malformed notices only on the default branch | F1 missing parent, F1 follow |
| branch named by its leaf | all three F12 tests, two F1 tests |
| later child continues the branch | F12 two leaves |
| first root not `main` | all three F12 tests, two F1 tests |
| no empty `main` | F1 all malformed, unknown/empty |
| redacted thinking shown | mapping |
| `EACCES` on a component rethrown | seat directory at mode 000 |
| `incomplete` never set | F2, F5 |
| `next` re-reads fresh instead of pinned | F2, F5, F12, F1 follow |
| follow skips the history check | F12 duplicate id |
| follow skips the id digest | F12 duplicate id |
| follow reads the default branch | F12 two leaves, F1 follow |
| catalogue writes a file | F17 in six fixtures |
| wrong thinking visibility | mapping |
| raw tool ids | mapping, schema |
| id namespaces dropped | F1 |
| surrogate pair split | fragments |

Removed since revision 1: "no bridge" and "follow ignores branching", whose
behaviour no longer exists.

The equivalent mutant: removing the "file is shorter" check changes nothing
observable. A shorter file cannot supply the pinned length, so the prefix
digest differs and `source-replaced` still follows. The check stays because
it gives the refusal without hashing a truncated read.

Board routes: 11 of 11 caught. The mutants were: routes before the guard, no
`nosniff`, every refusal 422, no query validation, unknown parameters
allowed, `reconcile` dropped, cursor ignored, registrations ignored, a
cursor without branch served, `unavailable` falling to 422, and a CORS
header sent.

## 5. Known limits

- **No `openat` in Node.** A directory component swapped between the checks
  and the open is caught by the re-check after the open, not prevented.
  Nothing is read from a descriptor whose (dev, ino) does not match.
- **The `fstat` mismatch branch of F8** is covered by design, not by a
  test. The F8 test swaps in a symlink, which `O_NOFOLLOW` refuses first. A
  plain-file swap between `lstat` and `open` needs a race the suite cannot
  schedule.
- **An in-place rewrite with the same inode** is detected by the digest of
  the pinned prefix, as the brief requires. Growth after the pin is not a
  change.
- **Follow compares entry ids, not content, for the parts already served.**
  The pinned byte prefix is still re-hashed, so a rewrite of served bytes is
  caught. An appended entry that reuses a served id is caught when it moves
  that entry off the branch or puts another id in its place (F12 duplicate
  id, both cases). If it keeps the same id in the same position with new
  content, follow does not notice: the parts already served keep the old
  text, while a fresh open shows the new one. Pi never writes duplicate
  ids.
- **Cost.** Each page reads and hashes the whole pinned prefix; §4.3 has the
  cost on real files.
- **One actor.** Cursors live in memory and are lost when the board restarts;
  the refusal is `cursor-unknown` with reconcile. `local-operator` is the
  only actor, which is not multi-actor safety (CHAT-04R).
- **Claude history** waits for B1 and CHAT-03 (D2).

## 6. After review

- Commit condition (Darkwing): `serve.mjs` imports
  `../../conversation/src/reader.mjs` at load, so `packages/conversation/`
  lands in the same commit as the routes or an earlier one, after Filbert's
  approval.
- Pushing goes through Sage with Jason's word. The live board needs a restart
  after the commit to serve the routes.
- The WebUI proxy allowlist for the two routes is Console work and is not in
  this packet.