stack/packages/queue/README.md

# Queue

The work queue as data (#1508). `docs/plans/queue.json` holds the rows and an
append-only log of every operation. The table in `docs/plans/QUEUE.md`
between the two `mosaic-queue` markers is generated from it. The
specification is section 8 of
`agents/filbert/work/queue-as-data-plan-2026-09-26.md`. This README states
what the code does and the limits it accepts.

Until A2 adds `scripts/mosaic queue`, run the CLI directly from the canonical
checkout. Messages already name `scripts/mosaic queue`.

```sh
node packages/queue/src/cli.mjs list
node packages/queue/src/cli.mjs next darkwing
node packages/queue/src/cli.mjs move 9 in-progress --op darkwing-9-start-1 --by darkwing
node packages/queue/src/cli.mjs verify --current
node --test packages/queue/tests/
scripts/test-queue.sh
```

## Verbs

| Verb | Lock | Writes |
|---|---|---|
| `list`, `show ID`, `next [SEAT]` | no (see "Unlocked reads") | nothing |
| `add`, `move`, `release`, `assign`, `note`, `set` | yes | one log entry, the witness, the table |
| `genesis`, `accept-history` | yes, privileged | one log entry, the witness, the table |
| `render` | yes | the table, only when it is stale |
| `render --check`, `verify [--current]` | yes | nothing |
| `verify --snapshot DIR` | no | nothing; reads only the files it is given |
| `sync [--op ID]` | yes | fsyncs and the witness; logs nothing |
| `snapshot --out DIR` | yes | copies of both files into an empty DIR outside the repository |
| `unlock [--check-gate]` | no; takes the unlock gate | removes a dead or mismatched lock |

Every change takes `--op ID` (8 to 72 characters of `[a-z0-9._-]`,
starting with a letter or digit, not ending in `.outcome`) and an actor from `--by NAME` or
`$MOSAIC_AGENT_NAME`. Reads need no actor. `next` with neither a SEAT nor
`$MOSAIC_AGENT_NAME` refuses. The privileged actors are `jason` and `sage`.

Exit codes: 0 ok; 1 the operation failed; 2 invalid data or refused;
3 uncertain (visible or durable, not acknowledged); 4 usage.

Until piece D, `move ID in-review` needs `--candidate`: an existing file is
read as a manifest (one `<sha256>  <path>` line per file), anything else as a
commit reachable from `refs/heads` or `refs/tags`. The candidate is frozen
for the round.

The review's issue follows lead decision 23. A row with no issues can't
request review. A row with one issue uses it. A row with several needs
`--issue N`, one of its issues. Later rounds keep the previous round's issue
unless `--issue` names another; if the row no longer lists the kept issue,
the request refuses until `--issue` names one.

`move ID done` from in-review needs `--evidence
comment=<id>,round=<n>,candidate=<digest>`. The round must be the current
one and the digest its candidate's, so a comment from an earlier round
can't close a later one, even when the candidate is the same.

## Where the files live

- `docs/plans/queue.json`: `{version, canonicalRoot, revision, rows, log}`,
  serialized deterministically. A file that doesn't re-serialize byte for
  byte, or doesn't replay from genesis to its `rows`, is refused by every
  verb.
- `docs/plans/QUEUE.md`: hand-written header, generated body between the
  markers. Each log entry records the SHA-256 of the body it rendered, so a
  body is `current`, `stale` (an earlier render) or `unknown` (no render).
- `.git/mosaic-queue.head`: the witness, `{revision, logDigest, fileSha,
  at}` for the last write whose directory fsync succeeded. It holds no rows.
- `.git/mosaic-queue.lock` and `.git/mosaic-queue.unlock`: the lock and the
  unlock gate.

## Canonical checkout only

Every verb except `verify --snapshot` refuses unless the working directory's
toplevel is the `canonicalRoot` recorded at genesis, its git directory is
`<root>/.git` (a linked worktree fails), HEAD names the genesis branch, and
the CLI's own file lies under the root. `GIT_DIR`, `GIT_WORK_TREE` and
`GIT_COMMON_DIR` refuse. A symlink to the root works. Only one host is
supported.

## Trust boundary

The queue is cooperative (J2). An actor is whoever `--by` or
`$MOSAIC_AGENT_NAME` claims. The CLI checks each claimed actor against the
transition matrix and records it, and replay rechecks every entry, but
nothing stops a seat from claiming another name or writing the file with
other tools. Detection is what the queue offers: replay catches a hand edit,
the witness catches a rollback, and piece E lists changes for review.
Enforcement needs a boundary outside #1508.

## The git side of the protocol

The lock only binds the queue CLI. Git doesn't honour it. So these rules
are protocol, not enforcement:

- While the working `queue.json` has ops not yet committed, no one runs
  `checkout`, `stash`, `restore`, `reset` or a branch switch that touches
  `docs/plans/queue.json` or `docs/plans/QUEUE.md`. Only the lead restores,
  and only by the manual recovery below.
- Seats commit with plain `git commit`, never `--no-verify`, so the queue
  guard runs.
- Merge, rebase, cherry-pick, revert and `am` skip the guard. On the queue
  branch they are the lead's.
- Nobody disables, replaces or overrides the guard: no `core.hooksPath` in
  any scope, no `-c core.hooksPath=`, no `GIT_CONFIG_*` environment, no edit
  to `.git/hooks/pre-commit` and no change to its mode.

The witness detects lost history afterwards. It can't prevent a same-user
git write. A passing replay shows the file agrees with itself, not that no
history was lost.

## Committing the queue

The CLI never commits. The lead runs `scripts/queue-commit.sh -m MSG`. It
commits exactly the snapshot bytes of the two queue files on top of HEAD
through a temporary index and `commit-tree`, so nothing else that is staged
is swept in:

1. Record H, then check the guard is active: `.git/hooks/pre-commit` is a
   regular executable file owned by the user with HEAD's bytes, no
   `core.hooksPath` is set in any scope, and a canary run of `git hook run
   pre-commit` on a temporary index passes clean and refuses a changed
   queue entry. Refuse if a queue path is staged.
2. `queue snapshot` under the lock.
3. The base is `H:docs/plans/queue.json` from the object database.
4. HEAD's archived code, unpacked outside any repository, runs its queue
   tests and `verify --snapshot` against the base.
5. to 7. Blobs, a tree from a temporary index, `commit-tree`, the guard
   checks again, then `update-ref` with H as the expected old value. If the
   branch moved at any point after step 1, this fails and nothing is
   published.
8. Reconcile the shared index's two queue entries with `git reset -q --
   docs/plans/queue.json docs/plans/QUEUE.md`, only if HEAD is the new
   commit and those entries are still H's. Otherwise exit 3 and print the
   command. Until it runs, the guard refuses ordinary commits.

`scripts/git-hooks/pre-commit` is the guard. `scripts/queue-commit.sh
--install-hook` installs it (jason or sage). It refuses any `git commit`
whose index entries for the two queue files differ from HEAD's.

Bootstrap: the implementation lands with no `queue.json`; the lead installs
the guard, runs `queue genesis` from the reviewed migration map, then
`scripts/queue-commit.sh --genesis -m MSG`. Every mutating verb refuses with
`genesis not committed` until that commit exists. After that,
`scripts/test-queue.sh` also runs `verify` on the live queue.

Committing still needs its own authorization. The script never pushes.

## Durability

Supported: Linux with `docs/plans/` and `.git/` on local ext4, xfs or btrfs
(tmpfs for tests). The CLI checks the filesystem type and refuses others.
The host-crash claims assume that rename replaces its target atomically and
that fsync of a directory persists the renamed entry. The tests kill
processes with SIGKILL at each step. **Power loss is not tested.**

A write has three separate points:

| Point | When | Meaning |
|---|---|---|
| visible | the rename returns | readers see the new revision |
| durable | the directory fsync succeeds and the witness is written | it survives a host crash, within the assumptions above |
| acknowledged | the receipt `ok <op> rev N ...` prints | the caller may treat the op as done |

**The unacknowledged-op rule.** Only an op whose receipt printed is done.
If a call exits 3, is killed, or prints nothing, retry the same op id with
the same arguments. A recorded op answers with its original receipt and
changes nothing. A different verb or arguments under a used op id refuses.
`queue sync` confirms a visible tail and prints each op that became durable
without being acknowledged. Tell the seat named there.

## Unlocked reads

`list`, `show` and `next` read the witness, then `queue.json`, without the
lock. A file ahead of the witness prints `rev N visible, not confirmed
durable`. Any other disagreement takes the lock and rechecks before
reporting, so a write in progress is never reported as lost history.

## Known windows and limits

- **Check to rename.** A write checks that `queue.json` is unchanged since
  it was read, then renames its temp file over it. A git write between the
  check and the rename is not caught by the check. The witness catches the
  result afterwards. The table write has the same window: a hand edit to
  QUEUE.md between its check and its rename can be overwritten.
- **Same-seat claims.** A claim is the seat name only. Two sessions of the
  same seat are one claimant; the queue can't tell them apart and doesn't
  try.
- **Snapshot verify** checks that the given pair is valid, that the table
  is its render, and that its log extends the base. It does not certify
  witness continuity. Canonical `verify`, under the lock, does that.
- **Commit candidates** stay retrievable only while some ref keeps the
  commit. The queue keeps the manifest text, not the source bytes.
- **Locks** are never removed for their age. `unknown` and `invalid` locks
  wait for a person. A host rename makes old locks `unknown`.
- `queue unlock --check-gate` classifies a stale unlock gate. Remove the
  gate by hand only after it says `dead` or `mismatch` and no queue command
  is running.

## Manual recovery

For an invalid `queue.json`, lost history (the file doesn't extend the
witness) or a missing witness. No verb repairs anything; `accept-history`
is the only verb that runs in these states, and a file holding genesis
alone with no witness can also be confirmed by `sync`.

1. Copy the current `queue.json` bytes to
   `agents/<lead>/work/queue-recovery-<date>/` before touching anything.
   Restoring HEAD would discard every op written since the last queue
   commit.
2. Find the lost ops, the entries after the last committed revision, in the
   preserved bytes, `git stash list`, the reflog, the receipts seats hold,
   and issue comments carrying `mosaic-queue-op` markers.
3. Account for external effects. A lost op id is no longer deduplicated,
   and reusing it could act again.
4. Put a valid file in place: the restored version with the lost ops
   re-applied under new op ids, or a reviewed hand-built file.
5. Run `queue accept-history --op ID --reason TEXT --yes` as a privileged
   actor. It records the old witness (or its absence), the new revision and
   the reason, then rewrites the witness. The at-most-once guarantee does
   not hold for the lost range, and the entry says so.

A stale table needs a person to look, then `queue render`. An unknown table
needs the table restored with git, or the edit re-applied as queue ops,
then `queue render`.

## Tests

`node --test packages/queue/tests/` runs everything in scratch repositories
under the system temp directory; nothing touches this checkout's `.git`.
Faults reach the code only through options the tests pass in (`io`, `proc`,
`hook`, `now`, `readOrder`, `lockWaitMs`); the CLI passes none.

- `data.test.mjs`: serialization, replay, the transition matrix, `next`
  ordering, render.
- `lock.test.mjs`: every lock and unlock schedule in 8.4.
- `store.test.mjs`: genesis, canonical checks, op ids, claims, briefs,
  views, snapshot.
- `write.test.mjs`: every write fault, SIGKILL at each step, git
  interference, unlocked-read races.
- `commit.test.mjs`: `queue-commit.sh` and the guard, through PATH shims
  that run an action at an exact point in the procedure.