--- r1.md +++ docs/plans/2026-09-26_ledger-t3-source.md @@ -1,8 +1,11 @@ # Ledger: a read-only T3 thread source for Table 2 (Gate F brief) -Brief only, no code. Darkwing wrote it on 2026-09-26 at Sage's request. Filbert -reviews it, and Jason sees it on the decision sheet before anyone builds it. -Issue #1506. +Brief only, no code. Darkwing wrote it on 2026-09-26 at Sage's request, issue +#1506. R1 (sha256 08959a05) went to Filbert, whose review asked for +revisions: `agents/filbert/work/ledger-t3-source-review-2026-09-26.md`, sha256 +19dda29a. This is R2. It takes every finding, and it records Sage's rulings +on the three open questions. Section 1 has one measurement that differs from +the review. ## Why @@ -18,8 +21,8 @@ ## Where T3 keeps messages T3 keeps its state in one SQLite database, `~/.t3/userdata/state.sqlite`, in -WAL mode (`state.sqlite-wal` and `state.sqlite-shm` sit beside it). Three -projection tables are enough: +WAL mode (`state.sqlite-wal` and `state.sqlite-shm` sit beside it). The counts +need three projection tables: - `projection_projects`: `project_id`, `workspace_root`, `deleted_at`. - `projection_threads`: `thread_id`, `project_id`, `title`, `archived_at`, @@ -27,51 +30,72 @@ - `projection_thread_messages`: `message_id` (primary key), `thread_id`, `role` (`user` or `assistant`), `text`, `created_at` (ISO UTC). -One more table is optional. In `orchestration_events`, each +The JSON diagnostic reads one more. In `orchestration_events`, each `thread.message-sent` event carries `metadata_json.origin`. Messages typed in the T3 app carry an `appVersion` there. Messages sent through T3's API or MCP -tools, which is how seats talk to each other, don't. See the cross-check below. +tools, which is how seats talk to each other, don't. The same directory also holds `secrets/`, `clerk-tokens.json` and other settings files. The reader opens `state.sqlite` and nothing else, and it selects named columns only, never `*`. -## Reading it, with T3 running or not +## 1. Reading it, with T3 running or not -The file stays on disk whether T3 runs or not. The reader opens it with Node's -built-in `node:sqlite` (`DatabaseSync`, `file:?mode=ro`, `readOnly: -true`). That needs no dependency, and Node 26.8.1 prints no warning for it. I -read the live database this way today, while T3 was running, with no errors -and no locks. A WAL reader sees every committed message, including those still -in the `-wal` file. - -Two rules: -- Never open with `immutable=1` and never copy the file. Both skip the WAL - and silently lose the newest messages. A copy of the three files is also - not atomic. -- If T3 stopped uncleanly and left a `-wal` without its `-shm`, a read-only - connection may be unable to rebuild the index. If the open fails, the - ledger reports it and refuses. I have not tested this case or the fully - stopped case. Both are acceptance checks below. +The reader uses Node's built-in `node:sqlite` (`DatabaseSync`). That needs no +dependency, and Node 26.8.1 (SQLite 3.53.4) prints no warning for it. -## Jason or agent +- **URI.** Build it with `pathToFileURL(dbPath)` and set `mode=ro` through + `searchParams`, then pass `readOnly: true`. A `?`, `#` or `%` in the home + path would break a string-built URI. +- **One snapshot.** Run every query, from the schema checks through the + diagnostic, inside one `BEGIN` … `COMMIT`. In autocommit mode each + statement sees its own snapshot while T3 writes between them. +- **Busy timeout.** Set `DatabaseSync`'s `timeout` to 5 s. A transient + `SQLITE_BUSY` during a T3 checkpoint then waits instead of failing. A busy + error after the timeout exits 1 like any open failure. +- **No `immutable=1` and no copy.** Both lose the WAL. Filbert found worse + than lost messages: with a table created inside the WAL, `immutable=1` + fails with `no such table`. + +What happens on disk. Filbert and I both tested these in scratch +directories: + +| State | Directory writable | Result | +|---|---|---| +| T3 running, writer attached, newest rows only in `-wal` | yes | reads them | +| `-wal` without `-shm` (writer killed, `-shm` removed) | yes | reads the WAL rows and creates `-shm` | +| `-wal` without `-shm` | no | open fails, SQLite 14 | +| T3 stopped cleanly, no `-wal` or `-shm` | yes | reads, then leaves an empty `-wal` and a 32 KiB `-shm` | +| T3 stopped cleanly | no | fails, SQLite 1544 "attempt to write a readonly database" | + +In every case the main file's bytes stayed the same. The last row is where +Filbert and I differ. His review says the stopped-case read works with the +directory read-only. In my run it failed with and without the read +transaction. The build's test settles it. Either way a failed open is exit 1. + +So the accurate claim: the reader never writes the main database file. Like +any SQLite connection, it may create or update `-wal` and `-shm` beside it +and takes read locks in `-shm`. T3 opens normally afterwards. + +## 2. Jason or agent Reuse the 6a rule. The first line of `text` decides: a T3 header or the tmux preamble counts as agent, `control-board` as the sender counts as board, and anything else counts as human. Messages with role `user` count; assistant messages don't. -6a has a defect this source would expose. Its regex allows only a lowercase -class (`class=[a-z-]+`). Seats send uppercase classes: Sage's DECISION, INFO, -REVIEW-REQUEST and REVIEW-NOTE, and my own REVIEW-REQUEST. In this project's -threads, 16 real agent headers fail on that alone and would count as human. -The fix is to make the class match case-insensitive. It belongs in this build -or just before it, reviewed with it. The ms-communications table lists -lowercase names, so the fix follows what seats send, not the table. +The class fix rides in this build (Sage's ruling). HEAD's +`packages/ledger/src/ledger.mjs:81` (tmux) and `:83` (T3) both allow only +`class=[a-z-]+`. Both become case-insensitive. Seats send uppercase classes: +Sage's DECISION, INFO, REVIEW-REQUEST and REVIEW-NOTE, and my own +REVIEW-REQUEST. In this project's threads 16 real agent headers failed on +that alone at 20:54Z. The ms-communications table lists lowercase names, so +the fix follows what seats send, not the table. Cross-check, read at 2026-09-26T20:54Z for the mosaic-stack project (209 user messages outside imported and deleted threads, every one with its -`thread.message-sent` event): +`thread.message-sent` event). Filbert's later read agreed, plus messages sent +since. | T3 origin | Header matches 6a | Count | |---|---|---| @@ -83,110 +107,193 @@ No message typed in the app carries a header, and every API message in this project carries one of the three forms. The 14 free-text ones are older Discord Bot thread headers such as `[from: SetSpark coordinator (…) -> to: -Discord Bot (…)]`, written before the guide fixed the format. With the class -fix they still count as human. That's 14 wrong human counts, all dated -2026-09-17 to 2026-09-22. - -Recommendation: the header rule decides, as Sage asked. The reader also -reports one diagnostic number, not used in any table: user messages the rule -calls human that T3 recorded as sent through the API. That count is how the -uppercase-class bug showed up, and it would catch the next format drift. The -origin field is T3's internal metadata, not a documented contract, so it -shouldn't decide anything. I'd make it JSON only, so Table 2's layout stays -the same. - -## Thread to seat - -A thread counts for this checkout only if its project's `workspace_root` is -the ledger's repository root. That is `/mnt/storage/src/mosaic-stack`, project -`34050c07`. - -Thread IDs change whenever Jason starts a new thread for a seat, so there's no -fixed map. T3-AGENT-COMMS.md already names threads after the seat ("Darkwing", -"Sage", "Dewey in Claude"). Proposed rule: a thread belongs to seat `` when -`` is a real directory under `agents/` and the lower-cased title equals -`` or starts with `` followed by a space. Several threads can map to one -seat. Their counts add up, as several Pi session files already do. +Discord Bot (…)]`, written before the guide fixed the format. Sage ruled they +stay as recorded: they count as human, dated 2026-09-17 to 2026-09-22. + +The header rule decides. The JSON also carries one diagnostic that feeds no +table or total: user messages the rule calls human that T3 recorded as sent +through the API. That number exposed the class bug and would catch the next +format drift. `origin` is T3's internal metadata, not a documented contract, +so it decides nothing. If `orchestration_events` or a column it needs is +missing, the diagnostic reads `unknown` and the report goes on (Sage's +ruling on F5). Missing tables the counts depend on still exit 1. + +## 3. Thread to seat + +**Project.** A thread counts for this checkout only if its project's +`workspace_root` equals the ledger's repository root, byte for byte. The CLI +already takes that root from the realpath of its own URL, today +`/mnt/storage/src/mosaic-stack`, project `34050c07`. So a T3 project opened +through the compatibility symlink `~/src/mosaic-stack-dev-test` doesn't +match, and "no project row" is the right refusal. The README says so. + +**Title rule.** Thread IDs change whenever Jason starts a new thread for a +seat, so there's no fixed map. T3-AGENT-COMMS.md already names threads after +the seat ("Darkwing", "Sage", "Dewey in Claude"). A thread belongs to seat +`` when `` is a real directory under `agents/` and the lower-cased +title equals `` or starts with `` followed by a space. So "Sagebrush" +stays unmapped. Several threads can map to one seat, and their counts add +up, as several Pi session files already do. Today that maps Sage, Darkwing, Filbert, Dewey and Rocko (one thread each, -created 2026-09-26), plus "Darkwing in Claude" (archived) and "Dewey in -Claude". Three threads map to no seat. Two are imported and excluded anyway -("FINDINGS.md review" and "[dragon-lin:darkwing -> …"). The third is -"Discord Bot" with 68 user messages: 54 without a header, and the 14 -free-text headers above. The guide's own advice, titles like `review: -`, will produce more unmapped threads. - -Unmapped threads go in one Table 2 row, `t3:unmapped`, so Jason's messages -there still count toward the Human column and the human-per-closed ratio. The -other choice is to drop them, which would hide those 54 headerless prompts. -That is Jason's decision. I recommend the row. - -A seat's row sums its Pi and T3 counts. JSON splits them by source. Nothing is -counted twice: every T3 session today runs on `claudeAgent` or `codex`, which -don't write `.pi/state`, and Filbert found no T3 header in any Pi log. +created 2026-09-26, titles set by hand), plus "Darkwing in Claude" (archived) +and "Dewey in Claude". Researcher has a directory and no thread. Three +threads map to no seat. Two are imported and excluded anyway ("FINDINGS.md +review" and "[dragon-lin:darkwing -> …"). The third is "Discord Bot" with 68 +user messages: 54 without a header, and the 14 free-text headers. + +Titles are current state, and T3 can write them itself. They go wrong three +ways. T3 auto-titles an unnamed thread from Jason's first prompt, so "Rocko +review of the plan" maps to rocko. A rename moves the whole history to +another row. A seat thread titled for a topic drops into `t3:unmapped`. +None of this changes the Human total or the human-per-closed ratio. It only +moves counts between rows, but Gate F reads one seat's row. + +**Header cross-check.** The headers already say which seat a thread belongs +to. For every user message whose header matches the fixed 6a rule and whose +`to:` id equals the message's own `thread_id`: +- in a mapped thread, the `to:` role, lower-cased, must equal that thread's + seat; +- in an unmapped thread, the `to:` role must not be a seat name. + +A conflict exits 1 and names the thread id, its title and both roles. A +header whose `to:` id is some other thread is not checked. The check reads +message text only, not T3 metadata. In a live read at 21:02Z every header +agreed: all 104 addressed to their own thread carried the full thread id and +named that thread's seat (Sage 40, Darkwing 15, Filbert 18, Dewey 15, Rocko +16). + +It catches a seat thread renamed to another seat or to a topic, once any +agent writes to it. It also catches an auto-titled thread that agents +address by a different seat. It misses a thread no agent ever writes to. +Such a thread can only add human counts to a seat's row, never hide them, so +for Gate F it errs toward a visible failure. The README says so. + +**Unmapped row.** Unmapped threads go in one Table 2 row, `t3:unmapped` +(Sage's ruling), so their human messages still reach the Human column and +the human-per-closed ratio. + +**Mapping in the JSON.** For each seat, the T3 thread ids and titles that +made its row, and the unmapped thread ids and titles. Anyone checking a Gate +F result can then see which threads the row came from. + +A seat's row sums its Pi and T3 counts, and the JSON splits them by source. +Nothing is counted twice. Every T3 session today runs on `claudeAgent` or +`codex`, which don't write `.pi/state`, and Filbert found no T3 header in any +Pi log (6a record). -Excluded, with the reason stated in the README: +**Excluded,** with the reason stated in the README: - Imported threads (`thread_id` starting `import:`, events marked `historyImport`). They are partial copies of Claude Code sessions, not T3 traffic: 55 user messages in two threads here. - Deleted threads (`deleted_at` set). Across all projects there are 3, with 3 messages. Archived threads count. - -## What fails closed - -With the T3 source on, each of these refuses the report with exit 1, the -code the ledger already uses for unreadable session evidence. The report -never falls back to Pi logs alone. As with `--no-issues`, `--no-t3` turns the -source off, and the report then says T3 was not read. -- The database is missing, unreadable, or won't open read-only (including - the `-wal` without `-shm` case). This differs from the Pi reader, which - treats a missing `.pi` as no messages. A missing Pi directory means no Pi - seats ran here. A missing T3 database on this host means the path or T3 - changed, and a silent zero is the failure Gate F exists to prevent. -- A required table or column is missing. The reader checks `PRAGMA +- Threads in other T3 projects. Live, there is a project at `/home/jwoltje` + and a deleted one at `/mnt/storage/src`. A thread in either could work on + this repository and would not be counted. The workspace-root rule is still + the right one, but the README names this blind spot. + +## 4. What fails closed + +The source is on by default (Sage's ruling). `--no-t3` turns it off, and the +report then says T3 was not read. `--t3-db ` reads another database +file instead of `~/.t3/userdata/state.sqlite`. It exists for fixtures and +gets the same checks. + +Each of these refuses the report with exit 1, the code the ledger already +uses for unreadable session evidence. The report never falls back to Pi logs +alone. Where the database is missing or won't open, the message names +`--no-t3`. +- The database is missing or unreadable, or won't open read-only. That + includes a directory that isn't writable when SQLite needs to create + `-shm`, and a busy error after the timeout. The Pi reader treats a missing + `.pi` as no messages, and this departs from it on purpose. A missing Pi + directory means no Pi seats ran here. A missing T3 database on this host + means the path or T3 changed, and a silent zero is the failure Gate F + exists to prevent. +- `~/.t3`, `~/.t3/userdata` or `state.sqlite` is a symlink. With `--t3-db`, + the file and its directory are checked. The Pi reader checks every + ancestor too, but it skips symlinked entries. Skipping one named file + would be another silent zero, so this reader refuses. +- A table or column the counts need is missing. The reader checks `PRAGMA table_info` and names what's missing. This catches a T3 upgrade that changes the schema. - No project row, or more than one non-deleted row, for this repository root. - A counted row has a bad `role`, non-string `text`, or a `created_at` that doesn't parse. The Pi reader already refuses malformed JSONL and bad timestamps the same way. -- `state.sqlite` or `~/.t3/userdata` is a symlink. The Pi reader skips - symlinked entries instead. For one named file, skipping would be another - silent zero, so this reader refuses. - -The source never writes to the database. It never reads other files in -`~/.t3`, and it passes no message text beyond `messageKind` and -`issueNumbers`, the same rule as for Pi logs. The one outside effect is -SQLite's own: a WAL reader takes read locks in the `-shm` file, as T3's own -connections do. - -## Decisions for Jason - -1. The source is on by default, with `--no-t3` to turn it off. The other - choice is off by default with `--t3` to turn it on. I recommend on by - default, because Gate F exists to count these messages. -2. Unmapped threads get a `t3:unmapped` row. The other choice is to drop - them. I recommend the row. -3. The 14 free-text headers from 09-17 to 09-22 stay counted as human. Fixing - them would mean loosening the header grammar for history only, and I don't - recommend it. - -## Acceptance for the build - -- Fixture databases built with `node:sqlite` in a temp dir, in WAL mode: - seat threads and an unmapped thread; imported, deleted and archived - threads; all three header forms, uppercase classes included; a message - outside the date range; another project with the same seat titles. -- Each fail-closed case above has its own test, including a schema column - removed and `-wal` without `-shm`. One test opens a database whose newest - message is still in the WAL and counts it. Another reads a database closed - cleanly with no writer attached, which is the T3-stopped case. -- The class fix is proven against HEAD's `messageKind`: an uppercase class - counts as agent after the fix and as human before it. +- A header conflicts with the title mapping (section 3). + +The reader never reads other files in `~/.t3`. It passes no message text +beyond `messageKind`, `issueNumbers` and the header's `to:` role and id, the +same rule as for Pi logs. + +## 5. Rulings + +Sage ruled on the three questions R1 put to Jason, as lead calls: +1. The source is on by default. A missing or unreadable database exits 1, + and the message names `--no-t3`. +2. Unmapped threads get the `t3:unmapped` row. +3. The 14 free-text headers stay as recorded. They show only in the JSON + diagnostic. + +Sage also ruled that the class fix rides in this build, and that a missing +diagnostic table reads `unknown` (F5). + +## 6. Acceptance for the build + +**No test opens the real `~/.t3`.** Both places in +`packages/ledger/tests/ledger.test.mjs` that spawn `cli.mjs` (the shared +`run()` helper and the direct `spawnSync` at line 66) set `HOME` to the +fixture's temp directory. A test that forgets `--t3-db` or `--no-t3` then +finds no database and fails closed. The existing tests aren't about T3. Each +gets an empty fixture database at the fixture `HOME`'s default path, with +one project row for the fixture root. So they run with the source on, and +their expected rows don't change. One test asserts that a `HOME` with no +database exits 1 and names `--no-t3`. + +Fixture databases are built with `node:sqlite` in a temp directory, in WAL +mode: +- seat threads and an unmapped thread; imported, deleted and archived + threads; a message outside the date range; +- all three header forms, with uppercase classes in both the tmux preamble + and the T3 header; +- a thread with the same seat title in another project; +- a seat thread renamed to another seat, with an agent header to its own + id, which exits 1; +- a "Sagebrush" title, which stays unmapped; +- a thread titled "Researcher", which maps to the seat that has no thread + live. + +WAL states, each with its own test: +- The newest message is only in `-wal`, with the writer still attached (the + live-T3 case). It is counted. +- T3 stopped: the database closed cleanly with no writer. Counts are + correct, and the main file's bytes are unchanged afterwards. +- `-wal` without `-shm` in a writable directory: made by a child writer with + `wal_autocheckpoint=0` that is SIGKILLed, then `-shm` deleted. The WAL + rows are counted. +- `-wal` without `-shm` in a directory that isn't writable: exit 1, naming + `--no-t3`. Skipped when the tests run as root, where the mode bits don't + bind. +- The stopped case in a directory that isn't writable: exit 1, naming + `--no-t3`, as I measured it. If the build reads there instead, the builder + changes this test to assert correct counts and records the correction in + the BUILD-LOG entry. Skipped as root too. + +Also: +- Each other fail-closed case in section 4 has its own test, including a + removed schema column and each symlink. +- A missing `orchestration_events` gives `unknown` for the diagnostic and + the same counts. +- The class fix is proven against HEAD's `messageKind`. An uppercase class + in either preamble counts as agent after the fix and as human before it. +- The JSON lists each seat's threads and the unmapped threads. - A read against the live database gives the counts in this brief, allowing - for messages sent since. -- The ledger README's counting rules name the new source, the mapping rule - and the exclusions. + for messages sent since. It exits 0 with no header conflict. +- The ledger README's counting rules name the new source, both flags, the + mapping rule and the header check, and the exclusions. That includes the + symlinked-checkout case and the other-project blind spot. - No suite runs the ledger tests, so the BUILD-LOG entry names the test file. ## Not in scope @@ -194,5 +301,6 @@ - Claude Code transcripts (`~/.claude/projects`) and Codex sessions (`~/.codex/sessions`). T3's database already holds every message T3 delivered, so those files would only duplicate it. -- Any write to T3, any T3 API call, or anything that needs T3 running. +- Any T3 API call, anything that needs T3 running, and any write beyond + SQLite's own `-wal` and `-shm` handling. - Fixing the two tmux misclassifications Filbert found in 6a.