# Row 222 build: C2-03a fetch-only library sync Row 222, #1723, reviewer Dewey. Brief: C2-03a in the case 2 plan (sage-1853, sage-1854), with sage-2021 and sage-2022 adding to it. Built on Herb's row 215 candidate source e333af4dc (base f8276c5eb, packet 14a03d10f, Herb's BUILD.md blob f80f21919d95), then rebased with no conflicts onto 215's landing e54e05463 (sage-2099) as branch megavolt/row222-r2. The old branch megavolt/row222 is kept. Between e333af4dc and e54e05463, two of 222's files changed. broker.mjs gained row 249's hostLineLeft lines, which 222 doesn't touch. stack-copy.sh gained row 66's `tools/unslop-hook` in COPY_PATHS, two lines below 222's `packages/library/bin`. In the range-diff, commit 1 differs only in that context line, and commits 2 and 3 are identical. ## What it does - `packages/library/src/sync.mjs` (new): - `syncCommand('fetch', url, opts)` builds the one git command. - `sync(context, opts)` fetches the configured branch into `refs/remotes/library/`, then publishes under the C2-04 library lock: - a fast-forward moves `refs/heads/served`; - a non-fast-forward is held as `library-diverged` until a human accepts the tip; - a failed fetch keeps serving the old commit and records `lastError`. - `packages/library/bin/askpass` (new, sh) is git's GIT_ASKPASS helper. It answers only the two prompts for the configured host and user. For the password it reads the first line of the slot file. The token never reaches argv, env, git config, a URL, the state file or a log. - `packages/library/src/mapping.mjs`: an optional `user` (the bot account name). Sync refuses an entry without it. - Bus: - `broker.mjs`: `library.sync` is human-only. It takes `{project, accept?}`, and a held code becomes a recorded refusal. - `runtime.mjs`: a timer syncs every entry with a `user` at start and every 60 s. - `webui/src/cli.mjs`: the Console gets the call. - `scripts/stack-copy.sh` copies `packages/library/bin`. - Docs: library README "Sync" section, SURVEY-RUN.md step 5. ## Scope rulings followed - sage-1853/1854: the timer lives in the bus runtime, not a separate unit. The slot is `/secrets//library-gitea`, mode 0600, owned by the bus uid. - sage-2021: `user` is optional in the mapping and required for sync. Test runs wait for Sage's "clear". - sage-2022 / Filbert N2 (#1716 comment 30071): - Sync never runs gc or prune and never fetches with `--prune`. `maintenance.auto=false` and `gc.auto=0` come from SAFE. - Test "Filbert N2" holds a served commit, lets sync move served, then reads the held commit's blob. - The read path is unchanged, so readLibraryFile's whole-tree read stays (see Known costs). ## Choices and tradeoffs 1. **One sync per mirror, refused as `library-sync-busy`.** The C2-04 lock covers publication only, and readers wait 5 s for it, so the fetch runs outside it. A module-level set refuses a second sync into the same mirror instead of letting two fetches race. A timer tick that finds its pass still running is skipped, not queued. 2. **Problems are read before the lock.** readLibrary over a large tree can take longer than a reader's 5 s lock wait. Problems belong to the tip alone, so they are computed outside and only published inside. 3. **The served ref moves by compare-and-swap.** `update-ref refs/heads/served ` (zero id on a first sync). If the state write then fails, the ref goes back, so the ref and the state file never disagree. 4. **State written via a temp file beside the mirror.** inspectMirror refuses unknown names inside the mirror, so the temp file goes in its parent directory (O_EXCL|O_NOFOLLOW, 0600). Then fsync, rename into the mirror, and fsync the directory. 5. **Divergence is held, even through a later fast-forward.** Once diverged, served stays put until `accept`. `diverged.at` keeps the first detection time, and `diverged.commit` follows the tip, because that is the commit accept must name. A tip that comes back to the served commit clears it. 6. **`accept` starts the record over.** It must equal the fetched tip (otherwise `library-accept-stale`). It writes a fresh record, which also clears a state left from another branch (readSyncState refuses a branch mismatch). 7. **`lastFetch` is the last successful fetch.** A failed fetch leaves it alone and sets `lastError`. 8. **`credential.helper=` reset in argv.** It stops a user or system helper from storing or offering the token. With GIT_TERMINAL_PROMPT=0 from gitEnv, askpass is the only source. 9. **`http.followRedirects=false`.** A redirect to another host would get the prompt for that host, which askpass refuses anyway. Not following redirects means no request reaches the other host at all (tested with a 302). 10. **`core.createObject=rename`.** A departure from the brief; see the next section. 11. **Extra env keys.** GIT_CEILING_DIRECTORIES (git never looks above the mirror) and MOSAIC_LIBRARY_HOST/USER/SLOT (the askpass inputs, no secret). The brief named GIT_ASKPASS only. 12. **`SAFE` exported from `packages/project/src/git.mjs`.** One word. Sync reuses the same safe `-c` set instead of copying it. 13. **Row 199's fd-3 helper not reused.** publishRemote (packages/seat/src/working.mjs) puts an inline `credential.helper` in argv and feeds the token to it on fd 3, so node reads the token first. The brief names GIT_ASKPASS. With askpass, node only lstats the slot and never opens it; the helper script reads it. The prompt checks do the same job as row 199's host check. 14. **The timer's test hook.** `startBroker({librarySync: {command, intervalMs, configDir}})` is refused unless `command` is a function. A JSON boot config can't carry a function, so a boot file can't set it, and process.mjs needs no change. 15. **No server.mjs edit.** server.mjs awaits every verb in LIBRARY_VERBS with a 120 s socket timeout. Adding `library.sync` to LIBRARY_VERBS covers it. ## Departure from the brief: `core.createObject=rename` Sage accepted this departure (sage-2024, rev 2898), and Dewey rules on it in review. The fetch argv (`syncCommand`, packages/library/src/sync.mjs) is: ``` [...SAFE, '-c', 'protocol.https.allow=always', '-c', 'http.followRedirects=false', '-c', 'credential.helper=', '-c', 'core.createObject=rename', '-C', mirror, 'fetch', '--no-tags', '--no-write-fetch-head', '--no-recurse-submodules', '--end-of-options', parsed.href, `+refs/heads/${branch}:refs/remotes/library/${branch}`] ``` Why: by default git files each new object by link, then unlink. While that runs, the object has nlink 2. C2-04's inspectMirror refuses any file in the mirror with nlink other than 1, so a reader running during a fetch could be refused. With `rename`, git never makes a second link. The argv test in sync.test.mjs pins the full sequence, including this flag. The race itself is not tested (sage-2024). ## Known costs and residuals - **readLibraryFile reads the whole tree to return one file** (C2-01 exclusions, Filbert's note on 215). 222 doesn't touch the read path, so it stays. - **The human transport times out at 5 s, a follow-up for Dewey to rule fix or defer.** packages/bus/src/human-cli.mjs builds its `Client` (packages/bus/src/client.mjs) with the default `timeout = 5000`. A fetch can take up to the GIT_LIMITS ceiling (100 s), so a human call to library.sync against a slow remote can return outcome-unknown while the sync finishes, and the state file records the result. The outer layers allow longer: busCaller and humanCall in packages/webui/src/bus.mjs use 30 s, and the bus server awaits library verbs for 120 s. 222 has no Console route to fix. packages/webui/src/serve.mjs has no library route at all (library.file, library.status and library.sync are passed to startServer and unused there), and neither serve.mjs nor human-cli.mjs is in 222's paths. Follow-up: when the Console gains a library.sync route, give that path a 120 s client timeout. - **Problems are unbounded until the state hits 8 MiB.** A tree with very many problems makes the state write fail with `library-state-invalid`, and served doesn't move. No cap per problem list was added. - **No behaviour test for the `credential.helper=` reset, a follow-up for Dewey (sage-2033).** Only the argv test pins the reset: mutant 3 (drop it) was killed by that test alone. Today the reset is defence in depth. gitEnv() sets GIT_CONFIG_GLOBAL=/dev/null and GIT_CONFIG_NOSYSTEM, and C2-04's inspectMirror refuses any mirror config key outside its allowlist, so no config git reads can name a helper. gitEnv() also builds a fresh env, so the caller's env can't add one. A behaviour test would have to plant a helper through the test's `command` wrapper (for example GIT_CONFIG_COUNT/KEY/VALUE, the way the CA wrapper adds GIT_SSL_CAINFO), then show that git never calls it. Left as a named follow-up. - **stack-copy.test.mjs is not changed.** It copies `ls-files -co`, which includes bin/askpass. The cli package is not mine. ## Tests - `packages/library/tests/sync.test.mjs` (new). It uses a fake Gitea over TLS (`packages/seat/tests/fake-gitea.mjs`), a made-up token and scratch roots, with no network. It covers the argv and env, the URL and user refusals, askpass prompts, first sync, fast-forward, Filbert N2, reader overlap, divergence and accept, accept across a branch change, a wrong token, a failed first fetch, a redirect, slot refusals, no user, and busy. - `packages/library/tests/mapping.test.mjs`: the `user` field. - `packages/bus/tests/library-sync.test.mjs` (new): bus refusals for agent, tester and reader; the human path; a refused boot hook; the timer. - Run on 4a0e0ab3a under grant sage-2028, each file whole: sync 15/15, mapping 8/8, packages/bus/tests/library.test.mjs 6/6, library-sync 5/5. No log contains the token. - Mutants, each against sync.test.mjs whole, all killed. After each one git diff was empty and askpass stayed 755. | Mutant | Killed by | |---|---| | protocol.https.allow -> protocol.allow=always | argv test | | drop http.followRedirects=false | argv test, 302 test | | drop the credential.helper= reset | argv test only (see Known costs) | | askpass answers any prompt | helper prompt test | | askpass prints the token for Username | helper test and 8 sync tests | The last mutant's log holds the made-up fixture token, because the mutant printed it. - After the rebase onto 215's landing: these 4 files again, plus packages/library/tests/mirror.test.mjs, packages/webui/tests/library-cli.test.mjs, packages/webui/tests/pinned-copy.test.mjs and packages/cli/tests/stack-copy.test.mjs.