stack/agents/megavolt/work/queue-222/BUILD.md

Row 222 build: C2-03a fetch-only library sync

Row 222, #1723, reviewer Dewey. Brief: C2-03a in the case 2 plan (sage-1853, sage-1854), with sage-2021 and sage-2022 adding to it. Built on Herb's row 215 candidate source e333af4dc (base f8276c5eb, packet 14a03d10f, Herb's BUILD.md blob f80f21919d), then rebased with no conflicts onto 215's landing e54e05463 (sage-2099) as branch megavolt/row222-r2. The old branch megavolt/row222 is kept.

Between e333af4dc and e54e05463, two of 222's files changed. broker.mjs gained row 249's hostLineLeft lines, which 222 doesn't touch. stack-copy.sh gained row 66's tools/unslop-hook in COPY_PATHS, two lines below 222's packages/library/bin. In the range-diff, commit 1 differs only in that context line, and commits 2 and 3 are identical.

What it does

• packages/library/src/sync.mjs (new):
  • syncCommand('fetch', url, opts) builds the one git command.
  • sync(context, opts) fetches the configured branch into refs/remotes/library/<branch>, then publishes under the C2-04 library lock:
    • a fast-forward moves refs/heads/served;
    • a non-fast-forward is held as library-diverged until a human accepts the tip;
    • a failed fetch keeps serving the old commit and records lastError.
• packages/library/bin/askpass (new, sh) is git's GIT_ASKPASS helper. It answers only the two prompts for the configured host and user. For the password it reads the first line of the slot file. The token never reaches argv, env, git config, a URL, the state file or a log.
• packages/library/src/mapping.mjs: an optional user (the bot account name). Sync refuses an entry without it.
• Bus:
  • broker.mjs: library.sync is human-only. It takes {project, accept?}, and a held code becomes a recorded refusal.
  • runtime.mjs: a timer syncs every entry with a user at start and every 60 s.
  • webui/src/cli.mjs: the Console gets the call.
• scripts/stack-copy.sh copies packages/library/bin.
• Docs: library README "Sync" section, SURVEY-RUN.md step 5.

Scope rulings followed

• sage-1853/1854: the timer lives in the bus runtime, not a separate unit. The slot is <configDir>/secrets/<business>/library-gitea, mode 0600, owned by the bus uid.
• sage-2021: user is optional in the mapping and required for sync. Test runs wait for Sage's "clear".
• sage-2022 / Filbert N2 (#1716 comment 30071):
  • Sync never runs gc or prune and never fetches with --prune. maintenance.auto=false and gc.auto=0 come from SAFE.
  • Test "Filbert N2" holds a served commit, lets sync move served, then reads the held commit's blob.
  • The read path is unchanged, so readLibraryFile's whole-tree read stays (see Known costs).

Choices and tradeoffs

1. One sync per mirror, refused as library-sync-busy. The C2-04 lock covers publication only, and readers wait 5 s for it, so the fetch runs outside it. A module-level set refuses a second sync into the same mirror instead of letting two fetches race. A timer tick that finds its pass still running is skipped, not queued.
2. Problems are read before the lock. readLibrary over a large tree can take longer than a reader's 5 s lock wait. Problems belong to the tip alone, so they are computed outside and only published inside.
3. The served ref moves by compare-and-swap. update-ref refs/heads/served <tip> <old> (zero id on a first sync). If the state write then fails, the ref goes back, so the ref and the state file never disagree.
4. State written via a temp file beside the mirror. inspectMirror refuses unknown names inside the mirror, so the temp file goes in its parent directory (O_EXCL|O_NOFOLLOW, 0600). Then fsync, rename into the mirror, and fsync the directory.
5. Divergence is held, even through a later fast-forward. Once diverged, served stays put until accept. diverged.at keeps the first detection time, and diverged.commit follows the tip, because that is the commit accept must name. A tip that comes back to the served commit clears it.
6. accept starts the record over. It must equal the fetched tip (otherwise library-accept-stale). It writes a fresh record, which also clears a state left from another branch (readSyncState refuses a branch mismatch).
7. lastFetch is the last successful fetch. A failed fetch leaves it alone and sets lastError.
8. credential.helper= reset in argv. It stops a user or system helper from storing or offering the token. With GIT_TERMINAL_PROMPT=0 from gitEnv, askpass is the only source.
9. http.followRedirects=false. A redirect to another host would get the prompt for that host, which askpass refuses anyway. Not following redirects means no request reaches the other host at all (tested with a 302).
10. core.createObject=rename. A departure from the brief; see the next section.
11. Extra env keys. GIT_CEILING_DIRECTORIES (git never looks above the mirror) and MOSAIC_LIBRARY_HOST/USER/SLOT (the askpass inputs, no secret). The brief named GIT_ASKPASS only.
12. SAFE exported from packages/project/src/git.mjs. One word. Sync reuses the same safe -c set instead of copying it.
13. Row 199's fd-3 helper not reused. publishRemote (packages/seat/src/working.mjs) puts an inline credential.helper in argv and feeds the token to it on fd 3, so node reads the token first. The brief names GIT_ASKPASS. With askpass, node only lstats the slot and never opens it; the helper script reads it. The prompt checks do the same job as row 199's host check.
14. The timer's test hook. startBroker({librarySync: {command, intervalMs, configDir}}) is refused unless command is a function. A JSON boot config can't carry a function, so a boot file can't set it, and process.mjs needs no change.
15. No server.mjs edit. server.mjs awaits every verb in LIBRARY_VERBS with a 120 s socket timeout. Adding library.sync to LIBRARY_VERBS covers it.

Departure from the brief: core.createObject=rename

Sage accepted this departure (sage-2024, rev 2898), and Dewey rules on it in review. The fetch argv (syncCommand, packages/library/src/sync.mjs) is:

[...SAFE, '-c', 'protocol.https.allow=always', '-c', 'http.followRedirects=false', '-c', 'credential.helper=',
 '-c', 'core.createObject=rename', '-C', mirror, 'fetch', '--no-tags', '--no-write-fetch-head', '--no-recurse-submodules', '--end-of-options',
 parsed.href, `+refs/heads/${branch}:refs/remotes/library/${branch}`]

Why: by default git files each new object by link, then unlink. While that runs, the object has nlink 2. C2-04's inspectMirror refuses any file in the mirror with nlink other than 1, so a reader running during a fetch could be refused. With rename, git never makes a second link.

The argv test in sync.test.mjs pins the full sequence, including this flag. The race itself is not tested (sage-2024).

Known costs and residuals

• readLibraryFile reads the whole tree to return one file (C2-01 exclusions, Filbert's note on 215). 222 doesn't touch the read path, so it stays.
• The human transport times out at 5 s, a follow-up for Dewey to rule fix or defer. packages/bus/src/human-cli.mjs builds its Client (packages/bus/src/client.mjs) with the default timeout = 5000. A fetch can take up to the GIT_LIMITS ceiling (100 s), so a human call to library.sync against a slow remote can return outcome-unknown while the sync finishes, and the state file records the result. The outer layers allow longer: busCaller and humanCall in packages/webui/src/bus.mjs use 30 s, and the bus server awaits library verbs for 120 s. 222 has no Console route to fix. packages/webui/src/serve.mjs has no library route at all (library.file, library.status and library.sync are passed to startServer and unused there), and neither serve.mjs nor human-cli.mjs is in 222's paths. Follow-up: when the Console gains a library.sync route, give that path a 120 s client timeout.
• Problems are unbounded until the state hits 8 MiB. A tree with very many problems makes the state write fail with library-state-invalid, and served doesn't move. No cap per problem list was added.
• No behaviour test for the credential.helper= reset, a follow-up for Dewey (sage-2033). Only the argv test pins the reset: mutant 3 (drop it) was killed by that test alone. Today the reset is defence in depth. gitEnv() sets GIT_CONFIG_GLOBAL=/dev/null and GIT_CONFIG_NOSYSTEM, and C2-04's inspectMirror refuses any mirror config key outside its allowlist, so no config git reads can name a helper. gitEnv() also builds a fresh env, so the caller's env can't add one. A behaviour test would have to plant a helper through the test's command wrapper (for example GIT_CONFIG_COUNT/KEY/VALUE, the way the CA wrapper adds GIT_SSL_CAINFO), then show that git never calls it. Left as a named follow-up.
• stack-copy.test.mjs is not changed. It copies ls-files -co, which includes bin/askpass. The cli package is not mine.

Tests

• packages/library/tests/sync.test.mjs (new). It uses a fake Gitea over TLS (packages/seat/tests/fake-gitea.mjs), a made-up token and scratch roots, with no network. It covers the argv and env, the URL and user refusals, askpass prompts, first sync, fast-forward, Filbert N2, reader overlap, divergence and accept, accept across a branch change, a wrong token, a failed first fetch, a redirect, slot refusals, no user, and busy.

• packages/library/tests/mapping.test.mjs: the user field.

• packages/bus/tests/library-sync.test.mjs (new): bus refusals for agent, tester and reader; the human path; a refused boot hook; the timer.

• Run on 4a0e0ab3a under grant sage-2028, each file whole: sync 15/15, mapping 8/8, packages/bus/tests/library.test.mjs 6/6, library-sync 5/5. No log contains the token.

• Mutants, each against sync.test.mjs whole, all killed. After each one git diff was empty and askpass stayed 755.

  Mutant	Killed by
  protocol.https.allow -> protocol.allow=always	argv test
  drop http.followRedirects=false	argv test, 302 test
  drop the credential.helper= reset	argv test only (see Known costs)
  askpass answers any prompt	helper prompt test
  askpass prints the token for Username	helper test and 8 sync tests

  The last mutant's log holds the made-up fixture token, because the mutant printed it.

• After the rebase onto 215's landing: these 4 files again, plus packages/library/tests/mirror.test.mjs, packages/webui/tests/library-cli.test.mjs, packages/webui/tests/pinned-copy.test.mjs and packages/cli/tests/stack-copy.test.mjs.