Compare commits

...
15 changed files with 873 additions and 14 deletions
+166
View File
@@ -0,0 +1,166 @@
# Row 222 build: C2-03a fetch-only library sync
Row 222, #1723, reviewer Dewey. Brief: C2-03a in the case 2 plan (sage-1853, sage-1854), with
sage-2021 and sage-2022 adding to it. Built on Herb's row 215 candidate source e333af4dc (base
f8276c5eb, packet 14a03d10f, Herb's BUILD.md blob f80f21919d95), then rebased with no conflicts
onto 215's landing e54e05463 (sage-2099) as branch megavolt/row222-r2. The old branch
megavolt/row222 is kept.
Between e333af4dc and e54e05463, two of 222's files changed. broker.mjs gained row 249's
hostLineLeft lines, which 222 doesn't touch. stack-copy.sh gained row 66's `tools/unslop-hook`
in COPY_PATHS, two lines below 222's `packages/library/bin`. In the range-diff, commit 1 differs
only in that context line, and commits 2 and 3 are identical.
## What it does
- `packages/library/src/sync.mjs` (new):
- `syncCommand('fetch', url, opts)` builds the one git command.
- `sync(context, opts)` fetches the configured branch into `refs/remotes/library/<branch>`,
then publishes under the C2-04 library lock:
- a fast-forward moves `refs/heads/served`;
- a non-fast-forward is held as `library-diverged` until a human accepts the tip;
- a failed fetch keeps serving the old commit and records `lastError`.
- `packages/library/bin/askpass` (new, sh) is git's GIT_ASKPASS helper. It answers only the two
prompts for the configured host and user. For the password it reads the first line of the slot
file. The token never reaches argv, env, git config, a URL, the state file or a log.
- `packages/library/src/mapping.mjs`: an optional `user` (the bot account name). Sync refuses an
entry without it.
- Bus:
- `broker.mjs`: `library.sync` is human-only. It takes `{project, accept?}`, and a held code
becomes a recorded refusal.
- `runtime.mjs`: a timer syncs every entry with a `user` at start and every 60 s.
- `webui/src/cli.mjs`: the Console gets the call.
- `scripts/stack-copy.sh` copies `packages/library/bin`.
- Docs: library README "Sync" section, SURVEY-RUN.md step 5.
## Scope rulings followed
- sage-1853/1854: the timer lives in the bus runtime, not a separate unit. The slot is
`<configDir>/secrets/<business>/library-gitea`, mode 0600, owned by the bus uid.
- sage-2021: `user` is optional in the mapping and required for sync. Test runs wait for Sage's
"clear".
- sage-2022 / Filbert N2 (#1716 comment 30071):
- Sync never runs gc or prune and never fetches with `--prune`. `maintenance.auto=false` and
`gc.auto=0` come from SAFE.
- Test "Filbert N2" holds a served commit, lets sync move served, then reads the held commit's
blob.
- The read path is unchanged, so readLibraryFile's whole-tree read stays (see Known costs).
## Choices and tradeoffs
1. **One sync per mirror, refused as `library-sync-busy`.** The C2-04 lock covers publication
only, and readers wait 5 s for it, so the fetch runs outside it. A module-level set refuses a
second sync into the same mirror instead of letting two fetches race. A timer tick that finds
its pass still running is skipped, not queued.
2. **Problems are read before the lock.** readLibrary over a large tree can take longer than a
reader's 5 s lock wait. Problems belong to the tip alone, so they are computed outside and
only published inside.
3. **The served ref moves by compare-and-swap.** `update-ref refs/heads/served <tip> <old>`
(zero id on a first sync). If the state write then fails, the ref goes back, so the ref and
the state file never disagree.
4. **State written via a temp file beside the mirror.** inspectMirror refuses unknown names
inside the mirror, so the temp file goes in its parent directory (O_EXCL|O_NOFOLLOW, 0600).
Then fsync, rename into the mirror, and fsync the directory.
5. **Divergence is held, even through a later fast-forward.** Once diverged, served stays put
until `accept`. `diverged.at` keeps the first detection time, and `diverged.commit` follows
the tip, because that is the commit accept must name. A tip that comes back to the served
commit clears it.
6. **`accept` starts the record over.** It must equal the fetched tip (otherwise
`library-accept-stale`). It writes a fresh record, which also clears a state left from
another branch (readSyncState refuses a branch mismatch).
7. **`lastFetch` is the last successful fetch.** A failed fetch leaves it alone and sets
`lastError`.
8. **`credential.helper=` reset in argv.** It stops a user or system helper from storing or
offering the token. With GIT_TERMINAL_PROMPT=0 from gitEnv, askpass is the only source.
9. **`http.followRedirects=false`.** A redirect to another host would get the prompt for that
host, which askpass refuses anyway. Not following redirects means no request reaches the
other host at all (tested with a 302).
10. **`core.createObject=rename`.** A departure from the brief; see the next section.
11. **Extra env keys.** GIT_CEILING_DIRECTORIES (git never looks above the mirror) and
MOSAIC_LIBRARY_HOST/USER/SLOT (the askpass inputs, no secret). The brief named
GIT_ASKPASS only.
12. **`SAFE` exported from `packages/project/src/git.mjs`.** One word. Sync reuses the same
safe `-c` set instead of copying it.
13. **Row 199's fd-3 helper not reused.** publishRemote (packages/seat/src/working.mjs) puts an
inline `credential.helper` in argv and feeds the token to it on fd 3, so node reads the token
first. The brief names GIT_ASKPASS. With askpass, node only lstats the slot and never opens
it; the helper script reads it. The prompt checks do the same job as row 199's host check.
14. **The timer's test hook.** `startBroker({librarySync: {command, intervalMs, configDir}})` is
refused unless `command` is a function. A JSON boot config can't carry a function, so a boot
file can't set it, and process.mjs needs no change.
15. **No server.mjs edit.** server.mjs awaits every verb in LIBRARY_VERBS with a 120 s socket
timeout. Adding `library.sync` to LIBRARY_VERBS covers it.
## Departure from the brief: `core.createObject=rename`
Sage accepted this departure (sage-2024, rev 2898), and Dewey rules on it in review. The fetch argv
(`syncCommand`, packages/library/src/sync.mjs) is:
```
[...SAFE, '-c', 'protocol.https.allow=always', '-c', 'http.followRedirects=false', '-c', 'credential.helper=',
'-c', 'core.createObject=rename', '-C', mirror, 'fetch', '--no-tags', '--no-write-fetch-head', '--no-recurse-submodules', '--end-of-options',
parsed.href, `+refs/heads/${branch}:refs/remotes/library/${branch}`]
```
Why: by default git files each new object by link, then unlink. While that runs, the object has
nlink 2. C2-04's inspectMirror refuses any file in the mirror with nlink other than 1, so a reader
running during a fetch could be refused. With `rename`, git never makes a second link.
The argv test in sync.test.mjs pins the full sequence, including this flag. The race itself is not
tested (sage-2024).
## Known costs and residuals
- **readLibraryFile reads the whole tree to return one file** (C2-01 exclusions, Filbert's
note on 215). 222 doesn't touch the read path, so it stays.
- **The human transport times out at 5 s, a follow-up for Dewey to rule fix or defer.**
packages/bus/src/human-cli.mjs builds its `Client` (packages/bus/src/client.mjs) with the default
`timeout = 5000`. A fetch can take up to the GIT_LIMITS ceiling (100 s), so a human call to
library.sync against a slow remote can return outcome-unknown while the sync finishes, and the
state file records the result. The outer layers allow longer: busCaller and humanCall in
packages/webui/src/bus.mjs use 30 s, and the bus server awaits library verbs for 120 s.
222 has no Console route to fix. packages/webui/src/serve.mjs has no library route at all
(library.file, library.status and library.sync are passed to startServer and unused there), and
neither serve.mjs nor human-cli.mjs is in 222's paths. Follow-up: when the Console gains a
library.sync route, give that path a 120 s client timeout.
- **Problems are unbounded until the state hits 8 MiB.** A tree with very many problems makes
the state write fail with `library-state-invalid`, and served doesn't move. No cap per
problem list was added.
- **No behaviour test for the `credential.helper=` reset, a follow-up for Dewey (sage-2033).**
Only the argv test pins the reset: mutant 3 (drop it) was killed by that test alone. Today the
reset is defence in depth. gitEnv() sets GIT_CONFIG_GLOBAL=/dev/null and GIT_CONFIG_NOSYSTEM,
and C2-04's inspectMirror refuses any mirror config key outside its allowlist, so no config
git reads can name a helper. gitEnv() also builds a fresh env, so the caller's env can't add
one. A behaviour test would have to plant a helper through the test's `command` wrapper (for
example GIT_CONFIG_COUNT/KEY/VALUE, the way the CA wrapper adds GIT_SSL_CAINFO), then show
that git never calls it. Left as a named follow-up.
- **stack-copy.test.mjs is not changed.** It copies `ls-files -co`, which includes bin/askpass.
The cli package is not mine.
## Tests
- `packages/library/tests/sync.test.mjs` (new). It uses a fake Gitea over TLS
(`packages/seat/tests/fake-gitea.mjs`), a made-up token and scratch roots, with no network.
It covers the argv and env, the URL and user refusals, askpass prompts, first sync,
fast-forward, Filbert N2, reader overlap, divergence and accept, accept across a branch
change, a wrong token, a failed first fetch, a redirect, slot refusals, no user, and busy.
- `packages/library/tests/mapping.test.mjs`: the `user` field.
- `packages/bus/tests/library-sync.test.mjs` (new): bus refusals for agent, tester and reader;
the human path; a refused boot hook; the timer.
- Run on 4a0e0ab3a under grant sage-2028, each file whole: sync 15/15, mapping 8/8,
packages/bus/tests/library.test.mjs 6/6, library-sync 5/5. No log contains the token.
- Mutants, each against sync.test.mjs whole, all killed. After each one git diff was empty
and askpass stayed 755.
| Mutant | Killed by |
|---|---|
| protocol.https.allow -> protocol.allow=always | argv test |
| drop http.followRedirects=false | argv test, 302 test |
| drop the credential.helper= reset | argv test only (see Known costs) |
| askpass answers any prompt | helper prompt test |
| askpass prints the token for Username | helper test and 8 sync tests |
The last mutant's log holds the made-up fixture token, because the mutant printed it.
- After the rebase onto 215's landing: these 4 files again, plus
packages/library/tests/mirror.test.mjs, packages/webui/tests/library-cli.test.mjs,
packages/webui/tests/pinned-copy.test.mjs and packages/cli/tests/stack-copy.test.mjs.
+2 -1
View File
@@ -61,7 +61,8 @@ it out, and the mirror's refs belong to the bus. If `HEAD` names no commit, stop
5. When Jason has a draft mapping, run it again with `--mapping`. The file holds the library entry
alone, `{"kind": "library", "remote": ..., "branch": ..., ...}`, without the business file or the
`"projects"` wrapper around it. `remote` and `branch` are required even though the survey doesn't
fetch. `mosaic` must be able to read the file. Set `vaultPrefix` to her books repo's folder name
fetch. `user` is optional here: it is the bot account name from design step 3, required for sync.
`mosaic` must be able to read the file. Set `vaultPrefix` to her books repo's folder name
in her vault. Without it, a link with the full path from the top of her vault resolves to nothing,
and the setup sheet tells her to write links that way.
+14 -6
View File
@@ -88,7 +88,8 @@ export const TASK_PAGE_MAX = 200;
// Row 148 (#1642): the human-only verbs behind the Console's document editor. The save is the one
// request allowed past the ordinary 64 KiB envelope (server.mjs and client.mjs), up to its content cap.
export const PROJECT_FILE_VERBS = Object.freeze(['project.files', 'project.file', 'project.file.save']);
export const LIBRARY_VERBS = Object.freeze(['library.file', 'library.status']);
// C2-03a: library.sync is the human's fetch, and with `accept` the one way past library-diverged.
export const LIBRARY_VERBS = Object.freeze(['library.file', 'library.status', 'library.sync']);
export const PROJECT_SAVE_VERB = 'project.file.save';
export const PROJECT_SAVE_REQUEST_MAX = 1024 * 1024 + 64 * 1024;
export const WEB_VERBS =Object.freeze(['web.search', 'web.fetch']);
@@ -2452,13 +2453,20 @@ export class Broker {
const { verb, args } = request;
try {
this.#human(s);
const fields = verb === 'library.file' ? ['project', 'path'] : ['project'];
keys(args, fields, fields);
const fields = verb === 'library.file' ? ['project', 'path'] : verb === 'library.sync' ? ['project', 'accept'] : ['project'];
keys(args, fields, verb === 'library.sync' ? ['project'] : fields);
if (!this.#library?.readFile || !this.#library?.status) fail('project-not-configured');
if (verb === 'library.sync' && !this.#library.sync) fail('project-not-configured');
const context = { business: this.#business(s.business), project: args.project };
const result = verb === 'library.file'
? await this.#library.readFile({ ...context, path: args.path })
: await this.#library.status(context);
let result;
if (verb === 'library.file') result = await this.#library.readFile({ ...context, path: args.path });
else if (verb === 'library.status') result = await this.#library.status(context);
else {
// A fetch that did not move served still answers with the code, so the refusal is recorded.
const { state, code } = await this.#library.sync(context, { accept: args.accept ?? null });
if (code !== null) fail(code);
result = { project: args.project, ...state };
}
this.#secretCheck(result);
return result;
} catch (error) {
+42 -1
View File
@@ -81,8 +81,16 @@ export async function startBroker({
// Row 199: the tests' scratch CA for publish's git (`-c http.sslCAInfo=...`). process.mjs drops
// it from the boot config, so only an in-process caller sets it.
publishTls = [],
// C2-03a: the tests' stand-in for syncCommand, a shorter timer and a scratch config dir for the
// slot, { command, intervalMs, configDir }. A boot config is JSON and can't carry a function, so
// a librarySync without one is refused.
librarySync = null,
}) {
let store, credentials, server, adapter;
if (librarySync !== null && (typeof librarySync?.command !== 'function'
|| (Object.hasOwn(librarySync, 'intervalMs') && !(Number.isSafeInteger(librarySync.intervalMs) && librarySync.intervalMs > 0))
|| (Object.hasOwn(librarySync, 'configDir') && typeof librarySync.configDir !== 'string')))
throw new BusError('invalid-host-request');
let store, credentials, server, adapter, syncTimer = null, syncPass = null;
try {
const references = {};
for (const [business, b] of Object.entries(businesses)) {
@@ -134,6 +142,13 @@ export async function startBroker({
for (const business of Object.values(businesses))
for (const [project, entry] of Object.entries(business.projects ?? {}))
if (entry.kind === 'library') await libraryModule.ensureMirror({ business, project, dataRoot });
// C2-03a: the slot is <configDir>/secrets/<business>/library-gitea (design 10 step 4). Sync only
// checks that it is there; the askpass helper reads it.
const slotDir = librarySync?.configDir ?? organization?.dir ?? null;
const syncOptions = business => ({
slot: slotDir ? join(slotDir, 'secrets', business.id, libraryModule.SLOT) : null,
...(librarySync ? { command: librarySync.command } : {}),
});
const broker = new Broker({
hostGrant: runRoot ? hostGrant({ configDir: organization.dir, runRoot }) : null,
// Row 235 (#1741, r6 X4, X5): the grant executor's claim, register and issue, never use().
@@ -145,6 +160,7 @@ export async function startBroker({
library: {
readFile: context => libraryModule.readLibraryFile({ ...context, dataRoot }),
status: context => libraryModule.libraryStatus({ ...context, dataRoot }),
sync: (context, { accept }) => libraryModule.sync({ ...context, dataRoot }, { ...syncOptions(context.business), accept }),
LibraryError: libraryModule.LibraryError,
},
project: {
@@ -340,6 +356,30 @@ export async function startBroker({
.filter((c) => c.instance === business + '/' + role)
.map(({ service, date, state }) => ({ service, date, state }));
}
// C2-03a: each library entry with a sync user fetches once at start, then every 60 s. A pass
// still running skips the tick. Only codes are logged, never git's output.
const timed = Object.values(businesses).flatMap(business => Object.entries(business.projects ?? {})
.filter(([, entry]) => entry.kind === 'library' && Object.hasOwn(entry, 'user')).map(([project]) => ({ business, project })));
const syncLog = entry => process.stderr.write(`library: ${JSON.stringify(entry)}\n`);
const syncAll = async () => {
for (const { business, project } of timed) {
try {
const { code } = await libraryModule.sync({ business, project, dataRoot }, syncOptions(business));
if (code !== null) syncLog({ event: 'library-sync-held', business: business.id, project, code });
} catch (e) {
if (e?.code === 'library-sync-busy') continue;
syncLog({ event: 'library-sync-refused', business: business.id, project,
code: e instanceof libraryModule.LibraryError ? e.code : 'library-sync-failed' });
}
}
};
const tick = () => { if (!syncPass) syncPass = syncAll().finally(() => { syncPass = null; }); };
if (timed.length) {
tick();
syncTimer = setInterval(tick, librarySync?.intervalMs ?? libraryModule.SYNC_INTERVAL_MS);
syncTimer.unref();
}
const stopSync = async () => { clearInterval(syncTimer); syncTimer = null; await syncPass; };
return {
broker,
credentials,
@@ -373,6 +413,7 @@ export async function startBroker({
launches: bound,
readers: readCaps,
async close() {
await stopSync();
await server.close();
await adapter?.close?.();
store.close();
+109
View File
@@ -0,0 +1,109 @@
// C2-03a: library.sync on the bus and the bus host's 60-second timer, with made-up local data.
import test from 'node:test';
import assert from 'node:assert/strict';
import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { Store } from '../src/store.mjs';
import { Broker, LIBRARY_VERBS } from '../src/broker.mjs';
import { startBroker } from '../src/runtime.mjs';
import { gitEnv } from '../../seat/src/checkout.mjs';
import { mappingEntry, repo } from '../../library/tests/helpers.mjs';
import * as library from '../../library/src/index.mjs';
const TOKEN = 'fixture-token-9b2e-made-up-0043';
const STATE = { version: 1, branch: 'main', served: 'a'.repeat(40), lastFetch: '2026-10-21T12:00:00.000Z', lastError: null, diverged: null, problems: [] };
function world(t, adapter) {
const root = mkdtempSync(join(tmpdir(), 'bus-library-sync-')), store = new Store(root);
t.after(() => { store.close(); rmSync(root, { recursive: true, force: true }); });
const business = { id: 'acme', human: 'jason', vars: { 'bus.tester': true },
projects: { books: { ...mappingEntry(), user: 'books-bot' } }, arbiters: { technical: 'coder', delivery: 'coder' }, roles: { coder: { authority: { withinRole: [], crossRole: [] } } } };
const broker = new Broker({ store, businesses: { acme: business }, library: adapter });
const human = broker.bindHuman({ business: 'acme', human: 'jason', via: 'cli', outsideAgent: true });
const agent = broker.bindLaunch({ business: 'acme', role: 'coder', run: 'library-sync-agent', harness: 'pi' });
const reader = broker.bindReader({ business: 'acme' }), tester = broker.bindTester({ business: 'acme' });
const call = (cap, verb, args = {}) => Promise.resolve().then(() => broker.request(cap, { verb, args }));
const refusals = () => store.all("SELECT body FROM events WHERE kind='action.refused'").map(row => JSON.parse(row.body).code);
return { store, broker, human, agent, reader, tester, call, refusals };
}
const adapter = (sync) => ({ readFile: () => {}, status: () => {}, sync, LibraryError: library.LibraryError });
test('C2-03a library.sync: every agent, the tester and a reader are refused before the adapter runs', async t => {
let touched = 0;
const f = world(t, adapter(async () => { touched++; return { state: STATE, code: null }; }));
await assert.rejects(f.call(f.agent, 'library.sync', { project: 'books' }), { code: 'human-required' });
await assert.rejects(f.call(f.tester, 'library.sync', { project: 'books' }), { code: 'human-required' });
await assert.rejects(f.call(f.reader, 'library.sync', { project: 'books' }), { code: 'read-only' });
assert.equal(touched, 0);
assert.ok(LIBRARY_VERBS.includes('library.sync'), 'the server awaits it like the other library verbs');
});
test('C2-03a library.sync: the human fetches, accept passes through, and a held sync is a recorded refusal', async t => {
const seen = [];
let code = null;
const f = world(t, adapter(async (context, options) => { seen.push({ context, options }); return { state: STATE, code }; }));
assert.deepEqual(await f.call(f.human, 'library.sync', { project: 'books' }), { project: 'books', ...STATE });
assert.equal(seen[0].context.business.id, 'acme');
assert.equal(seen[0].context.project, 'books');
assert.deepEqual(seen[0].options, { accept: null });
await f.call(f.human, 'library.sync', { project: 'books', accept: 'b'.repeat(40) });
assert.deepEqual(seen[1].options, { accept: 'b'.repeat(40) });
for (const held of ['library-diverged', 'library-fetch-failed', 'library-credential-missing']) {
code = held;
await assert.rejects(f.call(f.human, 'library.sync', { project: 'books' }), { code: held });
}
assert.deepEqual(f.refusals(), ['library-diverged', 'library-fetch-failed', 'library-credential-missing']);
for (const args of [{}, { project: 'books', extra: 1 }, { project: 'books', commit: 'b'.repeat(40) }])
await assert.rejects(f.call(f.human, 'library.sync', args), { code: 'invalid-request' });
});
test('C2-03a library.sync: a LibraryError keeps its code and a host without sync is not configured', async t => {
const f = world(t, adapter(async () => { throw new library.LibraryError('library-accept-stale', 'made up'); }));
await assert.rejects(f.call(f.human, 'library.sync', { project: 'books', accept: 'c'.repeat(40) }), { code: 'library-accept-stale' });
const none = world(t, adapter(undefined));
await assert.rejects(none.call(none.human, 'library.sync', { project: 'books' }), { code: 'project-not-configured' });
});
test('C2-03a runtime: a librarySync from a boot config, with no function, is refused', async t => {
const root = mkdtempSync(join(tmpdir(), 'bus-library-boot-'));
t.after(() => rmSync(root, { recursive: true, force: true }));
for (const librarySync of [{}, { intervalMs: 1 }, { command: 'git' }, { command: () => {}, intervalMs: 0 }, { command: () => {}, configDir: 7 }])
await assert.rejects(startBroker({ dataRoot: root, businesses: {}, librarySync }), { code: 'invalid-host-request' });
});
test('C2-03a runtime: the timer syncs each entry with a user at start and on each tick, logs codes only, and stops on close', { timeout: 20000 }, async t => {
const f = repo(t), dataRoot = join(f.root, 'data'), configDir = join(f.root, 'config');
mkdirSync(dataRoot, { mode: 0o700 });
mkdirSync(join(configDir, 'secrets', 'acme'), { recursive: true, mode: 0o700 });
writeFileSync(join(configDir, 'secrets', 'acme', library.SLOT), TOKEN + '\n', { mode: 0o600 });
const business = { id: 'acme', human: 'jason', arbiters: { technical: 'coder', delivery: 'coder' }, roles: { coder: { authority: { withinRole: [], crossRole: [] } } },
projects: { books: { ...mappingEntry(), user: 'books-bot' }, quiet: mappingEntry() } };
const calls = [];
// A git that succeeds without fetching: the remote-tracking ref stays absent, so each pass fails.
const command = (kind, url, opts) => { calls.push({ kind, url, opts }); return { argv: ['--version'], env: gitEnv() }; };
const lines = [], write = process.stderr.write;
process.stderr.write = function (chunk, ...rest) {
if (String(chunk).startsWith('library: ')) { lines.push(String(chunk)); return true; }
return write.call(this, chunk, ...rest);
};
t.after(() => { process.stderr.write = write; });
const runtime = await startBroker({ dataRoot, businesses: { acme: business }, librarySync: { command, intervalMs: 50, configDir } });
const deadline = Date.now() + 10000;
while (calls.length < 2 && Date.now() < deadline) await new Promise(r => setTimeout(r, 10));
await runtime.close();
const after = calls.length;
assert.ok(after >= 2, 'a pass at start and at least one tick');
assert.ok(calls.every(c => c.kind === 'fetch' && c.opts.slot === join(configDir, 'secrets', 'acme', library.SLOT) && c.opts.user === 'books-bot'));
assert.equal(new Set(calls.map(c => c.opts.mirror)).size, 1, 'the entry with no user is never synced');
await new Promise(r => setTimeout(r, 200));
assert.equal(calls.length, after, 'no pass after close');
assert.ok(lines.length >= 1);
for (const line of lines) {
assert.deepEqual(JSON.parse(line.slice('library: '.length)), { event: 'library-sync-held', business: 'acme', project: 'books', code: 'library-fetch-failed' });
assert.equal(line.includes(TOKEN), false);
}
const mirror = library.mirrorPath({ business, project: 'books', dataRoot });
assert.equal((await library.readSyncState(mirror, { branch: 'main' })).lastError.code, 'library-fetch-failed');
});
+37
View File
@@ -17,6 +17,7 @@ deep-frozen copy. `where` names the entry in messages. `fail(code, message)` def
"kind": "library",
"remote": "https://git.example.invalid/her/books.git",
"branch": "main",
"user": "books-bot",
"vaultPrefix": "books",
"types": { "Characters": "character", "Places": "place", "Research": "research" },
"manuscript": ["Chapters"],
@@ -27,6 +28,8 @@ deep-frozen copy. `where` names the entry in messages. `fail(code, message)` def
```
- `remote` is https with no credentials, query or fragment. `branch` is a plain ref name.
- `user` is the bot account name from design step 3, required for sync. It is optional in the
mapping and checked with row 199's account name pattern, `[A-Za-z0-9_.-]`, 1 to 100 characters.
- `vaultPrefix` is optional, one folder name.
- `types` maps a folder name to `character`, `place`, `research`, `tracker` or `summary`.
`manuscript` lists folder names and shares none with `types`.
@@ -128,6 +131,40 @@ Exit codes follow `packages/project`: 0 done, 2 the repo can't be read (`library
`library-commit`), 3 a bad mapping, 4 usage. `scripts/mosaic` sends `library survey` here. Other
library verbs will be bus verbs.
## Sync
`sync(context, { slot, accept })` (C2-03a, design 7.2) fetches the entry's `branch` from its `remote`
into `refs/remotes/library/<branch>` of the C2-04 mirror, then publishes under the library lock:
| Fetched tip | `served` | State |
|---|---|---|
| a fast-forward, or the first sync | moves to the tip | `problems` rebuilt from the tip |
| the served commit | stays | `lastFetch` set |
| not a fast-forward | stays | `diverged` holds the tip, `library-diverged`, until `accept` |
| fetch failed, slot missing | stays | `lastError` holds `library-fetch-failed` or `library-credential-missing` |
`lastFetch` is the last fetch that succeeded. Once diverged, `served` stays put until a human runs
`library.sync` with `accept` set to the fetched tip, even if a later tip is a fast-forward again. A
tip back at the served commit clears it. A
stale `accept` refuses `library-accept-stale`. `accept` also starts the state over, so it is how a
changed `branch` gets a fresh record. An entry with no `user` refuses `library-remote-invalid` and
runs no git.
`syncCommand('fetch', url, { mirror, branch, user, slot })` gives the one argv and env. The argv is
`SAFE` from `packages/project/src/git.mjs`, then `-c protocol.https.allow=always -c
http.followRedirects=false -c credential.helper= -c core.createObject=rename`, then `-C <mirror>` and
the fetch. `core.createObject=rename` is there because git's default files an object by link then
unlink, and a reader's mirror check refuses a file with two links. The env is
`gitEnv()` with `LC_ALL=C`, `GIT_ASKPASS` naming `bin/askpass`, and the host, user and slot path. The
helper answers only `Username for 'https://<host>': ` and `Password for 'https://<user>@<host>': `,
the second from the slot's first line, and exits 1 on any other prompt. Sync checks that the slot is
a private regular file and never opens it. The token is in no argv, env, URL, config or log.
Sync never runs gc or prune. A file read takes `served` under the lock and reads its objects after it
lets go, so an object must outlive the next move of `served`. The bus host runs `sync` for every entry
with a `user`, once at start and then every 60 seconds; the slot is
`<configDir>/secrets/<business>/library-gitea`. `library.sync` is the human's bus verb for the same.
## Tests
```sh
+17
View File
@@ -0,0 +1,17 @@
#!/bin/sh
# C2-03a: git's GIT_ASKPASS for library sync (packages/library/src/sync.mjs). It answers only the
# Username and Password prompts for the configured host and user, and exits 1 on anything else.
# The env names the slot path, never the token. printf is a shell builtin, so the token is in no argv.
host=$MOSAIC_LIBRARY_HOST user=$MOSAIC_LIBRARY_USER slot=$MOSAIC_LIBRARY_SLOT
test -n "$host" && test -n "$user" && test -n "$slot" || exit 1
case $host in *[!a-z0-9.:-]*) exit 1 ;; esac
case $user in *[!A-Za-z0-9_.-]*) exit 1 ;; esac
case $1 in
"Username for 'https://$host': ") printf '%s\n' "$user" ;;
"Password for 'https://$user@$host': ")
test -f "$slot" && test ! -L "$slot" || exit 1
IFS= read -r token < "$slot" || test -n "$token" || exit 1
test -n "$token" || exit 1
printf '%s\n' "$token" ;;
*) exit 1 ;;
esac
+1
View File
@@ -7,3 +7,4 @@ export { readLibrary, listTree, readTexts, parseFile, parseFrontMatter, headComm
export { renderItems, KINDS, NOTICES, UNREADABLE } from './render.mjs';
export { survey, formatSurvey, surveyMain } from './survey.mjs';
export { SYNC_STATE_FILE, mirrorPath, ensureMirror, withLibraryLock, servedCommit, readSyncState, libraryStatus, readLibraryFile } from './mirror.mjs';
export { syncCommand, sync, ASKPASS, SLOT, SYNC_INTERVAL_MS } from './sync.mjs';
+6 -1
View File
@@ -7,6 +7,8 @@ import { raise } from './problems.mjs';
export const TYPES = Object.freeze(['character', 'place', 'research', 'tracker', 'summary']);
const ID = /^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$/;
const BRANCH = /^[A-Za-z0-9][A-Za-z0-9._/-]{0,127}$/;
// C2-03a (sage-2021): the bot account sync answers git's Username prompt with. Row 199's NAME.
const USER = /^[A-Za-z0-9_.-]{1,100}$/;
const CONSTRAINT_KEYS = ['exclude', 'pin'];
// A folder segment from her vault: spaces and capitals are fine, separators, control characters
@@ -36,7 +38,7 @@ export function checkMapping(entry, where = 'library', { fail = raise } = {}) {
};
object(entry, '');
only(entry, ['kind', 'remote', 'branch', 'vaultPrefix', 'universes', 'types', 'manuscript'], 'entry');
only(entry, ['kind', 'remote', 'branch', 'user', 'vaultPrefix', 'universes', 'types', 'manuscript'], 'entry');
if (entry.kind !== 'library') bad('kind must be "library"');
let url;
try { url = new URL(entry.remote); } catch { bad('remote must be an https:// URL'); }
@@ -45,6 +47,8 @@ export function checkMapping(entry, where = 'library', { fail = raise } = {}) {
if (typeof entry.branch !== 'string' || !BRANCH.test(entry.branch) || entry.branch.includes('..')
|| entry.branch.includes('//') || entry.branch.endsWith('/') || entry.branch.endsWith('.lock'))
bad('branch must be a plain branch name');
if (Object.hasOwn(entry, 'user') && (typeof entry.user !== 'string' || !USER.test(entry.user)))
bad('user must be a plain account name');
if (Object.hasOwn(entry, 'vaultPrefix') && !segment(entry.vaultPrefix)) bad('vaultPrefix must be one folder name');
object(entry.types, 'types');
@@ -121,6 +125,7 @@ export function checkMapping(entry, where = 'library', { fail = raise } = {}) {
return deepFreeze({
kind: 'library', remote: entry.remote, branch: entry.branch,
...(Object.hasOwn(entry, 'user') ? { user: entry.user } : {}),
...(Object.hasOwn(entry, 'vaultPrefix') ? { vaultPrefix: entry.vaultPrefix } : {}),
universes, types: { ...entry.types }, manuscript,
});
+160
View File
@@ -0,0 +1,160 @@
// C2-03a: fetch-only sync of her repo into the C2-04 mirror (design 7.2). git fetches the
// configured branch into refs/remotes/library/<branch>. A fast-forward moves refs/heads/served,
// a non-fast-forward holds it until a human accepts, and a failed fetch keeps serving. The token
// never reaches argv, env, config, a URL or a log: git's GIT_ASKPASS helper reads it from the slot.
import { constants, closeSync, fsyncSync, lstatSync, openSync, renameSync, rmSync, writeSync } from 'node:fs';
import { randomBytes } from 'node:crypto';
import { dirname, join } from 'node:path';
import { fileURLToPath } from 'node:url';
import { gitEnv, runGit, GIT_LIMITS } from '../../seat/src/checkout.mjs';
import { SAFE, git } from '../../project/src/git.mjs';
import { checkMapping } from './mapping.mjs';
import { LibraryError } from './problems.mjs';
import { readLibrary } from './reader.mjs';
import { SYNC_STATE_FILE, ensureMirror, withLibraryLock, servedCommit } from './mirror.mjs';
export const ASKPASS = fileURLToPath(new URL('../bin/askpass', import.meta.url));
export const SLOT = 'library-gitea';
export const SYNC_INTERVAL_MS = 60_000;
// Row 199's patterns (packages/seat/src/working.mjs).
const HOST = /^[a-z0-9.-]{1,253}(:[0-9]{1,5})?$/;
const NAME = /^[A-Za-z0-9_.-]{1,100}$/;
const SHA = /^[a-f0-9]{40}$/;
const CODE = /^[a-z-]{1,64}$/;
const STATE_MAX = 8 * 1024 * 1024;
const ZERO = '0'.repeat(40);
const fail = (code, message = code) => { throw new LibraryError(code, message); };
// One sync per mirror in this process. The library lock covers publication only, so a second
// fetch into the same mirror is refused here rather than racing the first.
const running = new Set();
// The one command sync runs against her remote. `kind` is 'fetch'; there is no other.
export function syncCommand(kind, url, { mirror, branch, user, slot }) {
if (kind !== 'fetch') fail('library-remote-invalid', 'sync only fetches');
let parsed;
try { parsed = new URL(url); } catch { fail('library-remote-invalid', 'remote must be an https:// URL'); }
if (parsed.protocol !== 'https:' || parsed.username || parsed.password || parsed.search || parsed.hash
|| !HOST.test(parsed.host) || typeof user !== 'string' || !NAME.test(user))
fail('library-remote-invalid', 'remote must be https:// with a plain host and no credentials');
// core.createObject=rename: git's default files each object by link then unlink, and a reader's
// inspectMirror (C2-04) refuses a file with two links while that runs.
return {
argv: [...SAFE, '-c', 'protocol.https.allow=always', '-c', 'http.followRedirects=false', '-c', 'credential.helper=',
'-c', 'core.createObject=rename', '-C', mirror, 'fetch', '--no-tags', '--no-write-fetch-head', '--no-recurse-submodules', '--end-of-options',
parsed.href, `+refs/heads/${branch}:refs/remotes/library/${branch}`],
env: { ...gitEnv(), LC_ALL: 'C', GIT_CEILING_DIRECTORIES: dirname(mirror), GIT_ASKPASS: ASKPASS,
MOSAIC_LIBRARY_HOST: parsed.host, MOSAIC_LIBRARY_USER: user, MOSAIC_LIBRARY_SLOT: slot },
};
}
// Present and a regular file. Sync never opens it; only the helper reads it.
function slotPresent(slot) {
try { const st = lstatSync(slot); return st.isFile() && st.uid === process.getuid() && (st.mode & 0o077) === 0; }
catch { return false; }
}
async function isAncestor(mirror, older, newer) {
const result = await runGit([...SAFE, '-C', mirror, 'merge-base', '--is-ancestor', older, newer],
{ ...gitEnv(), GIT_CEILING_DIRECTORIES: dirname(mirror) });
if (result.ok) return true;
if (result.why === 'exit 1') return false;
fail('library-git', 'ancestry check refused');
}
// Temp file beside the mirror (inspectMirror refuses unknown names inside it), then one rename.
function writeState(mirror, state) {
const body = Buffer.from(JSON.stringify(state) + '\n');
if (body.length > STATE_MAX) fail('library-state-invalid', 'sync state over 8 MiB');
const temp = join(dirname(mirror), `.library-state-${randomBytes(8).toString('hex')}`);
let fd;
try {
fd = openSync(temp, constants.O_CREAT | constants.O_EXCL | constants.O_WRONLY | constants.O_NOFOLLOW, 0o600);
for (let at = 0; at < body.length;) at += writeSync(fd, body, at);
fsyncSync(fd); closeSync(fd); fd = undefined;
renameSync(temp, join(mirror, SYNC_STATE_FILE));
const dir = openSync(mirror, constants.O_RDONLY | constants.O_DIRECTORY);
try { fsyncSync(dir); } finally { closeSync(dir); }
} catch (error) {
if (fd !== undefined) closeSync(fd);
rmSync(temp, { force: true });
if (error instanceof LibraryError) throw error;
fail('library-state-invalid', 'sync state write refused');
}
}
const errorCode = error => typeof error?.code === 'string' && CODE.test(error.code) ? error.code : 'library-git';
// sync(context, { slot, accept, command, now, limits }) fetches once and publishes the result.
// It resolves { state, code }: code is null, or why served did not move ('library-fetch-failed',
// 'library-credential-missing', 'library-diverged' or a reader refusal). It throws for a bad
// entry, a busy mirror, a stale --accept or an unreadable state. Sync never runs gc or prune:
// a reader holds `served` after the lock is released and reads its objects later (Filbert N2).
export async function sync(context, { slot, accept = null, command = syncCommand, now = () => new Date(), limits = GIT_LIMITS } = {}) {
const mirror = await ensureMirror(context);
const mapping = checkMapping(context.business.projects[context.project]);
const { branch, user } = mapping;
if (typeof user !== 'string') fail('library-remote-invalid', 'library entry has no user for sync');
if (accept !== null && (typeof accept !== 'string' || !SHA.test(accept))) fail('library-accept-invalid', 'accept takes a full commit id');
if (running.has(mirror)) fail('library-sync-busy', 'a sync of this library is running');
running.add(mirror);
try {
let code = null, tip = null, read = null;
if (typeof slot !== 'string' || !slotPresent(slot)) code = 'library-credential-missing';
else {
const { argv, env } = command('fetch', mapping.remote, { mirror, branch, user, slot });
const result = await runGit(argv, env, limits);
// git's stderr is never kept or logged: the code is the whole record.
if (!result.ok) code = 'library-fetch-failed';
else {
try { tip = await git(mirror, ['rev-parse', '--verify', '--end-of-options', `refs/remotes/library/${branch}^{commit}`]); }
catch { code = 'library-fetch-failed'; }
}
}
if (tip !== null && !SHA.test(tip)) { tip = null; code = 'library-fetch-failed'; }
if (accept !== null && code === null && accept !== tip) fail('library-accept-stale', 'accept names a commit other than the fetched tip');
// Problems belong to the tip alone. Read them before the lock: readers wait only 5 s for it.
const problemsOf = async () => {
try { read = read ?? await readLibrary({ repo: mirror, commit: tip, mapping }); return read.problems.map(p => ({ ...p })); }
catch (error) { code = errorCode(error); return null; }
};
if (tip !== null && (accept !== null || tip !== await servedCommit(mirror))) await problemsOf();
if (code !== null && accept !== null) accept = null;
return await withLibraryLock(mirror, async ({ readSyncState }) => {
const at = now().toISOString();
const served = await servedCommit(mirror);
// --accept starts the record over, so it also clears a state from another branch.
const prev = accept !== null
? { version: 1, branch, served, lastFetch: null, lastError: null, diverged: null, problems: [] }
: await readSyncState({ branch });
let next = { ...prev }, target = served;
if (code !== null) next.lastError = { code, at };
else if (accept !== null) {
const problems = await problemsOf();
if (problems === null) next.lastError = { code, at };
else { target = tip; next = { ...next, served: tip, lastFetch: at, lastError: null, diverged: null, problems }; }
} else if (served === tip) next = { ...next, lastFetch: at, lastError: null, diverged: null };
else if (prev.diverged !== null || (served !== null && !await isAncestor(mirror, served, tip))) {
// Held until a human accepts, even if a later tip is a fast-forward again. `at` stays the
// first detection; `commit` follows the tip, which is what --accept must name.
code = 'library-diverged';
next = { ...next, lastFetch: at, lastError: null, diverged: { commit: tip, at: prev.diverged?.at ?? at } };
} else {
const problems = await problemsOf();
if (problems === null) next = { ...next, lastFetch: at, lastError: { code, at } };
else { target = tip; next = { ...next, served: tip, lastFetch: at, lastError: null, problems }; }
}
const moveServed = async args => {
try { await git(mirror, ['update-ref', ...args]); } catch { fail('library-git', 'served ref update refused'); }
};
if (target !== served) await moveServed(['refs/heads/served', target, served ?? ZERO]);
try { writeState(mirror, next); }
catch (error) {
// The ref goes back, so served and the state on disk still match.
if (target !== served) await moveServed(served === null ? ['-d', 'refs/heads/served', target] : ['refs/heads/served', served, target]);
throw error;
}
return { state: next, code };
});
} finally { running.delete(mirror); }
}
+8
View File
@@ -44,6 +44,14 @@ test('shape refusals: kind, remote, branch, unknown keys, types, manuscript, fol
refuses(e => { e.universes['bad id'] = e.universes.second; delete e.universes.second; return e; }, 'library-mapping-invalid', /plain id/);
});
test('C2-03a sage-2021: user is optional, kept when given, and must be a plain account name', () => {
assert.equal(Object.hasOwn(mapping(), 'user'), false);
assert.equal(mapping(e => ({ ...e, user: 'books-bot' })).user, 'books-bot');
assert.equal(mapping(e => ({ ...e, user: 'A_b.c-9'.padEnd(100, 'x') })).user.length, 100);
for (const user of ['', 'a'.repeat(101), 'books bot', 'bot@host', 'bot:pw', 'bot/x', 'bøt', 'bot\n', 7, null])
refuses(e => ({ ...e, user }), 'library-mapping-invalid', /user must be a plain account name/);
});
test('a hybrid universe is a shape the query code can take (1.4), with sources that must exist', () => {
const hybrid = e => { e.universes.cross = { kind: 'hybrid', name: 'Crossover', sources: ['harbor', 'second'], books: ['Crossover'],
constraints: [{ exclude: 'second/Characters' }, { pin: { entity: 'Book One/Characters/Mara Vell.md', field: 'eyes', value: 'grey' } }],
+306
View File
@@ -0,0 +1,306 @@
// C2-03a: fetch-only sync (design 7.2). Her remote is row 199's fake Gitea on 127.0.0.1 with a
// scratch CA; the test reaches it only through the `command` parameter, which keeps syncCommand's
// argv and adds the CA to its env. The token is made up and must appear nowhere but the slot.
import test, { before, after } from 'node:test';
import assert from 'node:assert/strict';
import { accessSync, constants, mkdirSync, mkdtempSync, readdirSync, readFileSync, rmSync, statSync, symlinkSync, unlinkSync, writeFileSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { dirname, join } from 'node:path';
import { spawnSync } from 'node:child_process';
import { scratchCa, fakeGitea } from '../../seat/tests/fake-gitea.mjs';
import { SAFE } from '../../project/src/git.mjs';
import { mappingEntry, repo, git, BASE } from './helpers.mjs';
import { syncCommand, sync, ASKPASS, SLOT, SYNC_INTERVAL_MS } from '../src/sync.mjs';
import { mirrorPath, readSyncState, readLibraryFile, SYNC_STATE_FILE } from '../src/mirror.mjs';
import { readLibrary } from '../src/reader.mjs';
import { checkMapping } from '../src/mapping.mjs';
import { survey } from '../src/survey.mjs';
const TOKEN = 'fixture-token-5d1c-made-up-0042';
const USER = 'books-bot';
const scratch = mkdtempSync(join(tmpdir(), 'library-sync-ca-'));
let tls;
before(() => { tls = scratchCa(join(scratch, 'ca')); });
after(() => rmSync(scratch, { recursive: true, force: true }));
const clock = () => { let n = 0; return () => new Date(Date.UTC(2026, 9, 21, 12, 0, n++)); };
async function world(t, { token = TOKEN, redirect = null, user = USER } = {}) {
const f = repo(t);
const gitea = await fakeGitea({ dir: join(f.root, 'gitea'), tls, user: USER, token, redirect });
t.after(() => gitea.close());
const remote = gitea.repo('writer', 'books');
mkdirSync(dirname(remote), { recursive: true });
git(f.root, 'clone', '-q', '--bare', '--no-local', f.repo, remote);
const dataRoot = join(f.root, 'data');
mkdirSync(dataRoot, { mode: 0o700 });
const slot = join(f.root, 'config', 'secrets', 'acme', SLOT);
mkdirSync(dirname(slot), { recursive: true, mode: 0o700 });
writeFileSync(slot, TOKEN + '\n', { mode: 0o600 });
const entry = { ...mappingEntry(), remote: `${gitea.origin}/writer/books.git`, ...(user === null ? {} : { user }) };
const context = { business: { id: 'acme', projects: { books: entry } }, project: 'books', dataRoot };
const command = (kind, url, opts) => { const c = syncCommand(kind, url, opts); return { argv: c.argv, env: { ...c.env, GIT_SSL_CAINFO: tls.ca } }; };
const now = clock();
const run = (options = {}) => sync(context, { slot, command, now, ...options });
// A commit in the work repo, pushed to the fake remote; `from` rewinds first, for a divergence.
const push = (files, { from = null } = {}) => {
if (from) git(f.repo, 'reset', '-q', '--hard', from);
for (const [path, text] of Object.entries(files)) {
const file = join(f.repo, path);
if (text === null) unlinkSync(file);
else { mkdirSync(dirname(file), { recursive: true }); writeFileSync(file, text); }
}
git(f.repo, 'add', '-A'); git(f.repo, 'commit', '-qm', 'next');
git(f.repo, 'push', '-q', '--force', remote, 'HEAD:refs/heads/main');
return git(f.repo, 'rev-parse', 'HEAD');
};
const mirror = () => mirrorPath(context);
const state = () => readSyncState(mirror(), { branch: 'main' });
return { ...f, gitea, remote, dataRoot, slot, entry, context, command, run, push, mirror, state };
}
// Every file under `dir`, with the token searched as bytes.
function holdsToken(dir) {
for (const name of readdirSync(dir, { recursive: true })) {
const file = join(dir, name);
if (statSync(file).isFile() && readFileSync(file).includes(TOKEN)) return file;
}
return null;
}
test('C2-03a argv: the fixed sequence, one https allow, no other protocol key', () => {
const opts = { mirror: '/srv/data/projects/acme/books', branch: 'main', user: USER, slot: '/srv/config/secrets/acme/library-gitea' };
const { argv, env } = syncCommand('fetch', 'https://git.example.invalid/writer/books.git', opts);
assert.deepEqual(argv, [...SAFE, '-c', 'protocol.https.allow=always', '-c', 'http.followRedirects=false', '-c', 'credential.helper=',
'-c', 'core.createObject=rename', '-C', opts.mirror, 'fetch', '--no-tags', '--no-write-fetch-head', '--no-recurse-submodules', '--end-of-options',
'https://git.example.invalid/writer/books.git', '+refs/heads/main:refs/remotes/library/main']);
const protocol = argv.filter((a, i) => argv[i - 1] === '-c' && a.startsWith('protocol.'));
assert.deepEqual(protocol, ['protocol.allow=never', 'protocol.https.allow=always']);
assert.equal(argv.includes('--prune'), false, 'no fetch drops objects a reader may hold (Filbert N2)');
assert.equal(argv.some(a => /^(gc|prune|repack)$/.test(a)), false);
assert.equal(SYNC_INTERVAL_MS, 60_000);
// gosalyn-0068: LC_ALL=C. The helper sits outside the mirror, and the env names the slot path only.
assert.equal(env.LC_ALL, 'C');
assert.equal(env.GIT_ASKPASS, ASKPASS);
assert.equal(ASKPASS.startsWith(opts.mirror), false);
accessSync(ASKPASS, constants.X_OK);
assert.deepEqual(Object.keys(env).sort(), ['GIT_ASKPASS', 'GIT_CEILING_DIRECTORIES', 'GIT_CONFIG_GLOBAL', 'GIT_CONFIG_NOSYSTEM',
'GIT_MERGE_AUTOEDIT', 'GIT_TERMINAL_PROMPT', 'HOME', 'LC_ALL', 'MOSAIC_LIBRARY_HOST', 'MOSAIC_LIBRARY_SLOT', 'MOSAIC_LIBRARY_USER', 'PATH']);
assert.equal(env.MOSAIC_LIBRARY_SLOT, opts.slot);
assert.equal(env.MOSAIC_LIBRARY_HOST, 'git.example.invalid');
assert.equal(syncCommand('fetch', 'https://127.0.0.1:4430/w/b.git', opts).env.MOSAIC_LIBRARY_HOST, '127.0.0.1:4430');
});
test('C2-03a argv: only an https URL with a plain host, no userinfo, and a plain user', () => {
const opts = { mirror: '/m', branch: 'main', user: USER, slot: '/s' };
for (const url of ['http://git.example.invalid/b.git', 'ssh://git.example.invalid/b.git', 'file:///srv/b.git', 'ext::sh -c x',
'https://[email protected]/b.git', 'https://bot:[email protected]/b.git', 'https://git.example.invalid/b.git?x=1',
'https://git.example.invalid/b.git#x', 'https://[::1]/b.git', 'not a url'])
assert.throws(() => syncCommand('fetch', url, opts), { code: 'library-remote-invalid' }, url);
for (const user of [undefined, '', 'a b', 'bot@x', 'bot:x', 'a'.repeat(101)])
assert.throws(() => syncCommand('fetch', 'https://git.example.invalid/b.git', { ...opts, user }), { code: 'library-remote-invalid' });
assert.throws(() => syncCommand('push', 'https://git.example.invalid/b.git', opts), { code: 'library-remote-invalid' });
});
test('C2-03a helper: answers only the exact prompts for the configured host and user (dewey-0765)', t => {
const dir = mkdtempSync(join(tmpdir(), 'library-askpass-'));
t.after(() => rmSync(dir, { recursive: true, force: true }));
const slot = join(dir, 'slot');
writeFileSync(slot, TOKEN + '\nsecond line\n', { mode: 0o600 });
const env = { PATH: '/usr/bin:/bin', MOSAIC_LIBRARY_HOST: '127.0.0.1:4430', MOSAIC_LIBRARY_USER: USER, MOSAIC_LIBRARY_SLOT: slot };
const ask = (prompt, change = {}) => spawnSync(ASKPASS, [prompt], { env: { ...env, ...change }, encoding: 'utf8' });
const user = ask(`Username for 'https://127.0.0.1:4430': `);
assert.equal(user.status, 0); assert.equal(user.stdout, USER + '\n');
assert.equal(user.stdout.includes(TOKEN), false, 'the Username answer is never the token');
const pass = ask(`Password for 'https://${USER}@127.0.0.1:4430': `);
assert.equal(pass.status, 0); assert.equal(pass.stdout, TOKEN + '\n', 'the first line of the slot only');
for (const prompt of [
`Username for 'https://other.example.invalid': `, `Password for 'https://${USER}@other.example.invalid': `,
`Username for 'https://127.0.0.1:4431': `, `Password for 'https://${USER}@127.0.0.1:4431': `, `Username for 'https://127.0.0.1': `,
`Password for 'https://[email protected]:4430': `, `Password for 'https://127.0.0.1:4430': `,
`Username for 'http://127.0.0.1:4430': `, `Username for 'https://127.0.0.1:4430':`, 'Are you sure you want to continue connecting (yes/no)? ', '']) {
const r = ask(prompt);
assert.equal(r.status, 1, prompt); assert.equal(r.stdout, '', prompt);
}
for (const change of [{ MOSAIC_LIBRARY_SLOT: '' }, { MOSAIC_LIBRARY_USER: '' }, { MOSAIC_LIBRARY_HOST: '' },
{ MOSAIC_LIBRARY_SLOT: join(dir, 'absent') }, { MOSAIC_LIBRARY_USER: 'bot@x' }]) {
const r = ask(`Password for 'https://${change.MOSAIC_LIBRARY_USER ?? USER}@127.0.0.1:4430': `, change);
assert.equal(r.status, 1); assert.equal(r.stdout, '');
}
const link = join(dir, 'link'); symlinkSync(slot, link);
assert.equal(ask(`Password for 'https://${USER}@127.0.0.1:4430': `, { MOSAIC_LIBRARY_SLOT: link }).status, 1);
writeFileSync(slot, '\n', { mode: 0o600 });
assert.equal(ask(`Password for 'https://${USER}@127.0.0.1:4430': `).status, 1, 'an empty first line is no token');
});
test('C2-03a first sync: served is the fetched tip, HEAD is served, the survey reads it, and the token stays in the slot', async t => {
const w = await world(t);
const { state, code } = await w.run();
assert.equal(code, null);
assert.equal(state.served, w.commit);
assert.equal(git(w.mirror(), 'rev-parse', 'refs/heads/served'), w.commit);
assert.equal(git(w.mirror(), 'symbolic-ref', 'HEAD'), 'refs/heads/served');
assert.equal((await survey({ repo: w.mirror() })).commit, state.served);
assert.deepEqual(await w.state(), state);
assert.equal(state.lastFetch, '2026-10-21T12:00:00.000Z');
assert.equal(state.lastError, null);
assert.ok(w.gitea.requests.some(r => r.user === USER), 'the configured user answered the prompt');
assert.ok(w.gitea.requests.every(r => r.user === null || r.user === USER));
// R-problems are recorded without holding the head: BASE has an unmapped top-level note.
assert.ok(state.problems.some(p => p.code === 'R2' && p.path === 'Inbox Note.md'));
assert.equal(git(w.mirror(), 'remote'), '');
assert.equal(git(w.mirror(), 'config', '--list', '--local').includes(TOKEN), false);
assert.equal(holdsToken(w.dataRoot), null);
assert.equal(JSON.stringify(syncCommand('fetch', w.entry.remote, { mirror: w.mirror(), branch: 'main', user: USER, slot: w.slot })).includes(TOKEN), false);
assert.deepEqual(readdirSync(dirname(w.mirror())), ['books'], 'no temp state file is left beside the mirror');
assert.equal(statSync(join(w.mirror(), SYNC_STATE_FILE)).mode & 0o777, 0o600);
});
test('C2-03a fast-forward: served moves, problems are rebuilt, and a repeat changes only lastFetch', async t => {
const w = await world(t);
await w.run();
const next = w.push({ 'Inbox Note.md': null, 'Loose.md': 'made up\n' });
const { state, code } = await w.run();
assert.equal(code, null);
assert.equal(state.served, next);
assert.ok(state.problems.some(p => p.path === 'Loose.md'));
assert.equal(state.problems.some(p => p.path === 'Inbox Note.md'), false);
const again = await w.run();
assert.deepEqual({ ...again.state, lastFetch: null }, { ...state, lastFetch: null });
assert.notEqual(again.state.lastFetch, state.lastFetch);
});
test('C2-03a Filbert N2: a read that took served before a sync still finds its objects after served moves', async t => {
const w = await world(t);
await w.run();
const held = (await w.state()).served;
const blob = git(w.mirror(), 'rev-parse', `${held}:Book One/Characters/Tomas.md`);
const next = w.push({ 'Book One/Characters/Tomas.md': null });
assert.equal((await w.run()).state.served, next);
git(w.mirror(), 'cat-file', '-e', blob);
const read = await readLibrary({ repo: w.mirror(), commit: held, mapping: checkMapping(w.entry) });
assert.ok(read.files.some(file => file.path === 'Book One/Characters/Tomas.md'));
await assert.rejects(readLibraryFile({ ...w.context, path: 'Book One/Characters/Tomas.md' }), { code: 'project-read-missing' });
});
test('C2-03a overlap: a reader during a sync sees the old pair or the new one, never a mix', async t => {
const w = await world(t);
const first = (await w.run()).state.served;
const next = w.push({ 'Loose.md': 'made up\n' });
let done = false;
const syncing = w.run().finally(() => { done = true; });
const seen = new Set();
while (!done) {
const s = await w.state();
assert.ok([first, next].includes(s.served));
assert.equal(s.problems.some(p => p.path === 'Loose.md'), s.served === next);
seen.add(s.served);
}
assert.equal((await syncing).state.served, next);
assert.ok(seen.has(first));
});
test('C2-03a diverged: a non-fast-forward keeps serving and holds until a human accepts the tip', async t => {
const w = await world(t);
await w.run();
const ahead = w.push({ 'Loose.md': 'made up\n' });
assert.equal((await w.run()).state.served, ahead);
const rewritten = w.push({ 'Other.md': 'made up\n' }, { from: w.commit });
const held = await w.run();
assert.equal(held.code, 'library-diverged');
assert.equal(held.state.served, ahead);
assert.equal(git(w.mirror(), 'rev-parse', 'refs/heads/served'), ahead);
assert.deepEqual(held.state.diverged, { commit: rewritten, at: held.state.lastFetch });
// A later fast-forward of served is still held: only --accept moves it.
const later = w.push({ 'Third.md': 'made up\n' }, { from: ahead });
const still = await w.run();
assert.equal(still.code, 'library-diverged');
assert.equal(still.state.served, ahead);
assert.deepEqual(still.state.diverged, { commit: later, at: held.state.diverged.at });
await assert.rejects(w.run({ accept: rewritten }), { code: 'library-accept-stale' });
await assert.rejects(w.run({ accept: 'not-a-commit' }), { code: 'library-accept-invalid' });
assert.equal((await w.state()).served, ahead);
const accepted = await w.run({ accept: later });
assert.equal(accepted.code, null);
assert.equal(accepted.state.served, later);
assert.equal(accepted.state.diverged, null);
assert.ok(accepted.state.problems.some(p => p.path === 'Third.md'));
assert.deepEqual(await w.state(), accepted.state);
});
test('C2-03a accept starts the record over, so a changed branch gets a fresh state', async t => {
const w = await world(t);
await w.run();
git(w.remote, 'branch', 'drafts', 'main');
w.entry.branch = 'drafts';
await assert.rejects(w.run(), { code: 'library-state-invalid' });
const accepted = await w.run({ accept: w.commit });
assert.equal(accepted.state.branch, 'drafts');
assert.equal(accepted.state.served, w.commit);
});
test('C2-03a fetch failure: a wrong token keeps serving and records the error, lastFetch unchanged', async t => {
const w = await world(t);
const good = (await w.run()).state;
writeFileSync(w.slot, 'made-up-wrong-token\n', { mode: 0o600 });
w.push({ 'Loose.md': 'made up\n' });
const { state, code } = await w.run();
assert.equal(code, 'library-fetch-failed');
assert.equal(state.served, good.served);
assert.equal(state.lastFetch, good.lastFetch);
assert.equal(state.lastError.code, 'library-fetch-failed');
assert.deepEqual(state.problems, good.problems);
assert.deepEqual(await w.state(), state);
writeFileSync(w.slot, TOKEN + '\n', { mode: 0o600 });
assert.equal((await w.run()).state.lastError, null, 'the next good fetch clears it');
});
test('C2-03a a failed first fetch records the error with nothing served', async t => {
const w = await world(t, { token: 'made-up-other-token' });
const { state, code } = await w.run();
assert.equal(code, 'library-fetch-failed');
assert.equal(state.served, null);
assert.deepEqual(await w.state(), state);
});
test('C2-03a a 302 to another host fails the fetch and the other host sees nothing', async t => {
const f = repo(t);
const other = await fakeGitea({ dir: join(f.root, 'other'), tls, user: USER, token: TOKEN });
t.after(() => other.close());
const w = await world(t, { redirect: other.origin });
const { state, code } = await w.run();
assert.equal(code, 'library-fetch-failed');
assert.equal(state.served, null);
assert.ok(w.gitea.requests.length > 0);
assert.equal(other.requests.length, 0);
});
test('C2-03a the slot: missing or unsafe refuses before git, and sync never opens it', async t => {
const w = await world(t);
rmSync(w.slot);
const missing = await w.run();
assert.equal(missing.code, 'library-credential-missing');
assert.equal(missing.state.lastError.code, 'library-credential-missing');
assert.equal(w.gitea.requests.length, 0);
writeFileSync(w.slot, TOKEN + '\n', { mode: 0o644 });
assert.equal((await w.run()).code, 'library-credential-missing', 'a slot others can read is refused');
assert.equal((await w.run({ slot: null })).code, 'library-credential-missing');
assert.equal(w.gitea.requests.length, 0);
});
test('C2-03a sage-2021: an entry with no user refuses library-remote-invalid and runs no git', async t => {
const w = await world(t, { user: null });
let called = 0;
await assert.rejects(sync(w.context, { slot: w.slot, command: (...a) => { called++; return w.command(...a); } }), { code: 'library-remote-invalid' });
assert.equal(called, 0);
assert.equal(w.gitea.requests.length, 0);
assert.deepEqual(readdirSync(w.mirror()).includes(SYNC_STATE_FILE), false);
});
test('C2-03a one sync per mirror: an overlapping call refuses library-sync-busy', async t => {
const w = await world(t);
const results = await Promise.allSettled([w.run(), w.run()]);
assert.deepEqual(results.map(r => r.status).sort(), ['fulfilled', 'rejected']);
assert.equal(results.find(r => r.status === 'rejected').reason.code, 'library-sync-busy');
assert.equal((await w.run()).code, null, 'the mirror is free again afterwards');
});
+1 -1
View File
@@ -4,7 +4,7 @@ import { spawn } from 'node:child_process';
import { gitEnv, runGit, GIT_LIMITS } from '../../seat/src/checkout.mjs';
import { ProjectError } from './paths.mjs';
const SAFE = ['-c', 'core.hooksPath=/dev/null', '-c', 'core.fsmonitor=false', '-c', 'core.ignorestat=false',
export const SAFE = ['-c', 'core.hooksPath=/dev/null', '-c', 'core.fsmonitor=false', '-c', 'core.ignorestat=false',
'-c', 'protocol.allow=never', '-c', 'submodule.recurse=false', '-c', 'commit.gpgsign=false',
'-c', 'maintenance.auto=false', '-c', 'gc.auto=0'];
export async function git(repo, args, identity = {}) {
+3 -3
View File
@@ -64,7 +64,7 @@ try {
// the system config, Business can't resolve authority and Settings has
// no notifier. A refusal printed here names no path (redactor).
// --send without a bus refuses to start rather than serve a composer that can't send.
let bus = null, send = null, settingsWrite = null, projectFiles = null, projectFile = null, libraryFile = null, libraryStatus = null, projectSave = null, chat = null, nameWrite = null, assistantStop = null, instancesWrite = null, routesWrite = null, busLine, system = null;
let bus = null, send = null, settingsWrite = null, projectFiles = null, projectFile = null, libraryFile = null, libraryStatus = null, librarySync = null, projectSave = null, chat = null, nameWrite = null, assistantStop = null, instancesWrite = null, routesWrite = null, busLine, system = null;
try {
system = loadSystem();
const { dataRoot } = system;
@@ -78,7 +78,7 @@ try {
projectFiles = args => call('project.files', args);
projectFile = args => call('project.file', args);
libraryFile = args => call('library.file', args); libraryStatus = args => call('library.status', args);
if (sendOn) {send = args => call('message.send', args);settingsWrite=args=>call('limits.set',args);projectSave=args=>call('project.file.save',args);nameWrite=args=>call('name.set',args);assistantStop=args=>call('assistant.stop',args);instancesWrite=args=>call('instances.set',args);routesWrite=args=>call('routes.set',args);}
if (sendOn) {send = args => call('message.send', args);librarySync=args=>call('library.sync',args);settingsWrite=args=>call('limits.set',args);projectSave=args=>call('project.file.save',args);nameWrite=args=>call('name.set',args);assistantStop=args=>call('assistant.stop',args);instancesWrite=args=>call('instances.set',args);routesWrite=args=>call('routes.set',args);}
busLine = `Bus: ${options.business ?? 'the running bus host\'s business'}`;
} catch (err) {
const why = redactor({ configDir: configDir() })(err.message);
@@ -98,7 +98,7 @@ try {
}
let server;
try {
server = await startServer({ board: options.board, port: options.port, bus, reads, preview: options.preview, sso, proof, send, settingsWrite, projectFiles, projectFile, libraryFile, libraryStatus, projectSave, chat, nameWrite, assistantStop, instancesWrite, routesWrite });
server = await startServer({ board: options.board, port: options.port, bus, reads, preview: options.preview, sso, proof, send, settingsWrite, projectFiles, projectFile, libraryFile, libraryStatus, librarySync, projectSave, chat, nameWrite, assistantStop, instancesWrite, routesWrite });
} catch (err) { sso?.close(); throw err; }
let testerServer = null;
if (testerPort !== null) {
+1 -1
View File
@@ -33,7 +33,7 @@ COPY_PATHS=(
packages/conversation/src packages/discord/src packages/discord/bin packages/discord/extension packages/harness/src
packages/seat/src packages/tasks/src packages/tasks/fields.json packages/tasks/requirements.json
packages/webui/src packages/webui/README.md packages/cli/README.md
packages/queue/src packages/library/src
packages/queue/src packages/library/src packages/library/bin
# Whole directory: every in-repo plug-in module ships with the copy (row 131).
packages/plugins
# The unslop check both harness modules mount and list in the bundle (row 66).