diff --git a/BUILD-LOG.md b/BUILD-LOG.md index 92487f340..2b31fcb6d 100644 --- a/BUILD-LOG.md +++ b/BUILD-LOG.md @@ -5176,3 +5176,4 @@ Before: managed project/thread verbs had no Console conversation area. After: fi 2026-10-20 05:10:34 UTC | Row 178 #1676 round 2 verified | Before: C1 render refusals lacked killing tests and row 181 base was pending. Rebased onto 931651f2d, kept all COUNTED/count code and tests, appended T178 tail; added independent owner/collision/absolute/charset/header and empty-loopback proof tests. Four files singly 156/156, all 24 mutants killed, metadata and sandbox fixture probes pass; scratch sweep empty. Packet agents/rocko/work/queue-178/r2/, candidate 9300a670, inventory d0dcde7b (59/59), scan 58 files no findings; in-review revs 2166-2167, expected no-token request failure. No live actions or commits. 2026-10-21T01:28:05Z | Darkwing | row 232 #1734 installer I1 build, round 1 in review | Before: stack-release.sh managed only user units (systemctl --user), and stack-copy.sh had no bundle copy. After: --units external (adopt, activate, rollback --to, status, confirm, revert, gate --since, AWAITING-UNITS, stale-start, external-pm, --plan --expect-from) and stack-copy check --units external, copy --bundle FILE SHA, wrapper ID --create; user mode byte-identical to the base by test. First build 0af0c73c6 on 00d4cc1f6; rebased onto row 200 (cb5fdb064) as a0ffabc3f on darkwing/row232-r1; --sso refused under external (D9), row 200 config_checks run under external (D10). Gate on Sage's slot (sage-1618): cli 561/0, every scripts/test-*.sh green; 17/17 mutants killed. Results commit 84f1a91e6 (pushed, seat branch only). Manifest 7a3b4ea1 (9 paths), queue revs 2595-2596, request comment 29825, reviewer Filbert. Packet agents/darkwing/work/queue-232/BUILD.md. 2026-10-21T04:50:44Z | Darkwing | row 233 #1735 installer I3 build, round 1 in review | Before: root had no upgrade driver for root-owned units and no host run sheet; stack-release.sh had no rehearse --backup last and no external plan refusals. After: scripts/stack-upgrade.sh upgrade and rollback (r6 A1-A5, A2f, R1-R4) with 20 tests, docs/guides/host-run-sheet.md, TOOLS.md lines, and on Sage's rulings stack-release.sh --backup last, BACKUP_STORE and the --plan refusals open-activation and migrate-required (3 tests). Base e65a3ab4d (row 232's landing). Gate on Sage's slot (sage-1704) on d99166ade: cli 616/0, every scripts/test-*.sh green, 1201/0 summed by awk; m1-m8 8/8 killed. Correction: the process sweep after the gate found an orphaned sleep 60 from the lock test's flock holder (N1); 08db6ac3f fixes it test only, Sage ruled the gate stands (sage-1707), rerun c0 20/0 and m1 killed with 0 left (sage-1708). Tip 1a1dc80ca (pushed, seat branch only). Manifest 464ce9eb (7 paths), queue revs 2644 and 2647, request comment 29901, reviewer Filbert. Packet agents/darkwing/work/queue-233/BUILD.md. +2026-10-21T09:00Z | Herb | row 215 C2-04 implementation started | Before: business files reject library entries and root consumers assume code projects; the organization writer can change all project mappings. Plan: approved manifest 2522025af on C2-01 a2c72ac95, Filbert reply to herb-0056, scope sage-1848/1852/1860. Focused slot sage-1869 observed expected red mapping and M9 cases, then 14/0 green for new T1/T2 cases, no skips; clear herb-0062 with empty test/fixture process scan. After that clear, T3/T4 tests and host-only mirror/bus implementation continue under sage-1874, unrun while Sage holds the slot. Full singles, suites and implementation review remain pending. No source commit, push or live change. diff --git a/agents/herb/work/queue-215/BUILD.md b/agents/herb/work/queue-215/BUILD.md index 7357ecec0..b459b43e7 100644 --- a/agents/herb/work/queue-215/BUILD.md +++ b/agents/herb/work/queue-215/BUILD.md @@ -177,3 +177,95 @@ Planning stage. No product code, package test, suite, image build, live operatio or push for row 215 has happened. Product implementation awaits Filbert's manifest approval. Sage owns suite and full-gate grants. Row 197's proof commit is separate and is not part of this row's changes. + +## Implementation ledger + +Plan: agents/herb/work/queue-215/MANIFEST.md at 2522025af, approved by +Filbert in the reply to herb-0056. Execution is inline by Herb under the +executing-plans and test-driven-development skills. No agent delegation. + +Pre-flight: T1's validated mapping feeds T2's protected document and T3's +mirror context; T3's lock/state helpers feed T4's injected host adapter. +The shared contract is pinned above, with no interface conflict found. + +Sage-1867 overrides COMMON's singles allowance: every test and executable +probe needs an explicit slot grant. No further probe/test has run since +that reminder. Read-only inspection and test authoring continue while the +slot is held. The first red execution is pending; product code is unchanged. + +Filbert's approved-manifest build notes: ensureMirror creates the lock in +its staged directory before publication; a missing lock in an existing +mirror refuses library-state-invalid. The top-level metadata allowlist +permits only normal bare Git entries plus the state and stable lock files. +Tests cover an unknown entry. The host needs /usr/bin/flock; ENOENT yields +a handled library-state-invalid refusal, never an unhandled child error. +Lock acquisition uses /usr/bin/flock directly and asynchronous spawn. + +T1: consumer and loader regression tests authored, not run. T2: direct M9 +write and serialization regression tests authored, not run. No completion +or passing claim is made for either task. + +Update, sage-1869 focused slot, 2026-10-21T08:47:48Z to 08:51:07Z: +the busy check was CLEAR before the first test. T1's mapping regression +failed 0/1 on unsupported kind as expected, then passed 2/0 with the publish +refusal included. Other focused greens: business CLI 1/0, resolver 1/0, +instance settings 1/0, seat checkout 1/0, project paths 1/0, sandboxed launcher +1/0, and trackers-boot whole 2/0. T2's direct writer case failed 0/1 on the +missing refusal, then its whole new file passed 4/0. Green total 14/0 with +no skips or cancellations. Logs and counts: granted-t1/. The final /proc +scan found no test process or fixture cwd/argv reference; its private root +was removed. Clear sent as herb-0062. Existing files' other cases have not +been run yet, so T1/T2's full-file acceptance is still pending. + +Sage-1874 directs continued authoring and implementation while the landing +gate/proof holds the slot. T3/T4 tests were written first. Their initial +red runs cannot execute during that hold; implementation proceeds under +that owner direction, with no claim of a witnessed red or green for T3/T4. +Their baseline red and candidate checks need a later explicit grant. + +Implementation ruling: initialization uses GNU /usr/bin/mv --no-clobber +--no-target-directory for its bounded staging rename, rather than Node's +renameSync, which could replace an empty directory made by a competing +actor. This preserves pre-existing directories and needs no new package on +the Linux/coreutils bus host. The host also needs /usr/bin/flock. Neither +command has been executed for T3/T4 yet. Publication errors fail closed. +Sync-state reads are capped at 8 MiB, including a checked extra byte; an +oversized or concurrently changed file refuses library-state-invalid. + +T5 static dependency audit found two partial pinned-copy fixtures that copy +business/src but not its newly imported library mapping validator: +cli/tests/stack-release.test.mjs:1772 and stack-release-fixture.mjs:309 on +the base. Each now includes packages/library/src in the existing copy list. +This changes fixture closure only, no assertion or timeout. The complete +stack-release file and each test file importing that shared fixture require +a later granted single-file run. They are outside the four-file T3/T4 slot. + +Sage-1879 places the four-file T3/T4 focused run after their 212+197 proof, +before Gizmoduck, and requires their explicit clear before starting. No run +has occurred after herb-0062. Row 197's N7 base-image-tag finding is left to +Sage's DEFERRED entry; no row 197 source change is made here. + + +Focused T3/T4 grant sage-1882, 2026-10-21 09:13-09:18Z: busy check +CLEAR. Final whole-file results: mirror 16/0, bus library 6/0, WebUI +library-cli 1/0, project-runtime 3/0, all with zero skips/cancellations. +Syntax checks passed for the four files and five changed host modules. +Logs, all intermediate failures, counts and empty /proc cleanup scan are +in granted-t34/. Baseline a2c72ac95: three focused bus cases 0/3 +(unknown-verb), CLI 0/1 (missing callbacks). Mirror baseline 0/1 is an +import refusal for the new module, not a witnessed behavioral red. +An initial whole baseline bus attempt left a runtime after its expected +failure; its exact runner was stopped, no process remained, and the +baseline was narrowed to the three bounded broker/socket cases. +Candidate fixture setup errors and a mock-child timeout were corrected +before the final whole-file passes. No suite or Docker/image/proof ran. + +Filbert's 09:09 publication note: no dependence on mv -n's skipped-rename +exit status across coreutils versions. Save staging dev/ino before mv; +only its disappearance plus the same dev/ino at the mirror proves a win. +A remaining original staging inode means a loss, requiring a valid winner. +Only that owned original staging inode is removed. Numeric no-clobber +statuses 0 and 1, including a diagnostic, are tested with a competing +winner. Spawn errors, signals/timeouts and inconsistent outcomes refuse. +Coreutils must provide atomic --no-clobber --no-target-directory semantics; +it need not use any particular skipped-rename status convention. diff --git a/agents/herb/work/queue-215/MANIFEST.md b/agents/herb/work/queue-215/MANIFEST.md index bd9cd4610..d366238c8 100644 --- a/agents/herb/work/queue-215/MANIFEST.md +++ b/agents/herb/work/queue-215/MANIFEST.md @@ -65,6 +65,11 @@ resolve.test.mjs, instance-settings.test.mjs; seat/tests/checkout.test.mjs; cli/tests/launcher.test.mjs, trackers-boot.test.mjs; project/tests/project.test.mjs; bus/tests/project-runtime.test.mjs. All paths are under packages/. +T5 dependency-audit addition: cli/tests/stack-release.test.mjs and its +stack-release-fixture.mjs helper copy packages/library/src alongside +business/src, whose validator now imports the mapping checker. No assertions +or timeouts change. Whole-file dependent singles require a later Sage slot. + New tests: packages/business/tests/library-projects-protected.test.mjs; packages/library/tests/mirror.test.mjs; packages/bus/tests/library.test.mjs; packages/webui/tests/library-cli.test.mjs. Reuse existing made-up fixtures. diff --git a/packages/bus/src/broker.mjs b/packages/bus/src/broker.mjs index e99cf7625..238dd8354 100644 --- a/packages/bus/src/broker.mjs +++ b/packages/bus/src/broker.mjs @@ -88,6 +88,7 @@ export const TASK_PAGE_MAX = 200; // Row 148 (#1642): the human-only verbs behind the Console's document editor. The save is the one // request allowed past the ordinary 64 KiB envelope (server.mjs and client.mjs), up to its content cap. export const PROJECT_FILE_VERBS = Object.freeze(['project.files', 'project.file', 'project.file.save']); +export const LIBRARY_VERBS = Object.freeze(['library.file', 'library.status']); export const PROJECT_SAVE_VERB = 'project.file.save'; export const PROJECT_SAVE_REQUEST_MAX = 1024 * 1024 + 64 * 1024; export const WEB_VERBS =Object.freeze(['web.search', 'web.fetch']); @@ -267,6 +268,7 @@ export class Broker { #organization; #credentialVerbs; #project; + #library; #usage; #limits; #harnessOn; @@ -276,7 +278,7 @@ export class Broker { #stops = new Map(); #clock = 0; // credentialStatus is the runtime's metadata-only Credentials.status(); the broker never gets use(). - constructor({ store, businesses, secretCheck = () => {}, credentialStatus = () => [], organization = null, project = null, usage = null, limits = null, harnessOn = null, credentialVerbs = null, hostGrant = null }) { + constructor({ store, businesses, secretCheck = () => {}, credentialStatus = () => [], organization = null, project = null, library = null, usage = null, limits = null, harnessOn = null, credentialVerbs = null, hostGrant = null }) { if (!object(businesses) || !Object.keys(businesses).length) fail('invalid-business'); this.#businesses = structuredClone(businesses); this.#store = store; @@ -287,6 +289,7 @@ export class Broker { this.#credentialVerbs = credentialVerbs; this.#hostGrant = hostGrant; this.#project = project; + this.#library = library; this.#usage = usage; this.#limits = limits; // Row 65: harnessOn(business, name) is true when the organization file's plugins.harness switches @@ -1743,6 +1746,8 @@ export class Broker { return this.#requestProject(s, { verb: request.verb, args: structuredClone(args) }); if (PROJECT_FILE_VERBS.includes(request.verb)) return this.#requestProjectFile(s, { verb: request.verb, args: structuredClone(args) }); + if (LIBRARY_VERBS.includes(request.verb)) + return this.#requestLibrary(s, { verb: request.verb, args: structuredClone(args) }); if (CREDENTIAL_VERBS.includes(request.verb)) return this.#requestCredential(s, request.verb, structuredClone(args)); if (["limits", "limits.set", "organization.set", "organization.revision", "plugins", "plugins.set", "instances", "instances.set", "routes", "routes.set"].includes(request.verb)) { // Row 123 (#1617, REQ-COS-1, REQ-LIMIT-1): organization.set is limits.set for a role whose @@ -1992,6 +1997,26 @@ export class Broker { throw this.#refused(s, ProjectFailure && e instanceof ProjectFailure ? new BusError(e.code) : e, request); } } + // C2-04 M4: only a verified human reads library text or status. Host Git + // functions stay injected, preserving this module's sandbox import closure. + async #requestLibrary(s, request) { + const { verb, args } = request; + try { + this.#human(s); + const fields = verb === 'library.file' ? ['project', 'path'] : ['project']; + keys(args, fields, fields); + if (!this.#library?.readFile || !this.#library?.status) fail('project-not-configured'); + const context = { business: this.#business(s.business), project: args.project }; + const result = verb === 'library.file' + ? await this.#library.readFile({ ...context, path: args.path }) + : await this.#library.status(context); + this.#secretCheck(result); + return result; + } catch (error) { + const Failure = this.#library?.LibraryError; + throw this.#refused(s, Failure && error instanceof Failure ? new BusError(error.code) : error, request); + } + } // The eight task verbs reach the trusted S3 handler here. It runs outside any transaction, gets the // cap to authorize and record through authorize() and recordTask(); refusals are recorded as in request(). async requestTask(cap, request, handler) { diff --git a/packages/bus/src/runtime.mjs b/packages/bus/src/runtime.mjs index 9de88d265..977bd3462 100644 --- a/packages/bus/src/runtime.mjs +++ b/packages/bus/src/runtime.mjs @@ -3,6 +3,7 @@ import { join } from 'node:path'; import { fileURLToPath } from 'node:url'; import { Store } from './store.mjs'; import * as projectModule from '../../project/src/index.mjs'; +import * as libraryModule from '../../library/src/index.mjs'; import { Broker, BusError } from './broker.mjs'; import { Credentials } from './credentials.mjs'; import { serve } from './server.mjs'; @@ -92,7 +93,7 @@ export async function startBroker({ const protectedRoots = [ fileURLToPath(new URL('../../../', import.meta.url)), ...repoRoots, - ...Object.values(businesses).flatMap((b) => Object.values(b.projects ?? {}).map((p) => p.root)), + ...Object.values(businesses).flatMap((b) => Object.values(b.projects ?? {}).filter(p => p.kind !== 'library').map((p) => p.root)), ]; // Row 199 (#1697, design D1): the per-agent store is /secrets/agents. const storeRoot = organization ? join(organization.dir, 'secrets', 'agents') : null; @@ -123,12 +124,22 @@ export async function startBroker({ const withTokens = (id, result) => ({ ...result, tokens: limits.report(id) }); // Row 210 (#1707, design D7 step 6) and row 234 (#1740, r6 E2): the run dir and the host enrollment half. const runRoot = organization ? join(organization.dir, 'secrets', 'run') : null; + // C2-04: mirrors are host state, never session checkouts. An unsafe + // existing mirror refuses startup before a socket can become ready. + for (const business of Object.values(businesses)) + for (const [project, entry] of Object.entries(business.projects ?? {})) + if (entry.kind === 'library') await libraryModule.ensureMirror({ business, project, dataRoot }); const broker = new Broker({ hostGrant: runRoot ? hostGrant({ configDir: organization.dir, runRoot }) : null, usage, limits, store, businesses, + library: { + readFile: context => libraryModule.readLibraryFile({ ...context, dataRoot }), + status: context => libraryModule.libraryStatus({ ...context, dataRoot }), + LibraryError: libraryModule.LibraryError, + }, project: { readSpec: context => projectModule.readSpec({ ...context, dataRoot }), writeRecord: context => projectModule.writeRecord({ ...context, dataRoot }), diff --git a/packages/bus/src/server.mjs b/packages/bus/src/server.mjs index d974f75de..53d38dd85 100644 --- a/packages/bus/src/server.mjs +++ b/packages/bus/src/server.mjs @@ -1,6 +1,6 @@ import { createServer } from 'node:net'; import { lstatSync, chmodSync, unlinkSync } from 'node:fs'; -import { BusError, TASK_VERBS, WEB_VERBS, HOST_VERBS, HOSTS_VERBS, PROJECT_FILE_VERBS, PROJECT_SAVE_VERB, PROJECT_SAVE_REQUEST_MAX } from './broker.mjs'; +import { BusError, TASK_VERBS, WEB_VERBS, HOST_VERBS, HOSTS_VERBS, PROJECT_FILE_VERBS, LIBRARY_VERBS, PROJECT_SAVE_VERB, PROJECT_SAVE_REQUEST_MAX } from './broker.mjs'; const LIMIT = 65536; // Row 148: a document save may carry up to 1 MiB of text. Only a request that names that verb near // its start (the proof object comes first) may pass LIMIT; the parsed verb is checked again below. @@ -122,7 +122,7 @@ export async function serve({ broker, path, authenticateHuman = null, authentica } // Row138: only project Git effects need an await; preserve every other sync path below. if (r.verb === 'usage.begin' || r.verb === 'usage.end' || - r.verb === 'spec.read' || r.verb === 'record.write' || PROJECT_FILE_VERBS.includes(r.verb) || + r.verb === 'spec.read' || r.verb === 'record.write' || PROJECT_FILE_VERBS.includes(r.verb) || LIBRARY_VERBS.includes(r.verb) || // Row 234 (#1740): a grant raise awaits its keyscan, and hosts.check its ssh. HOSTS_VERBS.includes(r.verb) || (r.verb === 'decision.raise' && (Object.hasOwn(r.args ?? {}, 'spec_path') || Object.hasOwn(r.args ?? {}, 'spec_commit') || diff --git a/packages/bus/tests/library.test.mjs b/packages/bus/tests/library.test.mjs new file mode 100644 index 000000000..6af2fd338 --- /dev/null +++ b/packages/bus/tests/library.test.mjs @@ -0,0 +1,123 @@ +// C2-04 M4 and real async transport, with made-up local library data. +import test from 'node:test'; +import assert from 'node:assert/strict'; +import { chmodSync, existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { Store } from '../src/store.mjs'; +import { Broker } from '../src/broker.mjs'; +import { serve } from '../src/server.mjs'; +import { Client } from '../src/client.mjs'; +import { startBroker } from '../src/runtime.mjs'; +import { mappingEntry, repo, git, BASE, MARKER } from '../../library/tests/helpers.mjs'; +import * as library from '../../library/src/index.mjs'; + +function world(t, adapter = {}) { + const root = mkdtempSync(join(tmpdir(), 'bus-library-')), store = new Store(root); + t.after(() => { store.close(); rmSync(root, { recursive: true, force: true }); }); + const business = { id: 'acme', human: 'jason', vars: { 'bus.tester': true }, + projects: { books: mappingEntry() }, arbiters: { technical: 'coder', delivery: 'coder' }, roles: { coder: { authority: { withinRole: [], crossRole: [] } } } }; + const broker = new Broker({ store, businesses: { acme: business }, library: adapter }); + const human = broker.bindHuman({ business: 'acme', human: 'jason', via: 'cli', outsideAgent: true }); + const agent = broker.bindLaunch({ business: 'acme', role: 'coder', run: 'library-agent', harness: 'pi' }); + const reader = broker.bindReader({ business: 'acme' }), tester = broker.bindTester({ business: 'acme' }); + const call = (cap, verb, args = {}) => Promise.resolve().then(() => broker.request(cap, { verb, args })); + return { root, store, broker, business, human, agent, reader, tester, call }; +} + +test('C2-04 M4: both verbs reject every agent before arguments or adapter access', async t => { + let touched = 0; + const f = world(t, { readFile: () => { touched++; }, status: () => { touched++; } }); + for (const verb of ['library.file', 'library.status']) { + await assert.rejects(f.call(f.agent, verb, { unknown: true }), { code: 'human-required' }); + await assert.rejects(f.call(f.tester, verb, { unknown: true }), { code: 'human-required' }); + await assert.rejects(f.call(f.reader, verb, { unknown: true }), { code: 'read-only' }); + } + assert.equal(touched, 0); + const refusals = f.store.all("SELECT body FROM events WHERE kind='action.refused'").map(row => JSON.parse(row.body)); + assert.equal(refusals.length, 6); + assert.equal(JSON.stringify(refusals).includes('unknown'), false); + const absent = world(t, null); + for (const verb of ['library.file', 'library.status']) + await assert.rejects(absent.call(absent.agent, verb, {}), { code: 'human-required' }); +}); + +test('C2-04 bus: human reads validate keys and return adapter results without path leakage', async t => { + const seen = []; + const f = world(t, { readFile: async args => { seen.push(args); return { project: 'books', path: 'Book One/A.md', commit: 'a'.repeat(40), content: '# Made up\n' }; }, + status: async args => { seen.push(args); return { project: 'books', served: null, problems: [] }; }, LibraryError: library.LibraryError }); + assert.equal((await f.call(f.human, 'library.file', { project: 'books', path: 'Book One/A.md' })).content, '# Made up\n'); + assert.equal((await f.call(f.human, 'library.status', { project: 'books' })).served, null); + assert.equal(seen[0].business.id, 'acme'); assert.equal(seen[1].project, 'books'); + for (const [verb, args] of [['library.file', { project: 'books' }], ['library.file', { project: 'books', path: 'Book One/A.md', commit: 'a'.repeat(40) }], ['library.status', { project: 'books', extra: 1 }]]) + await assert.rejects(f.call(f.human, verb, args), { code: 'invalid-request' }); +}); + +test('C2-04 socket: each async library verb resolves its result rather than serializing {}', async t => { + const f = world(t, { readFile: async () => ({ project: 'books', path: 'Book One/A.md', commit: 'a'.repeat(40), content: '# Fixture\n' }), + status: async () => ({ project: 'books', served: null, problems: [] }), LibraryError: library.LibraryError }); + const path = join(f.root, 'bus.sock'); + const server = await serve({ broker: f.broker, path, authenticateHuman: () => f.broker.bindHuman({ business: 'acme', human: 'jason', via: 'cli', outsideAgent: true }) }); + t.after(() => server.close()); + const human = new Client({ path, human: { fixture: true } }); + assert.deepEqual(await human.call('library.status', { project: 'books' }), { project: 'books', served: null, problems: [] }); + assert.equal((await human.call('library.file', { project: 'books', path: 'Book One/A.md' })).content, '# Fixture\n'); +}); + +test('C2-04 runtime: protectedRoots skips library but retains real code-root credential protection', async t => { + const f = repo(t), dataRoot = join(f.root, 'data'), token = join(f.root, 'fixture-token'); + mkdirSync(dataRoot, { mode: 0o700 }); writeFileSync(token, 'made-up-fixture-token-123456', { mode: 0o600 }); + const business = { id: 'acme', human: 'jason', projects: { books: mappingEntry(), stack: { root: f.repo } }, + arbiters: { technical: 'coder', delivery: 'coder' }, roles: { coder: { authority: { withinRole: [], crossRole: [] }, credentials: { gitea: { file: token, rotateBy: '2099-01-01' } } } } }; + const runtime = await startBroker({ dataRoot, businesses: { acme: business } }); + await runtime.close(); + const mirror = library.mirrorPath({ business, project: 'books', dataRoot }); + assert.equal(git(mirror, 'rev-parse', '--is-bare-repository'), 'true'); + const bad = join(f.repo, 'credential'); writeFileSync(bad, 'made-up-fixture-token-123456', { mode: 0o600 }); + business.roles.coder.credentials.gitea.file = bad; + await assert.rejects(startBroker({ dataRoot, businesses: { acme: business } })); +}); + +test('C2-04 runtime: unsafe existing mirror refuses boot and never publishes a socket', async t => { + const f = repo(t), dataRoot = join(f.root, 'data'); + mkdirSync(dataRoot, { mode: 0o700 }); + const business = { id: 'acme', human: 'jason', projects: { books: mappingEntry() }, arbiters: { technical: 'coder', delivery: 'coder' }, roles: { coder: { authority: { withinRole: [], crossRole: [] } } } }; + const mirror = join(dataRoot, 'projects/acme/books'); mkdirSync(mirror, { recursive: true }); + writeFileSync(join(mirror, 'keep'), 'made-up note'); + await assert.rejects(startBroker({ dataRoot, businesses: { acme: business } }), { code: 'library-git' }); + assert.equal(existsSync(join(dataRoot, 'bus.sock')), false); + assert.equal(readFileSync(join(mirror, 'keep'), 'utf8'), 'made-up note'); +}); + +test('C2-04 runtime: real library adapters reach the socket and lock waits leave other bus reads responsive', { timeout: 15000 }, async t => { + const f = repo(t), dataRoot = join(f.root, 'data'); + mkdirSync(dataRoot, { mode: 0o700 }); + const business = { id: 'acme', human: 'jason', projects: { books: mappingEntry() }, arbiters: { technical: 'coder', delivery: 'coder' }, roles: { coder: { authority: { withinRole: [], crossRole: [] } } } }; + const runtime = await startBroker({ dataRoot, businesses: { acme: business } }); + t.after(() => runtime.close()); + const cap = runtime.broker.bindHuman({ business: 'acme', human: 'jason', via: 'cli', outsideAgent: true }); + const path = join(f.root, 'human-fixture.sock'); + const server = await serve({ broker: runtime.broker, path, authenticateHuman: () => runtime.broker.bindHuman({ business: 'acme', human: 'jason', via: 'cli', outsideAgent: true }) }); t.after(() => server.close()); + const human = new Client({ path, human: { fixture: true } }); + assert.equal((await human.call('library.status', { project: 'books' })).served, null); + const mirror = library.mirrorPath({ business, project: 'books', dataRoot }); + git(f.repo, '-c', 'protocol.file.allow=always', 'push', '-q', mirror, 'HEAD:refs/heads/served'); + writeFileSync(join(mirror, library.SYNC_STATE_FILE), JSON.stringify({ version: 1, branch: 'main', served: f.commit, + lastFetch: null, lastError: null, diverged: null, problems: [] }), { mode: 0o600 }); + const file = await human.call('library.file', { project: 'books', path: 'Book One/Characters/Tomas.md' }); + assert.equal(file.content, BASE['Book One/Characters/Tomas.md']); + assert.equal(file.commit, f.commit); + let release, done = false; + const hold = library.withLibraryLock(mirror, () => new Promise(r => { release = r; })); + const deadline = Date.now() + 5000; + while (!release && Date.now() < deadline) await new Promise(r => setTimeout(r, 5)); + assert.equal(typeof release, 'function'); + t.after(async () => { release(); await hold; }); + const pending = human.call('library.status', { project: 'books' }).then(result => { done = true; return result; }); + const ordinary = await Promise.race([human.call('agents'), new Promise((_, reject) => { + const timer = setTimeout(() => reject(Error('ordinary bus read blocked by lock acquisition')), 1000); timer.unref(); + })]); + assert.ok(ordinary && typeof ordinary === 'object'); assert.equal(done, false); + release(); await hold; + assert.equal((await pending).served, f.commit); +}); diff --git a/packages/business/src/business.mjs b/packages/business/src/business.mjs index 1149f2b39..e32a1dc51 100644 --- a/packages/business/src/business.mjs +++ b/packages/business/src/business.mjs @@ -10,6 +10,7 @@ import { checkVars } from "./vars.mjs"; import { parseCredentialRef } from "./credentials.mjs"; import { loadRole } from "./role.mjs"; import { checkPlugins } from "../../plugins/src/kinds.mjs"; +import { checkMapping } from "../../library/src/mapping.mjs"; import { forkProblem, giteaName, parseGiteaUrl, HOST_SERVICE } from "./vocabulary.mjs"; import { requireObject, rejectUnknownKeys, requireId, requirePositiveInt, requireString, requireDistinctList, readJsonFile, deepFreeze, @@ -229,6 +230,10 @@ export function validateBusinessDocument(document, file, { rolesDir }) { for (const [id, entry] of Object.entries(document.projects)) { requireId(id, `${file} project id`); requireObject(entry, `${file} projects.${id}`); + if (entry.kind === 'library') { + projects[id] = checkMapping(entry, `${file} projects.${id}`, { fail: (code, message) => refuse(`${code}: ${message}`) }); + continue; + } rejectUnknownKeys(entry, ["root", "repo", "remote"], `${file} projects.${id}`); if (typeof entry.root !== "string" || !isAbsolute(entry.root) || normalize(entry.root) !== entry.root || entry.root.includes("\0")) { refuse(`${file} projects.${id}.root must be a normalized absolute path`); @@ -245,6 +250,7 @@ export function validateBusinessDocument(document, file, { rolesDir }) { for (const [instance, role] of Object.entries(roles)) { for (const project of role.publish ?? []) { if (!Object.hasOwn(projects, project)) refuse(`${file} roles.${instance}.publish names ${project}, which the business file doesn't declare`); + if (projects[project].kind === 'library') refuse(`${file} roles.${instance}.publish names library project ${project}; library repositories cannot be published by agents`); if (!projects[project].remote) refuse(`${file} roles.${instance}.publish names ${project}, which has no "remote"`); const problem = forkProblem(projects[project].remote, role.identity); if (problem) refuse(`${file} roles.${instance}.publish names ${project}: ${problem}`); diff --git a/packages/business/src/cli.mjs b/packages/business/src/cli.mjs index 09cab38f3..1b7dc781d 100644 --- a/packages/business/src/cli.mjs +++ b/packages/business/src/cli.mjs @@ -58,7 +58,7 @@ function checkRefs(refs, forbiddenRoots) { } function forbiddenRoots(config, business) { - return [REPO, config.dataRoot, ...Object.values(business.projects).map((p) => p.root)]; + return [REPO, config.dataRoot, ...Object.values(business.projects).filter(p => p.kind !== 'library').map((p) => p.root)]; } function validate(args) { @@ -69,6 +69,7 @@ function validate(args) { const projects = {}; const warnings = []; for (const [id, entry] of Object.entries(business.projects)) { + if (entry.kind === 'library') { projects[id] = 'library'; continue; } const file = projectFilePath(entry.root); if (!existsSync(file)) { projects[id] = "absent"; @@ -115,7 +116,7 @@ function resolveCommand(args) { if (project !== null) { const entry = Object.hasOwn(business.projects, project) ? business.projects[project] : null; if (!entry) fail(2, `business ${business.id} declares no project ${JSON.stringify(project)}`); - loaded = loadProject(entry.root); + if (entry.kind !== 'library') loaded = loadProject(entry.root); } const resolved = resolveInstance({ system: systemVars(config), business, project: loaded, instance }); const checked = checkRefs(Object.values(resolved.credentials), forbiddenRoots(config, business)); diff --git a/packages/business/src/instance-settings.mjs b/packages/business/src/instance-settings.mjs index b406b1972..eef0d7f09 100644 --- a/packages/business/src/instance-settings.mjs +++ b/packages/business/src/instance-settings.mjs @@ -139,6 +139,7 @@ export function instanceSettings({ dir, rolesDir, system, levels = () => null }) } checkRoutes({ proposed, old: before, levels }); for (const [project, entry] of Object.entries(proposed.projects)) { + if (entry.kind === 'library') continue; if (!existsSync(projectFilePath(entry.root))) continue; const loaded = loadProject(entry.root); if (loaded.id !== project) throw new OrganizationError("organization-project-mismatch"); diff --git a/packages/business/src/organization.mjs b/packages/business/src/organization.mjs index 6256af1dd..493ced9ee 100644 --- a/packages/business/src/organization.mjs +++ b/packages/business/src/organization.mjs @@ -185,12 +185,18 @@ export function organizationWriter({ dir, rolesDir, barrier = () => {} }) { const oldBytes = regular(file); const before = sha256(oldBytes); if (revision !== before) fail("revision-conflict", "organization revision changed; read again"); - validateBusinessDocument(document, file, { rolesDir }); const old = JSON.parse(oldBytes); + // C2-04 M9: every organization setter preserves the whole project + // mapping, including library playgrounds and hybrid universes. + if (canonicalJson(document?.projects) !== canonicalJson(old.projects)) + fail('organization-projects-protected'); + validateBusinessDocument(document, file, { rolesDir }); const bytes = JSON.stringify(document, null, 2) + "\n"; // Check and log exactly what lands on disk, for every setter: a value JSON // can't carry (a Date, a Map) serializes as something else (row 131, sage-0314). const written = JSON.parse(bytes); + if (canonicalJson(written.projects) !== canonicalJson(old.projects)) + fail('organization-projects-protected'); validateBusinessDocument(written, file, { rolesDir }); const after = sha256(bytes); const transaction = randomUUID(); diff --git a/packages/business/src/resolve.mjs b/packages/business/src/resolve.mjs index 2aacede7b..839662bd0 100644 --- a/packages/business/src/resolve.mjs +++ b/packages/business/src/resolve.mjs @@ -41,7 +41,7 @@ export function resolveInstance({ system, business, project = null, instance }) ]; let projectId = null; if (project) { - const declared = Object.entries(business.projects).find(([, p]) => projectFilePath(p.root) === project.file); + const declared = Object.entries(business.projects).find(([, p]) => p.kind !== 'library' && projectFilePath(p.root) === project.file); if (!declared || declared[0] !== project.id) { refuse(`project ${project.id} (${project.file}) isn't declared under that id in business ${business.id}`); } diff --git a/packages/business/tests/business.test.mjs b/packages/business/tests/business.test.mjs index bed5b14d3..3a3f901e5 100644 --- a/packages/business/tests/business.test.mjs +++ b/packages/business/tests/business.test.mjs @@ -8,6 +8,36 @@ import { BusinessError, businessFilePath, configDir, loadBusiness, validateBusin import { THREAD_IDLE_SECONDS } from "../src/business.mjs"; import { launchDoc, businessDoc, REPO, REPO_ROLES, rolesCopy, tmp, writeJson } from "./helpers.mjs"; +test('C2-04: library mappings load without roots, remain frozen, and retain mapping refusals', async t => { + const { mappingEntry } = await import('../../library/tests/helpers.mjs'); + const { rmSync } = await import('node:fs'); + const root = tmp(); t.after(() => rmSync(root, { recursive: true, force: true })); + const doc = businessDoc(root), file = join(root, 'businesses/acme.json'); + doc.projects = { books: mappingEntry(), ...doc.projects }; + const loaded = validateBusinessDocument(doc, file, { rolesDir: REPO_ROLES }); + const { checkMapping } = await import('../../library/src/mapping.mjs'); + assert.deepEqual(loaded.projects.books, checkMapping(mappingEntry())); + assert.equal(loaded.projects.books.root, undefined); + assert.ok(Object.isFrozen(loaded.projects.books.universes.harbor.books)); + const only = structuredClone(doc); delete only.projects.stack; + assert.deepEqual(Object.keys(validateBusinessDocument(only, file, { rolesDir: REPO_ROLES }).projects), ['books']); + doc.projects.books.universes.harbor.playground = ['Elsewhere/Playground/']; + refuses(() => validateBusinessDocument(doc, file, { rolesDir: REPO_ROLES }), /library-playground-outside-book/); + doc.projects.books = { ...mappingEntry(), root: '/forbidden' }; + refuses(() => validateBusinessDocument(doc, file, { rolesDir: REPO_ROLES }), /library-mapping-invalid/); +}); + +test('C2-04: an otherwise valid publish grant cannot name a library repo', async t => { + const { mappingEntry } = await import('../../library/tests/helpers.mjs'); + const { rmSync } = await import('node:fs'); + const root = tmp(); t.after(() => rmSync(root, { recursive: true, force: true })); + const doc = businessDoc(root); + doc.projects.books = { ...mappingEntry(), remote: 'https://git.example/writer/books' }; + doc.roles.coder.identity = { name: 'Coder', email: 'coder@example.invalid', gitea: { url: 'https://git.example', user: 'acme-coder' } }; + doc.roles.coder.publish = ['books']; + refuses(() => validateBusinessDocument(doc, join(root, 'businesses/acme.json'), { rolesDir: REPO_ROLES }), /publish.*library/); +}); + function refuses(fn, pattern, exitCode = 2) { assert.throws(fn, (error) => { assert.ok(error instanceof BusinessError, `expected BusinessError, got ${error}`); diff --git a/packages/business/tests/cli.test.mjs b/packages/business/tests/cli.test.mjs index 278e4369d..9c5de8fab 100644 --- a/packages/business/tests/cli.test.mjs +++ b/packages/business/tests/cli.test.mjs @@ -8,6 +8,24 @@ import { businessDoc, REPO, rolesCopy, systemConfig, tmp, tokenFile, writeJson } const CLI = join(REPO, "packages", "business", "src", "cli.mjs"); +test('C2-04: business validate and resolve skip a library before code projects and credential checks', async t => { + const { mappingEntry } = await import('../../library/tests/helpers.mjs'); + const { rmSync } = await import('node:fs'); + const s = setup(doc => { doc.projects = { books: mappingEntry(), ...doc.projects }; }); + t.after(() => rmSync(s.root, { recursive: true, force: true })); + writeJson(projectFilePath(s.doc.projects.stack.root), { projectVersion: 1, id: 'stack', vars: { 'tracker.project': 3 } }); + const checked = run(s, 'validate', 'acme'); + assert.equal(checked.code, 0, checked.err); + assert.deepEqual(JSON.parse(checked.out).projects, { books: 'library', stack: 'valid' }); + assert.doesNotMatch(checked.err, /project books: no project file/); + const library = run(s, 'resolve', 'acme', 'coder', '--project', 'books'); + assert.equal(library.code, 0, library.err); + assert.equal(JSON.parse(library.out).project, null); + const code = run(s, 'resolve', 'acme', 'coder', '--project', 'stack'); + assert.equal(code.code, 0, code.err); + assert.equal(JSON.parse(code.out).vars['tracker.project'], 3); +}); + // A complete scratch setup: system config, business file, roles copy. // HOME points into the scratch tree too, so nothing reads the real one. function setup(mutate) { diff --git a/packages/business/tests/instance-settings.test.mjs b/packages/business/tests/instance-settings.test.mjs index c0a3cec37..a7bc14643 100644 --- a/packages/business/tests/instance-settings.test.mjs +++ b/packages/business/tests/instance-settings.test.mjs @@ -19,6 +19,16 @@ after(() => roots.forEach(root => rmSync(root, { recursive: true, force: true }) const harnessOn = (doc, names = ["pi", "claude-code"]) => { doc.plugins = { ...(doc.plugins ?? {}), harness: names }; }; const LEVELS = { pi: ["off", "minimal", "low", "medium", "high", "xhigh", "max"], "claude-code": ["low", "medium", "high", "xhigh", "max"] }; +test('C2-04: instance settings skip a library while checking a later code project', async () => { + const { mappingEntry } = await import('../../library/tests/helpers.mjs'); + const f = fixture(doc => { doc.projects = { books: mappingEntry(), ...doc.projects }; }); + writeJson(projectFilePath(f.doc.projects.stack.root), { projectVersion: 1, id: 'stack', vars: { 'tracker.project': 3 } }); + const before = f.settings.read('acme'); + f.settings.set('acme', { revision: before.revision, instance: 'gate-215', op: 'add', definition: 'merge-gate' }, 'jason'); + assert.deepEqual(JSON.parse(readFileSync(f.file, 'utf8')).projects.books, mappingEntry()); + assert.ok(f.settings.read('acme').instances['gate-215']); +}); + function fixture(mutate, { levels } = {}) { const root = tmp(); roots.push(root); diff --git a/packages/business/tests/library-projects-protected.test.mjs b/packages/business/tests/library-projects-protected.test.mjs new file mode 100644 index 000000000..51672bd7b --- /dev/null +++ b/packages/business/tests/library-projects-protected.test.mjs @@ -0,0 +1,68 @@ +// C2-04 M9: use the writer directly, rather than relying on broker key filtering. +import test from 'node:test'; +import assert from 'node:assert/strict'; +import { readFileSync, rmSync } from 'node:fs'; +import { join } from 'node:path'; +import { organizationWriter } from '../src/organization.mjs'; +import { businessDoc, rolesCopy, tmp, writeJson } from './helpers.mjs'; +import { mappingEntry } from '../../library/tests/helpers.mjs'; + +function fixture(t) { + const root = tmp(); t.after(() => rmSync(root, { recursive: true, force: true })); + const doc = businessDoc(root); + const books = mappingEntry(); + books.universes.hybrid = { kind: 'hybrid', name: 'An invented crossover', + sources: ['harbor', 'second'], books: ['Hybrid Book'], playground: [], constraints: [], notes: 'Fixture only' }; + doc.projects = { books, ...doc.projects }; + const dir = join(root, 'config'), file = writeJson(join(dir, 'businesses/acme.json'), doc); + const writer = organizationWriter({ dir, rolesDir: rolesCopy(root) }); + // Give both snapshots an existing ordinary committed receipt to preserve. + doc.vars['tracker.pollSeconds'] = 61; + writer.write('acme', doc, { revision: writer.read('acme').revision, actor: 'jason' }); + const log = writer.logFile('acme'); + return { root, file, doc, writer, log }; +} + +function protectedWrite(f, change) { + const before = readFileSync(f.file), logBefore = readFileSync(f.log); + const next = structuredClone(f.writer.read('acme').document); change(next); + assert.throws(() => f.writer.write('acme', next, { revision: f.writer.read('acme').revision, actor: 'jason' }), + { code: 'organization-projects-protected' }); + assert.deepEqual(readFileSync(f.file), before); + assert.deepEqual(readFileSync(f.log), logBefore); +} + +test('C2-04 M9: changed playground and hybrid entries leave business and changes.jsonl byte-identical', t => { + const f = fixture(t); + protectedWrite(f, next => { next.projects.books.universes.harbor.playground = ['Book One/New Playground/']; }); + protectedWrite(f, next => { next.projects.books.universes.hybrid.notes = 'Changed hybrid rule'; }); +}); + +test('C2-04 M9: code projects, additions, deletions and array changes are protected too', t => { + const f = fixture(t); + protectedWrite(f, next => { next.projects.stack.root = join(f.root, 'different-code'); }); + protectedWrite(f, next => { next.projects.extra = { root: join(f.root, 'extra') }; }); + protectedWrite(f, next => { delete next.projects.stack; }); + protectedWrite(f, next => { next.projects.books.universes.hybrid.sources.reverse(); }); +}); + +test('C2-04 M9: canonical key order is allowed and a non-project setter still records its change', t => { + const f = fixture(t), old = f.writer.read('acme'); + const next = structuredClone(old.document); + next.projects = Object.fromEntries(Object.entries(next.projects).reverse()); + next.vars['tracker.pollSeconds'] = 62; + f.writer.write('acme', next, { revision: old.revision, actor: 'jason' }); + assert.equal(f.writer.read('acme').document.vars['tracker.pollSeconds'], 62); + const receipt = JSON.parse(readFileSync(f.log, 'utf8').trim().split('\n').at(-1)); + assert.equal(receipt.state, 'committed'); + assert.deepEqual(receipt.changes.map(c => c.path), ['vars.tracker.pollSeconds']); +}); + +test('C2-04 M9: a serialization hook cannot change projects after the canonical comparison', t => { + const f = fixture(t); + protectedWrite(f, next => { + const changed = structuredClone(next.projects); + changed.books.universes.hybrid.notes = 'Serialized change'; + Object.defineProperty(next.projects, 'toJSON', { value: () => changed }); + }); +}); diff --git a/packages/business/tests/resolve.test.mjs b/packages/business/tests/resolve.test.mjs index 5ce7298ca..b1e50f349 100644 --- a/packages/business/tests/resolve.test.mjs +++ b/packages/business/tests/resolve.test.mjs @@ -28,6 +28,16 @@ function setup(mutate, projectDoc) { const resolve = (s, instance) => resolveInstance({ system: s.system, business: s.business, project: s.project, instance }); +test('C2-04: a library first in projects does not hide a later project layer', async t => { + const { mappingEntry } = await import('../../library/tests/helpers.mjs'); + const { rmSync } = await import('node:fs'); + const s = setup(doc => { doc.projects = { books: mappingEntry(), ...doc.projects }; }, { vars: { 'tracker.project': 7 } }); + t.after(() => rmSync(s.root, { recursive: true, force: true })); + const result = resolve(s, 'coder'); + assert.equal(result.project, 'stack'); + assert.equal(result.vars['tracker.project'], 7); +}); + test("systemVars flattens the validated config", () => { const vars = systemVars({ environment: "development", dataRoot: "/d", execution: { backend: "docker", provider: "zai", model: "m", adapter: "pi" } }); assert.deepEqual({ ...vars }, { diff --git a/packages/cli/src/config.mjs b/packages/cli/src/config.mjs index 856320d34..d05faed44 100644 --- a/packages/cli/src/config.mjs +++ b/packages/cli/src/config.mjs @@ -59,6 +59,7 @@ function trackerFor(system, b, warn) { if (base["tracker.baseUrl"] === undefined) return null; const named = []; for (const [id, entry] of Object.entries(b.projects)) { + if (entry.kind === 'library') continue; if (!existsSync(projectFilePath(entry.root))) continue; const project = loadProject(entry.root); if (project.id !== id) throw new CliError(`project file ${project.file} has id ${project.id}, but business ${b.id} declares it as ${id}`, 3); diff --git a/packages/cli/src/launcher.mjs b/packages/cli/src/launcher.mjs index 40d08e050..4e18cfb48 100644 --- a/packages/cli/src/launcher.mjs +++ b/packages/cli/src/launcher.mjs @@ -351,7 +351,7 @@ export function createLauncher({ mkdirSync(workspace, { recursive: true, mode: 0o700 }); let checkouts; try { - checkouts = await ensureCheckouts({ workspace, projects: b.projects, env }); + checkouts = await ensureCheckouts({ workspace, projects: Object.fromEntries(Object.entries(b.projects).filter(([, entry]) => entry.kind !== 'library')), env }); } catch (e) { if (e instanceof CheckoutError) throw new Refusal("checkout-failed", `${e.project}: ${e.message}`); throw e; diff --git a/packages/cli/tests/launcher.test.mjs b/packages/cli/tests/launcher.test.mjs index bd8b6ea18..5cb5fef15 100644 --- a/packages/cli/tests/launcher.test.mjs +++ b/packages/cli/tests/launcher.test.mjs @@ -29,6 +29,16 @@ const FAKE = fileURLToPath(new URL("../../harness/tests/fixtures/fake-adapter.mj const HOST = fileURLToPath(new URL("./fixtures/launch-host.mjs", import.meta.url)); const skip = sandboxProblem() ?? false; +test('C2-04: launcher workspace clones code but never the library', { skip, timeout: 30000 }, async t => { + const { mappingEntry } = await import('../../library/tests/helpers.mjs'); + const f = await setup(t, doc => { doc.projects = { books: mappingEntry(), ...doc.projects }; return doc; }); + await f.launcher.launchPm(); + const workspace = workspaceDir(f.f.dataRoot, 'acme', 'chief'); + assert.equal(existsSync(join(workspace, 'stack', '.git')), true); + assert.equal(existsSync(join(workspace, 'books')), false); + assert.equal(await f.close(), 0); +}); + // chief opus, coder and reviewer sonnet, one of each family; cto is listed but // runs the system model, which is in no family. The chief of staff is the launcher. function edit(doc) { diff --git a/packages/cli/tests/stack-release-fixture.mjs b/packages/cli/tests/stack-release-fixture.mjs index 358ff4da5..9c28da250 100644 --- a/packages/cli/tests/stack-release-fixture.mjs +++ b/packages/cli/tests/stack-release-fixture.mjs @@ -306,7 +306,7 @@ for(const [id,store] of [[F,'// a copy from before SCHEMA_VERSION\n'],[A,storeTe for(const [f,text,mode] of [['scripts/mosaic',script(mosaicStub),0o755],['packages/webui/src/cli.mjs',[B,C1,C].includes(id)?`// fixture Console, never run\n// serve accepts '--tester-port'\n// serve accepts '--proof-header' (row 178)\n${[C1,C].includes(id)?"// serve accepts '--sso' (row 192)\n":''}`:'// fixture Console before tester listener, never run\n',0o644],['packages/bus/src/store.mjs',store,0o644],['packages/bus/src/migrate-schema.mjs',`(${migrateStub})();\n`,0o644],['packages/plugins/src/kinds.mjs',[C1,C].includes(id)?KINDS_NEW:KINDS_OLD,0o644],...ssoFiles(id).map(([f,text])=>[f,text,0o644])]){ mkdirSync(dirname(join(d,f)),{recursive:true});writeFileSync(join(d,f),text,{mode}); } - for(const p of ['packages/business/src','roles'])cpSync(join(repo,p),join(d,p),{recursive:true}); + for(const p of ['packages/business/src','packages/library/src','roles'])cpSync(join(repo,p),join(d,p),{recursive:true}); execFileSync('chmod',['-R','a-w',d]); } for(const [name,fn] of [['systemctl',systemctl],['journalctl',journalctl],['systemd-analyze',systemdAnalyze]])writeFileSync(join(SBIN,name),script(fn),{mode:0o755}); diff --git a/packages/cli/tests/stack-release.test.mjs b/packages/cli/tests/stack-release.test.mjs index daae6efb7..ea7e07f5e 100644 --- a/packages/cli/tests/stack-release.test.mjs +++ b/packages/cli/tests/stack-release.test.mjs @@ -1769,7 +1769,7 @@ export class Store { const RCOPIES=join(base,'rcopies');mkdirSync(RCOPIES); for(const [id,v,froms] of [[B,'3g',['3f']],[C,'3h',['3f','3g']],[A,'3g',['3f']],[C1,'3g',['3f']]]){ const d=join(RCOPIES,id); - for(const p of ['scripts/mosaic','scripts/mosaic-config.mjs','packages/cli/src/config.mjs','packages/cli/src/errors.mjs','packages/bus/src/business.mjs','packages/plugins/src/kinds.mjs','packages/business/src','roles'])cpSync(join(repo,p),join(d,p),{recursive:true}); + for(const p of ['scripts/mosaic','scripts/mosaic-config.mjs','packages/cli/src/config.mjs','packages/cli/src/errors.mjs','packages/bus/src/business.mjs','packages/plugins/src/kinds.mjs','packages/business/src','packages/library/src','roles'])cpSync(join(repo,p),join(d,p),{recursive:true}); if(id===C1)writeFileSync(join(d,'scripts','mosaic'),`#!/usr/bin/env bash\nexec node "${join(RCOPIES,B,'packages','cli','src','cli.mjs')}" "$@"\n`,{mode:0o755}); for(const [f,text] of [['packages/cli/src/cli.mjs',id===A?'// fixture host from before row 163\nprocess.exit(1);\n':script(rHost)],['packages/webui/src/cli.mjs',script(rConsole)],['packages/bus/src/store.mjs',rStore(v,froms)],['packages/bus/src/migrate-schema.mjs',script(rMigrate)],['packages/bus/src/broker.mjs','export class BusError extends Error { constructor(code) { super(code); this.code = code; } }\n']]){mkdirSync(dirname(join(d,f)),{recursive:true});writeFileSync(join(d,f),text)} execFileSync('chmod',['-R','a-w',d]); diff --git a/packages/cli/tests/trackers-boot.test.mjs b/packages/cli/tests/trackers-boot.test.mjs index a75cc3ff7..8d1a407c6 100644 --- a/packages/cli/tests/trackers-boot.test.mjs +++ b/packages/cli/tests/trackers-boot.test.mjs @@ -12,7 +12,20 @@ import { FakeVikunja } from "../../tasks/src/fake.mjs"; import { SCOPES } from "../../tasks/tests/world.mjs"; import { fixture, tmp } from "./helpers.mjs"; +test('C2-04: a library-only business with tracker.baseUrl boots without a project tracker', async t => { + const { mappingEntry } = await import('../../library/tests/helpers.mjs'); + const f = fixture(tmp(t), 'acme', doc => { doc.projects = { books: mappingEntry() }; return doc; }); + const warnings = []; + const boot = bootConfig({ system: loadSystem({ env: f.env }), businessId: 'acme', env: f.env, warn: text => warnings.push(text) }); + assert.equal(boot.trackers, undefined); + assert.ok(warnings.some(text => text.includes('no project file sets tracker.project'))); + const host = await startHost({ boot, business: 'acme', log: () => {} }); + t.after(() => host.close(0)); + assert.equal(await host.close(0), 0); +}); + test("bootConfig trackers reach the S3 adapter in the real broker child, which goes ready against a fake Vikunja", async (t) => { + const { mappingEntry } = await import('../../library/tests/helpers.mjs'); const root = tmp(t); const fake = new FakeVikunja(); const srv = await fake.listen(); @@ -26,6 +39,7 @@ test("bootConfig trackers reach the S3 adapter in the real broker child, which g fake.install(project); const f = fixture(root, "acme", (doc) => { + doc.projects = { books: mappingEntry(), ...doc.projects }; doc.vars["tracker.baseUrl"] = srv.url; doc.tracker.sync.botId = bots.sync; // The chief of staff (row 119) has no tracker bot. diff --git a/packages/library/src/index.mjs b/packages/library/src/index.mjs index a76a34554..eee6c4711 100644 --- a/packages/library/src/index.mjs +++ b/packages/library/src/index.mjs @@ -6,3 +6,4 @@ export { extractLinks, parseWikilink, resolver, formOf } from './links.mjs'; export { readLibrary, listTree, readTexts, parseFile, parseFrontMatter, headCommit, MAX_BLOB, LINK_FIELDS } from './reader.mjs'; export { renderItems, KINDS, NOTICES, UNREADABLE } from './render.mjs'; export { survey, formatSurvey, surveyMain } from './survey.mjs'; +export { SYNC_STATE_FILE, mirrorPath, ensureMirror, withLibraryLock, servedCommit, readSyncState, libraryStatus, readLibraryFile } from './mirror.mjs'; diff --git a/packages/library/src/mirror.mjs b/packages/library/src/mirror.mjs new file mode 100644 index 000000000..0aab7361d --- /dev/null +++ b/packages/library/src/mirror.mjs @@ -0,0 +1,257 @@ +// C2-04: host-only bare library storage. No remote, fetch, credential or +// working tree. The broker receives these functions through the runtime. +import { constants, closeSync, existsSync, fstatSync, lstatSync, mkdirSync, mkdtempSync, + openSync, readSync, readdirSync, realpathSync, rmSync } from 'node:fs'; +import { dirname, isAbsolute, join, normalize } from 'node:path'; +import { execFile, spawn } from 'node:child_process'; +import { projectRepo, noLinks } from '../../project/src/paths.mjs'; +import { git } from '../../project/src/git.mjs'; +import { checkMapping } from './mapping.mjs'; +import { classify } from './scope.mjs'; +import { LibraryError, RULES } from './problems.mjs'; +import { listTree, readLibrary, readTexts } from './reader.mjs'; + +export const SYNC_STATE_FILE = 'mosaic-library-sync.json'; +const LOCK_FILE = 'mosaic-library-sync.lock'; +const STATE_MAX = 8 * 1024 * 1024; +const SHA = /^[a-f0-9]{40}$/; +const fail = (code, message = code) => { throw new LibraryError(code, message); }; +const object = value => value !== null && typeof value === 'object' && !Array.isArray(value); +const keys = (value, expected) => object(value) && Object.keys(value).length === expected.length + && Object.keys(value).every(key => expected.includes(key)); +const optional = (value, check) => value === null || check(value); +const utc = value => { + if (typeof value !== 'string') return false; + try { return new Date(value).toISOString() === value; } catch { return false; } +}; +const empty = branch => ({ version: 1, branch, served: null, lastFetch: null, lastError: null, diverged: null, problems: [] }); + +function contextEntry(context) { + const { business, project } = context; + if (!Object.hasOwn(business?.projects ?? {}, project)) fail('project-not-declared'); + const entry = business.projects[project]; + if (entry?.kind !== 'library') fail('library-mapping-invalid', 'expected a library project'); + return checkMapping(entry); +} + +export function mirrorPath(context) { + contextEntry(context); + try { return projectRepo(context, { allowLibrary: true }); } + catch (error) { throw new LibraryError(error.code ?? 'library-git', 'unsafe mirror path'); } +} + +function ownedRoot(mirror) { + try { + if (typeof mirror !== 'string' || !isAbsolute(mirror) || normalize(mirror) !== mirror) throw Error(); + noLinks(mirror); + const st = lstatSync(mirror); + if (!st.isDirectory() || st.uid !== process.getuid() || (st.mode & 0o077) || realpathSync(mirror) !== mirror) throw Error(); + } catch { fail('library-git', 'expected a private owned bare mirror'); } +} + +function privateFile(st) { + return st.isFile() && st.uid === process.getuid() && st.nlink === 1 && (st.mode & 0o077) === 0; +} + +async function inspectMirrorUnchecked(mirror) { + ownedRoot(mirror); + const normal = ['HEAD', 'config', 'description', 'hooks', 'info', 'objects', 'refs', 'logs', 'packed-refs', 'shallow', 'FETCH_HEAD', 'ORIG_HEAD']; + const allowed = new Set([...normal, ...normal.filter(name => !['hooks', 'info', 'objects', 'refs', 'logs', 'description'].includes(name)).map(name => name + '.lock'), SYNC_STATE_FILE, LOCK_FILE]); + const top = readdirSync(mirror); + if (top.some(name => !allowed.has(name)) || ['HEAD', 'config', 'objects', 'refs'].some(name => !top.includes(name))) + fail('library-git', 'unexpected bare mirror layout'); + for (const path of ['commondir', 'objects/info/alternates', 'objects/info/http-alternates']) + if (existsSync(join(mirror, path))) fail('library-git', 'Git metadata indirection refused'); + const walk = (folder, topLevel = false) => { + for (const name of readdirSync(folder)) { + const file = join(folder, name); let st; + try { st = lstatSync(file); } + catch (error) { if (error.code === 'ENOENT') continue; throw error; } + const state = topLevel && [SYNC_STATE_FILE, LOCK_FILE].includes(name); + if (state) { if (!privateFile(st)) fail('library-state-invalid', 'unsafe state or lock metadata'); } + else if (st.isDirectory() && !st.isSymbolicLink()) walk(file); + else if (!st.isFile() || st.nlink !== 1 || st.uid !== process.getuid()) fail('library-git', 'unsafe Git metadata'); + } + }; + walk(mirror, true); + if (!top.includes(LOCK_FILE)) fail('library-state-invalid', 'mirror lock file missing'); + if (lstatSync(join(mirror, 'config')).size > 65536) fail('library-git', 'Git config too large'); + let config; + try { config = await git(mirror, ['config', '--no-includes', '--file', join(mirror, 'config'), '--list', '--null']); } + catch { fail('library-git', 'cannot inspect mirror config'); } + const values = { 'core.repositoryformatversion': ['0'], 'core.filemode': ['true', 'false'], 'core.bare': ['true'], + 'core.logallrefupdates': ['true', 'false'], 'core.ignorecase': ['true', 'false'] }; + let bare = false; + for (const item of config.split('\0').filter(Boolean)) { + const at = item.indexOf('\n'), key = item.slice(0, at), value = item.slice(at + 1); + if (key === 'core.bare' && value === 'true') bare = true; + if (at < 0 || !values[key]?.includes(value)) fail('library-git', 'unsupported bare mirror config'); + } + if (!bare) fail('library-git', 'expected a bare repository'); +} + +async function inspectMirror(mirror) { + try { await inspectMirrorUnchecked(mirror); } + catch (error) { + if (error instanceof LibraryError) throw error; + fail('library-git', 'unreadable bare mirror metadata'); + } +} + +export async function ensureMirror(context) { + const mirror = mirrorPath(context); + if (existsSync(mirror)) { await inspectMirror(mirror); return mirror; } + const parent = dirname(mirror); mkdirSync(parent, { recursive: true, mode: 0o700 }); noLinks(parent); + const staged = mkdtempSync(join(parent, '.library-init-')); + const original = lstatSync(staged); + const isOwnStage = st => st.isDirectory() && st.uid === process.getuid() && st.dev === original.dev && st.ino === original.ino; + try { + await git(staged, ['init', '--bare', '--initial-branch=served', '--template=']); + const fd = openSync(join(staged, LOCK_FILE), constants.O_CREAT | constants.O_EXCL | constants.O_WRONLY | constants.O_NOFOLLOW, 0o600); + closeSync(fd); + await inspectMirror(staged); + // Coreutils versions disagree on the status of a skipped no-clobber + // rename. The original inode, rather than that status, decides who won. + const moveError = await new Promise(resolve => execFile('/usr/bin/mv', ['--no-clobber', '--no-target-directory', staged, mirror], + { timeout: 5000, maxBuffer: 65536, env: { PATH: '/usr/bin:/bin', LC_ALL: 'C' } }, error => resolve(error))); + let remaining; + try { remaining = lstatSync(staged); } catch (error) { if (error.code !== 'ENOENT') throw error; } + if (remaining) { + if (!isOwnStage(remaining)) fail('library-git', 'staging directory was replaced'); + // A skipped rename is valid only when another intact mirror won. + // Signals, timeouts and spawn failures are always genuine errors. + if (moveError && (typeof moveError.code !== 'number' || moveError.killed || moveError.signal)) throw moveError; + } else { + if (!isOwnStage(lstatSync(mirror))) fail('library-git', 'mirror publication identity mismatch'); + if (moveError) throw moveError; + } + await inspectMirror(mirror); + return mirror; + } catch (error) { + if (error instanceof LibraryError) throw error; + fail('library-git', 'bare mirror initialization refused'); + } finally { + let remaining; + try { remaining = lstatSync(staged); } catch (error) { if (error.code !== 'ENOENT') throw error; } + if (remaining) { + if (!isOwnStage(remaining)) fail('library-git', 'refusing to remove replaced staging directory'); + rmSync(staged, { recursive: true }); + } + } +} + +// Each acquisition opens a distinct description. flock(1) inherits it as +// fd 3 and exits after acquisition; this parent's fd retains the lock. +export async function withLibraryLock(mirror, operation) { + ownedRoot(mirror); + let fd; + try { + try { + fd = openSync(join(mirror, LOCK_FILE), constants.O_RDWR | constants.O_NOFOLLOW | constants.O_NONBLOCK); + if (!privateFile(fstatSync(fd))) fail('library-state-invalid', 'unsafe library lock'); + } catch { fail('library-state-invalid', 'library lock missing or unsafe'); } + await new Promise((resolve, reject) => { + const child = spawn('/usr/bin/flock', ['--exclusive', '--wait', '5', '--conflict-exit-code', '75', '3'], + { env: { PATH: '/usr/bin:/bin', LC_ALL: 'C' }, stdio: ['ignore', 'ignore', 'ignore', fd] }); + child.once('error', () => reject(new LibraryError('library-state-invalid', 'host requires /usr/bin/flock'))); + child.once('close', code => code === 0 ? resolve() : reject(new LibraryError(code === 75 ? 'library-sync-busy' : 'library-state-invalid', 'library lock acquisition refused'))); + }); + try { + const heldStat = fstatSync(fd), pathStat = lstatSync(join(mirror, LOCK_FILE)); + if (!privateFile(pathStat) || pathStat.ino !== heldStat.ino || pathStat.dev !== heldStat.dev) throw Error(); + } catch { fail('library-state-invalid', 'library lock was replaced'); } + let held = true; + const pending = new Set(); + const readSyncState = options => { + if (!held) return Promise.reject(new LibraryError('library-state-invalid', 'library lock scope expired')); + const read = readStateHeld(mirror, options).finally(() => pending.delete(read)); + pending.add(read); read.catch(() => {}); + return read; + }; + try { return await operation({ readSyncState }); } + finally { + held = false; + // An accidentally unawaited scoped read must finish before this fd + // releases the lock. No new read can start after the callback ends. + await Promise.allSettled([...pending]); + } + } finally { if (fd !== undefined) closeSync(fd); } +} + +export async function servedCommit(mirror) { + await inspectMirror(mirror); + try { + const refs = await git(mirror, ['for-each-ref', '--format=%(refname) %(objectname)', 'refs/heads/served']); + const line = refs.split('\n').find(line => line.startsWith('refs/heads/served ')); + if (!line) { + if (existsSync(join(mirror, 'refs/heads/served'))) fail('library-state-invalid', 'invalid served ref'); + return null; + } + const commit = line.slice('refs/heads/served '.length); + if (!SHA.test(commit) || await git(mirror, ['cat-file', '-t', commit]) !== 'commit') fail('library-state-invalid', 'served ref must name a commit'); + return commit; + } catch { fail('library-state-invalid', 'served ref is unreadable'); } +} + +function validProblem(value) { + return object(value) && Object.keys(value).every(key => ['code', 'path', 'message', 'line'].includes(key)) + && Object.hasOwn(RULES, value.code) && value.message === RULES[value.code] + && typeof value.path === 'string' && value.path.length > 0 && value.path.length <= 4096 && !value.path.includes('\0') + && (!Object.hasOwn(value, 'line') || Number.isSafeInteger(value.line) && value.line > 0); +} + +async function readStateHeld(mirror, { branch } = {}) { + if (typeof branch !== 'string' || !branch) fail('library-state-invalid', 'configured branch required'); + const served = await servedCommit(mirror); + let fd, value; + try { + try { fd = openSync(join(mirror, SYNC_STATE_FILE), constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK); } + catch (error) { + if (error.code === 'ENOENT' && served === null) return empty(branch); + fail('library-state-invalid', 'sync state missing or unsafe'); + } + const st = fstatSync(fd); + if (!privateFile(st) || st.size > STATE_MAX) fail('library-state-invalid', 'sync state must be bounded and private'); + const bytes = Buffer.alloc(Math.min(st.size + 1, STATE_MAX + 1)); + let used = 0; + while (used < bytes.length) { const read = readSync(fd, bytes, used, bytes.length - used, null); if (!read) break; used += read; } + if (used > STATE_MAX || used > st.size) fail('library-state-invalid', 'state changed during bounded read'); + value = JSON.parse(new TextDecoder('utf-8', { fatal: true }).decode(bytes.subarray(0, used))); + } catch { fail('library-state-invalid', 'invalid sync state'); } + finally { if (fd !== undefined) closeSync(fd); } + if (!keys(value, ['version', 'branch', 'served', 'lastFetch', 'lastError', 'diverged', 'problems']) + || value.version !== 1 || value.branch !== branch || value.served !== served + || !optional(value.served, v => typeof v === 'string' && SHA.test(v)) || !optional(value.lastFetch, utc) + || !optional(value.lastError, v => keys(v, ['code', 'at']) && typeof v.code === 'string' && /^[a-z-]{1,64}$/.test(v.code) && utc(v.at)) + || !optional(value.diverged, v => keys(v, ['commit', 'at']) && typeof v.commit === 'string' && SHA.test(v.commit) && utc(v.at)) + || !Array.isArray(value.problems) || !value.problems.every(validProblem)) + fail('library-state-invalid', 'sync state does not match its contract or served ref'); + return value; +} + +export async function readSyncState(mirror, options) { + return withLibraryLock(mirror, lock => lock.readSyncState(options)); +} + +export async function libraryStatus(context) { + const mapping = contextEntry(context), mirror = mirrorPath(context); + return { project: context.project, ...await readSyncState(mirror, { branch: mapping.branch }) }; +} + +export async function readLibraryFile(context) { + if (!object(context) || Object.keys(context).some(key => !['business', 'project', 'dataRoot', 'path'].includes(key))) + fail('project-read-path', 'unknown library read arguments'); + const mapping = contextEntry(context), path = context.path; + if (typeof path !== 'string' || !path || isAbsolute(path) || path.includes('\0') || path.split('/').some(p => !p || p === '.' || p === '..') + || classify(mapping, path).scope === 'unmapped') fail('project-read-path', 'expected a mapped library note'); + const mirror = mirrorPath(context); + // Snapshot under the publication lock, then read only immutable objects. + const state = await readSyncState(mirror, { branch: mapping.branch }); + if (!state.served) fail('library-commit', 'library has never been synced'); + const read = await readLibrary({ repo: mirror, commit: state.served, mapping }); + if (!read.files.some(file => file.path === path)) fail('project-read-missing', 'file is absent or excluded at served commit'); + const entry = (await listTree(mirror, state.served)).find(entry => entry.path === path); + const texts = await readTexts(mirror, [entry]); + if (!texts.has(path)) fail('project-read-missing', 'file is not readable'); + return { project: context.project, path, commit: state.served, content: texts.get(path) }; +} diff --git a/packages/library/tests/mirror.test.mjs b/packages/library/tests/mirror.test.mjs new file mode 100644 index 000000000..ccc2e07c0 --- /dev/null +++ b/packages/library/tests/mirror.test.mjs @@ -0,0 +1,247 @@ +// C2-04: local made-up repositories only. No remote credential or live config. +import test from 'node:test'; +import assert from 'node:assert/strict'; +import { chmodSync, existsSync, linkSync, lstatSync, mkdirSync, readFileSync, readdirSync, rmSync, symlinkSync, unlinkSync, writeFileSync } from 'node:fs'; +import { join } from 'node:path'; +import { spawn } from 'node:child_process'; +import { once } from 'node:events'; +import { mappingEntry, repo, git, BASE, MARKER } from './helpers.mjs'; +import * as api from '../src/mirror.mjs'; + +const NOW = '2026-10-21T00:00:00.000Z'; +const MOD = new URL('../src/mirror.mjs', import.meta.url).href; +const state = (served = null) => ({ version: 1, branch: 'main', served, lastFetch: null, lastError: null, diverged: null, problems: [] }); +const stateFile = mirror => join(mirror, api.SYNC_STATE_FILE); +async function fixture(t, files = BASE) { + const f = repo(t, files), dataRoot = join(f.root, 'data'); + const context = { business: { id: 'acme', projects: { books: mappingEntry(), code: { root: f.repo } } }, project: 'books', dataRoot }; + const mirror = await api.ensureMirror(context); + const writeState = value => writeFileSync(stateFile(mirror), JSON.stringify(value), { mode: 0o600 }); + const seed = (commit = f.commit) => { + git(f.repo, '-c', 'protocol.file.allow=always', 'push', '-q', mirror, `${commit}:refs/heads/served`); + writeState(state(commit)); + }; + return { ...f, dataRoot, context, mirror, writeState, seed }; +} + +test('C2-04 mirror: exact default path, bare without checkout or remote, and idempotent initialization', async t => { + const f = await fixture(t); + assert.equal(api.SYNC_STATE_FILE, 'mosaic-library-sync.json'); + assert.equal(api.mirrorPath(f.context), join(f.dataRoot, 'projects/acme/books')); + assert.equal(f.mirror, api.mirrorPath(f.context)); + assert.equal(git(f.mirror, 'rev-parse', '--is-bare-repository'), 'true'); + assert.equal(git(f.mirror, 'remote'), ''); + assert.equal(existsSync(join(f.mirror, '.git')), false); + const lock = join(f.mirror, 'mosaic-library-sync.lock'), st = lstatSync(lock); + assert.equal(st.mode & 0o777, 0o600); + assert.equal(st.nlink, 1); + assert.equal(await api.ensureMirror(f.context), f.mirror); + assert.equal(lstatSync(lock).ino, st.ino, 'the lock inode is stable'); + assert.equal(await api.servedCommit(f.mirror), null); + assert.deepEqual(await api.libraryStatus(f.context), { project: 'books', ...state() }); +}); + +test('C2-04 mirror: simultaneous creators publish one intact mirror and leave no staging paths', async t => { + const f = repo(t), dataRoot = join(f.root, 'data'); + const c = { business: { id: 'acme', projects: { books: mappingEntry() } }, project: 'books', dataRoot }; + const results = await Promise.all(Array.from({ length: 4 }, () => api.ensureMirror(c))); + assert.deepEqual([...new Set(results)], [join(dataRoot, 'projects/acme/books')]); + assert.deepEqual(readdirSync(join(dataRoot, 'projects/acme')), ['books']); + assert.equal(git(results[0], 'rev-parse', '--is-bare-repository'), 'true'); +}); + +test('C2-04 mirror: skipped publication accepts both coreutils statuses without removing the winner', { timeout: 15000 }, async t => { + for (const status of [0, 1]) { + const f = repo(t), dataRoot = join(f.root, 'data'); + const context = { business: { id: 'acme', projects: { books: mappingEntry() } }, project: 'books', dataRoot }; + const creator = child(t, `import cp from 'node:child_process';import {cpSync,chmodSync,lstatSync,readdirSync} from 'node:fs';import {dirname} from 'node:path';import {syncBuiltinESMExports} from 'node:module'; + const original=cp.execFile;let winner; + cp.execFile=(command,args,options,callback)=>{if(command!=='/usr/bin/mv')return original(command,args,options,callback); + cpSync(args[2],args[3],{recursive:true});chmodSync(args[3],0o700);winner=lstatSync(args[3]).ino; + const error=${status} ? Object.assign(Error('fixture no-clobber diagnostic'),{code:${status}}) : null; + queueMicrotask(()=>callback(error));};syncBuiltinESMExports(); + const {ensureMirror}=await import(${JSON.stringify(MOD)});const mirror=await ensureMirror(${JSON.stringify(context)}); + process.send({same:lstatSync(mirror).ino===winner,entries:readdirSync(dirname(mirror))});process.disconnect();`); + const [result] = await Promise.race([once(creator, 'message'), creator.finished.then(result => { throw Error(JSON.stringify(result)); })]); + assert.deepEqual(result, { same: true, entries: ['books'] }); + assert.deepEqual(await creator.finished, { code: 0, stderr: '' }); + } +}); + +test('C2-04 mirror: preserve occupied directories and reject symlink ancestors and code overlaps', async t => { + const f = repo(t), dataRoot = join(f.root, 'data'); + const c = { business: { id: 'acme', projects: { books: mappingEntry() } }, project: 'books', dataRoot }; + const path = join(dataRoot, 'projects/acme/books'); mkdirSync(path, { recursive: true }); + writeFileSync(join(path, 'keep'), 'made-up human file'); + await assert.rejects(api.ensureMirror(c)); + assert.equal(readFileSync(join(path, 'keep'), 'utf8'), 'made-up human file'); + rmSync(dataRoot, { recursive: true }); symlinkSync(f.repo, dataRoot); + assert.throws(() => api.mirrorPath(c), { code: 'project-path' }); + unlinkSync(dataRoot); + c.business.projects.code = { root: dataRoot }; + assert.throws(() => api.mirrorPath(c), { code: 'project-path' }); +}); + +test('C2-04 mirror: existing metadata accepts only bare Git plus state and the precreated lock', async t => { + const f = await fixture(t); + writeFileSync(join(f.mirror, 'unknown-note'), 'do not silently allow this'); + await assert.rejects(api.ensureMirror(f.context), { code: 'library-git' }); + assert.equal(readFileSync(join(f.mirror, 'unknown-note'), 'utf8'), 'do not silently allow this'); + unlinkSync(join(f.mirror, 'unknown-note')); + git(f.mirror, 'config', 'remote.origin.url', 'https://example.invalid/fixture.git'); + await assert.rejects(api.ensureMirror(f.context), { code: 'library-git' }); + git(f.mirror, 'config', '--remove-section', 'remote.origin'); + unlinkSync(join(f.mirror, 'mosaic-library-sync.lock')); + await assert.rejects(api.ensureMirror(f.context), { code: 'library-state-invalid' }); + assert.equal(existsSync(join(f.mirror, 'mosaic-library-sync.lock')), false); +}); + +test('C2-04 state: missing state is empty only while the served ref is absent', async t => { + const f = await fixture(t); + assert.deepEqual(await api.readSyncState(f.mirror, { branch: 'main' }), state()); + f.seed(); unlinkSync(stateFile(f.mirror)); + await assert.rejects(api.readSyncState(f.mirror, { branch: 'main' }), { code: 'library-state-invalid' }); +}); + +test('C2-04 state: valid problems and sync metadata return no content and change no state bytes', async t => { + const f = await fixture(t); f.seed(); + const value = { ...state(f.commit), lastFetch: NOW, lastError: { code: 'library-git', at: NOW }, + diverged: { commit: 'a'.repeat(40), at: NOW }, + problems: [{ code: 'R4', path: 'Book One/Chapters/A.md', line: 2, message: 'link resolves to no file or to more than one' }] }; + f.writeState(value); const before = readFileSync(stateFile(f.mirror)); + assert.deepEqual(await api.readSyncState(f.mirror, { branch: 'main' }), value); + assert.equal(JSON.stringify(await api.libraryStatus(f.context)).includes(MARKER), false); + assert.deepEqual(readFileSync(stateFile(f.mirror)), before); +}); + +test('C2-04 state: strict keys, types, times, branch, commit and fixed R messages fail closed', async t => { + const f = await fixture(t); f.seed(); + const mutations = [ + s => { s.extra = true; }, s => { s.version = 2; }, s => { s.branch = 'other'; }, + s => { s.served = 'a'.repeat(40); }, s => { s.served = null; }, s => { s.served = 'ABC'; }, + s => { s.lastFetch = 'yesterday'; }, s => { s.lastFetch = 1; }, + s => { s.lastError = { code: 'bad code', at: NOW }; }, + s => { s.lastError = { code: 'library-git', at: NOW, stderr: MARKER }; }, + s => { s.diverged = { commit: 'not-a-sha', at: NOW }; }, + s => { s.problems = [{ code: 'R1', path: 'Book One/A.md', message: MARKER }]; }, + s => { s.problems = [{ code: 'R4', path: 'Book One/A.md', line: 0, message: 'link resolves to no file or to more than one' }]; }, + s => { s.problems = null; }, + ]; + for (const mutate of mutations) { + const s = state(f.commit); mutate(s); f.writeState(s); + const before = readFileSync(stateFile(f.mirror)); + await assert.rejects(api.readSyncState(f.mirror, { branch: 'main' }), { code: 'library-state-invalid' }); + assert.deepEqual(readFileSync(stateFile(f.mirror)), before); + } + writeFileSync(stateFile(f.mirror), '{'); + await assert.rejects(api.readSyncState(f.mirror, { branch: 'main' }), { code: 'library-state-invalid' }); +}); + +test('C2-04 state: symlink, hardlink, non-private state and missing lock refuse without replacement', async t => { + const f = await fixture(t); f.seed(); + const outside = join(f.root, 'outside.json'); writeFileSync(outside, JSON.stringify(state(f.commit)), { mode: 0o600 }); + unlinkSync(stateFile(f.mirror)); symlinkSync(outside, stateFile(f.mirror)); + await assert.rejects(api.readSyncState(f.mirror, { branch: 'main' }), { code: 'library-state-invalid' }); + unlinkSync(stateFile(f.mirror)); linkSync(outside, stateFile(f.mirror)); + await assert.rejects(api.readSyncState(f.mirror, { branch: 'main' }), { code: 'library-state-invalid' }); + unlinkSync(stateFile(f.mirror)); f.writeState(state(f.commit)); chmodSync(stateFile(f.mirror), 0o644); + await assert.rejects(api.readSyncState(f.mirror, { branch: 'main' }), { code: 'library-state-invalid' }); + chmodSync(stateFile(f.mirror), 0o600); unlinkSync(join(f.mirror, 'mosaic-library-sync.lock')); + await assert.rejects(api.readSyncState(f.mirror, { branch: 'main' }), { code: 'library-state-invalid' }); + assert.equal(existsSync(join(f.mirror, 'mosaic-library-sync.lock')), false); +}); + +test('C2-04 file: served commit, not HEAD or a caller commit, owns the exact returned text', async t => { + const f = await fixture(t); f.seed(); + const path = 'Book One/Characters/Tomas.md'; + writeFileSync(join(f.repo, path), 'newer text\n'); git(f.repo, 'commit', '-qam', 'newer'); + git(f.repo, '-c', 'protocol.file.allow=always', 'push', '-q', f.mirror, 'HEAD:refs/heads/unserved'); + git(f.mirror, 'symbolic-ref', 'HEAD', 'refs/heads/unserved'); + assert.deepEqual(await api.readLibraryFile({ ...f.context, path }), { project: 'books', path, commit: f.commit, content: BASE[path] }); + await assert.rejects(api.readLibraryFile({ ...f.context, path, commit: git(f.repo, 'rev-parse', 'HEAD') }), { code: 'project-read-path' }); + for (const path of ['../escape.md', '/absolute.md', 'Book One/Board.canvas', 'Inbox Note.md']) + await assert.rejects(api.readLibraryFile({ ...f.context, path }), { code: 'project-read-path' }); + await assert.rejects(api.readLibraryFile({ ...f.context, path: 'Book One/Missing.md' }), { code: 'project-read-missing' }); +}); + +test('C2-04 file: R1, R2 and R7 exclusions and never-synced mirrors cannot return file bodies', async t => { + const f = await fixture(t, { ...BASE, 'Book One/Bad.md': { link: 'Characters/Tomas.md' }, + 'Book One/Executable.md': { mode: 0o755, text: MARKER }, + 'Book One/Twin.md': MARKER, 'Book One/twin.md': MARKER }); + await assert.rejects(api.readLibraryFile({ ...f.context, path: 'Book One/Characters/Tomas.md' }), { code: 'library-commit' }); + f.seed(); + for (const path of ['Book One/Bad.md', 'Book One/Executable.md', 'Book One/Twin.md', 'Book One/twin.md']) + await assert.rejects(api.readLibraryFile({ ...f.context, path }), { code: 'project-read-missing' }); +}); + +function child(t, source) { + const env = { ...process.env }; delete env.NODE_TEST_CONTEXT; + const p = spawn(process.execPath, ['--input-type=module', '-e', source], { env, stdio: ['ignore', 'pipe', 'pipe', 'ipc'] }); + let stderr = ''; p.stderr.on('data', b => { stderr += b; }); p.stdout.resume(); + p.finished = new Promise(resolve => { p.once('error', e => resolve({ error: e })); p.once('close', code => resolve({ code, stderr })); }); + t.after(async () => { if (p.exitCode === null) p.kill('SIGKILL'); await p.finished; }); + return p; +} + +test('C2-04 lock: a first-sync and fast-forward reader wait for a consistent pair across processes', { timeout: 15000 }, async t => { + const f = await fixture(t); + // Transfer objects without serving them. No fetch or remote is added by the API. + git(f.repo, '-c', 'protocol.file.allow=always', 'push', '-q', f.mirror, 'HEAD:refs/heads/candidate'); + for (const first of [true, false]) { + if (!first) { f.seed(); writeFileSync(join(f.repo, 'Book One/New.md'), '# New\n'); git(f.repo, 'add', '.'); git(f.repo, 'commit', '-qm', 'new'); + git(f.repo, '-c', 'protocol.file.allow=always', 'push', '-q', f.mirror, 'HEAD:refs/heads/candidate'); } + const commit = git(f.repo, 'rev-parse', 'HEAD'); + const writer = child(t, `import {withLibraryLock} from ${JSON.stringify(MOD)};import {writeFileSync} from 'node:fs';import {git} from ${JSON.stringify(new URL('../../project/src/git.mjs', import.meta.url).href)}; + await withLibraryLock(${JSON.stringify(f.mirror)},async()=>{await git(${JSON.stringify(f.mirror)},['update-ref','refs/heads/served',${JSON.stringify(commit)}]);process.send('moved');await new Promise(r=>process.once('message',r));writeFileSync(${JSON.stringify(stateFile(f.mirror))},${JSON.stringify(JSON.stringify(state(commit)))},{mode:0o600});});process.disconnect();`); + const [message] = await once(writer, 'message'); assert.equal(message, 'moved'); + let complete = false; + const read = api.readSyncState(f.mirror, { branch: 'main' }).then(value => { complete = true; return value; }); + await new Promise(r => setTimeout(r, 50)); assert.equal(complete, false, 'the inconsistent pair is not observed'); + writer.send('publish'); + assert.deepEqual(await read, state(commit)); + assert.deepEqual(await writer.finished, { code: 0, stderr: '' }); + } +}); + +test('C2-04 lock: same-process waits, timeout code, callback expiry and throw release', { timeout: 15000 }, async t => { + const f = await fixture(t); let release, scoped; + const held = api.withLibraryLock(f.mirror, async lock => { scoped = lock.readSyncState; await new Promise(r => { release = r; }); }); + const deadline = Date.now() + 5000; + while (!release && Date.now() < deadline) await new Promise(r => setTimeout(r, 5)); + assert.equal(typeof release, 'function', 'first lock acquired before deadline'); + await assert.rejects(api.readSyncState(f.mirror, { branch: 'main' }), { code: 'library-sync-busy' }); + release(); await held; + await assert.rejects(scoped({ branch: 'main' }), { code: 'library-state-invalid' }); + await assert.rejects(api.withLibraryLock(f.mirror, async () => { throw Error('fixture throw'); }), /fixture throw/); + assert.deepEqual(await api.readSyncState(f.mirror, { branch: 'main' }), state()); +}); + +test('C2-04 lock: owner death releases the kernel lock without replacing its file', { timeout: 15000 }, async t => { + const f = await fixture(t), lock = join(f.mirror, 'mosaic-library-sync.lock'), ino = lstatSync(lock).ino; + const owner = child(t, `import {withLibraryLock} from ${JSON.stringify(MOD)};await withLibraryLock(${JSON.stringify(f.mirror)},async()=>{process.send('held');await new Promise(r=>process.once('message',r));});`); + const [message] = await once(owner, 'message'); assert.equal(message, 'held'); + owner.kill('SIGKILL'); await owner.finished; + assert.deepEqual(await api.readSyncState(f.mirror, { branch: 'main' }), state()); + assert.equal(lstatSync(lock).ino, ino); +}); + +test('C2-04 lock: a missing flock binary becomes a handled refusal', { timeout: 10000 }, async t => { + const f = await fixture(t); + const missing = child(t, `import cp from 'node:child_process';import {syncBuiltinESMExports} from 'node:module';import {EventEmitter} from 'node:events'; + const original=cp.spawn;cp.spawn=(command,...args)=>{if(command!=='/usr/bin/flock')return original(command,...args);const p=new EventEmitter();process.nextTick(()=>p.emit('error',Object.assign(Error('fixture missing flock'),{code:'ENOENT'})));return p;};syncBuiltinESMExports(); + const {readSyncState}=await import(${JSON.stringify(MOD)});try{await readSyncState(${JSON.stringify(f.mirror)},{branch:'main'});process.send('unexpected success');}catch(e){process.send(e.code);}process.disconnect();`); + const [code] = await once(missing, 'message'); + assert.equal(code, 'library-state-invalid'); + assert.deepEqual(await missing.finished, { code: 0, stderr: '' }); +}); + +test('C2-04 lock: an unawaited scoped read drains before the lock is released', { timeout: 10000 }, async t => { + const f = await fixture(t); let scoped, pending, complete = false; + await api.withLibraryLock(f.mirror, lock => { + scoped = lock.readSyncState; + pending = scoped({ branch: 'main' }).then(value => { complete = true; return value; }); + }); + assert.equal(complete, true); + assert.deepEqual(await pending, state()); + await assert.rejects(scoped({ branch: 'main' }), { code: 'library-state-invalid' }); +}); diff --git a/packages/project/src/paths.mjs b/packages/project/src/paths.mjs index ea3c6c400..0ef7cca63 100644 --- a/packages/project/src/paths.mjs +++ b/packages/project/src/paths.mjs @@ -31,14 +31,17 @@ const contains = (base, child) => { const rel = relative(base, child); return rel === '' || (rel !== '..' && !rel.startsWith('..' + sep) && !isAbsolute(rel)); }; -export function projectRepo({ business, project, dataRoot }) { +export function projectRepo({ business, project, dataRoot }, { allowLibrary = false } = {}) { checkedId(business?.id); checkedId(project); if (!Object.hasOwn(business?.projects ?? {}, project)) throw new ProjectError('project-not-declared', `business does not declare ${project}`); absolute(dataRoot); const entry = business.projects[project]; + if (entry.kind === 'library' && !allowLibrary) + throw new ProjectError('project-invalid', 'library projects are not record repositories'); const repo = absolute(entry.repo ?? join(dataRoot, 'projects', business.id, project)); for (const project of Object.values(business.projects)) { + if (project.kind === 'library') continue; const code = absolute(project.root); if (contains(code, repo) || contains(repo, code)) throw new ProjectError('project-path', 'record repository and code checkout overlap'); diff --git a/packages/project/tests/project.test.mjs b/packages/project/tests/project.test.mjs index 317645ebe..52279b4bb 100644 --- a/packages/project/tests/project.test.mjs +++ b/packages/project/tests/project.test.mjs @@ -24,6 +24,19 @@ async function ready(t, custom) { return { c, repo: initialized.repo, commit: initialized.commit }; } +test('C2-04: record APIs refuse a library before creating a repository', async t => { + const c = context(t); + c.business.projects = { books: { kind: 'library' }, ...c.business.projects }; + assert.equal(api.projectRepo(c), join(c.dataRoot, 'projects/acme/stack')); + const library = { ...c, project: 'books' }; + assert.throws(() => api.projectRepo(library), { code: 'project-invalid' }); + for (const author of ['jason', 'tester']) + await assert.rejects(api.initProject(library, { author }), { code: 'project-invalid' }); + assert.equal(readdirSync(c.root).includes('data'), false); + c.business.projects.stack.root = c.dataRoot; + assert.throws(() => api.projectRepo(c), /overlap/); +}); + test('init creates a checked versioned layout, valid templates and human-authored local history', async t => { const { c, repo, commit } = await ready(t); assert.equal(repo, join(c.dataRoot, 'projects/acme/stack')); diff --git a/packages/seat/src/checkout.mjs b/packages/seat/src/checkout.mjs index b33a4a6c7..daea10ece 100644 --- a/packages/seat/src/checkout.mjs +++ b/packages/seat/src/checkout.mjs @@ -311,7 +311,9 @@ async function update(root, dest, g) { export async function ensureCheckouts({ workspace, projects, env = process.env, limits = GIT_LIMITS }) { const g = (args, extra) => git(args, env, extra, limits); const out = []; - for (const [project, { root }] of Object.entries(projects)) { + for (const [project, entry] of Object.entries(projects)) { + if (entry.kind === 'library') continue; + const { root } = entry; const dest = join(workspace, project); let exists = true; try { diff --git a/packages/seat/tests/checkout.test.mjs b/packages/seat/tests/checkout.test.mjs index d93788482..23cfb33f9 100644 --- a/packages/seat/tests/checkout.test.mjs +++ b/packages/seat/tests/checkout.test.mjs @@ -12,6 +12,15 @@ import { join } from "node:path"; import { deflateSync } from "node:zlib"; import { CONFIG_MAX_BYTES, CheckoutError, ensureCheckouts, gitEnv } from "../src/checkout.mjs"; +test('C2-04: ensureCheckouts skips a library before a real clone and leaves no books path', async t => { + const w = world(t); + const result = await ensureCheckouts({ workspace: w.workspace, projects: { books: { kind: 'library' }, stack: { root: w.root } } }); + assert.deepEqual(result.map(r => r.project), ['stack']); + assert.equal(result[0].outcome, 'cloned'); + assert.equal(existsSync(join(w.workspace, 'books')), false); + assert.deepEqual(await ensureCheckouts({ workspace: w.workspace, projects: { books: { kind: 'library' } } }), []); +}); + function tmp(t) { const dir = mkdtempSync(join(tmpdir(), "mosaic-checkout-")); t.after(() => { diff --git a/packages/webui/src/cli.mjs b/packages/webui/src/cli.mjs index 080e3802b..dce1f677d 100644 --- a/packages/webui/src/cli.mjs +++ b/packages/webui/src/cli.mjs @@ -64,7 +64,7 @@ try { // the system config, Business can't resolve authority and Settings has // no notifier. A refusal printed here names no path (redactor). // --send without a bus refuses to start rather than serve a composer that can't send. - let bus = null, send = null, settingsWrite = null, projectFiles = null, projectFile = null, projectSave = null, chat = null, nameWrite = null, assistantStop = null, instancesWrite = null, routesWrite = null, busLine, system = null; + let bus = null, send = null, settingsWrite = null, projectFiles = null, projectFile = null, libraryFile = null, libraryStatus = null, projectSave = null, chat = null, nameWrite = null, assistantStop = null, instancesWrite = null, routesWrite = null, busLine, system = null; try { system = loadSystem(); const { dataRoot } = system; @@ -77,6 +77,7 @@ try { // Row 148: the spec and docs editor reads like the other reads and saves only under --send. projectFiles = args => call('project.files', args); projectFile = args => call('project.file', args); + libraryFile = args => call('library.file', args); libraryStatus = args => call('library.status', args); if (sendOn) {send = args => call('message.send', args);settingsWrite=args=>call('limits.set',args);projectSave=args=>call('project.file.save',args);nameWrite=args=>call('name.set',args);assistantStop=args=>call('assistant.stop',args);instancesWrite=args=>call('instances.set',args);routesWrite=args=>call('routes.set',args);} busLine = `Bus: ${options.business ?? 'the running bus host\'s business'}`; } catch (err) { @@ -97,7 +98,7 @@ try { } let server; try { - server = await startServer({ board: options.board, port: options.port, bus, reads, preview: options.preview, sso, proof, send, settingsWrite, projectFiles, projectFile, projectSave, chat, nameWrite, assistantStop, instancesWrite, routesWrite }); + server = await startServer({ board: options.board, port: options.port, bus, reads, preview: options.preview, sso, proof, send, settingsWrite, projectFiles, projectFile, libraryFile, libraryStatus, projectSave, chat, nameWrite, assistantStop, instancesWrite, routesWrite }); } catch (err) { sso?.close(); throw err; } let testerServer = null; if (testerPort !== null) { @@ -107,6 +108,7 @@ try { testerServer = await startServer({ host: '127.0.0.1', port: testerPort, board: options.board, tester: true, testerNoSignIn: ssoOn, testerProofHeader: proof?.header ?? null, testerAuthHeader: options.preview?.header ?? null, bus: testerBus, reads, projectFiles: args => call('project.files', args), projectFile: args => call('project.file', args), + libraryFile: args => call('library.file', args), libraryStatus: args => call('library.status', args), instancesWrite: args => call('instances.set', args), routesWrite: args => call('routes.set', args), nameWrite: args => call('name.set', args), // Row 177 (#1675): the tester's own chat, through the tester CLI; 7332's stays the human's. chat: createChat({ call }), diff --git a/packages/webui/tests/library-cli.test.mjs b/packages/webui/tests/library-cli.test.mjs new file mode 100644 index 000000000..a308e4c0d --- /dev/null +++ b/packages/webui/tests/library-cli.test.mjs @@ -0,0 +1,34 @@ +// C2-04 only wires callbacks. HTTP routes and pages are later rows. +import test from 'node:test'; +import assert from 'node:assert/strict'; +import { spawnSync } from 'node:child_process'; +import { readFileSync, writeFileSync } from 'node:fs'; +import { join } from 'node:path'; +import { tmp, fixture } from '../../cli/tests/helpers.mjs'; +import { HUMAN_CLI, TESTER_CLI } from '../../cli/src/transport.mjs'; + +test('C2-04 CLI: both listener callback pairs dispatch library verbs with their own transport', t => { + const root = tmp(t, 'library-cli-'), f = fixture(root), log = join(root, 'calls.jsonl'); + writeFileSync(log, ''); + const hook = join(root, 'hooks.mjs'); + // The loader supplies a fixture bus and captures callbacks instead of + // opening an HTTP listener. Production imports and routes are unchanged. + const serveSource = "export const DEFAULT_BOARD='http://127.0.0.1:9';export const previewOptions=()=>null;export const proofOptions=()=>null;export async function startServer(o){if(typeof o.libraryFile!=='function'||typeof o.libraryStatus!=='function')throw Error('missing library callbacks');await o.libraryFile({project:'books',path:'Book One/A.md'});await o.libraryStatus({project:'books'});return {address:()=>({port:0}),close:()=>{}};}"; + const busSource = `import {appendFileSync} from 'node:fs';export class BusReadError extends Error{};export const busCaller=o=>async(verb,args)=>{appendFileSync(${JSON.stringify(log)},JSON.stringify({verb,args,cli:o.cli??${JSON.stringify(HUMAN_CLI)}})+'\\n');return {};};`; + writeFileSync(hook, `import {registerHooks} from 'node:module'; + registerHooks({load(url,context,next){ + if(url.endsWith('/webui/src/serve.mjs'))return {format:'module',shortCircuit:true,source:${JSON.stringify(serveSource)}}; + if(url.endsWith('/webui/src/bus.mjs'))return {format:'module',shortCircuit:true,source:${JSON.stringify(busSource)}}; + return next(url,context); + }});`); + const result = spawnSync(process.execPath, [new URL('../src/cli.mjs', import.meta.url).pathname, + 'serve', '--business', 'acme', '--port', '0', '--tester-port', '7444'], + { env: { ...f.env, NODE_OPTIONS: `--import=${hook}` }, encoding: 'utf8', timeout: 10000 }); + assert.equal(result.status, 0, result.stdout + result.stderr); + const calls = readFileSync(log, 'utf8').trim().split('\n').filter(Boolean).map(JSON.parse); + assert.deepEqual(calls.map(c => [c.verb, c.cli]), [ + ['library.file', HUMAN_CLI], ['library.status', HUMAN_CLI], + ['library.file', TESTER_CLI], ['library.status', TESTER_CLI], + ]); + assert.equal(calls[0].args.path, 'Book One/A.md'); +});