fix(fleet): correct operator documentation validation
All checks were successful
ci/woodpecker/pr/ci Pipeline was successful

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Jarvis
2026-07-16 10:04:06 -05:00
parent 0aee2c0981
commit 17aa94ca1a
7 changed files with 214 additions and 45 deletions

View File

@@ -83,8 +83,12 @@ recorded in the M5 closure report and linked deferral evidence.
`mosaic agent` catalog.
- [x] Migration, quarantine, lifecycle, status, and troubleshooting documentation state that values of
legacy sensitive keys are never printed.
- [x] M5 documentation validation verifies links, schema/example validation, and that checklist
rows have owner/evidence or an explicit approved-existing deferral.
- [x] M5 documentation validation verifies required IA paths, local file and heading-fragment links,
the canonical roster through the production compiler/resolver, and fenced/canonical-example
safety checks.
- [ ] FCM-M5-001 does not deterministically assert owner/evidence/deferral metadata for every checklist
row. Closure and deferral reports provide human-reviewable evidence only; broader assertion
coverage remains unclaimed.
## Held downstream gates

View File

@@ -33,7 +33,7 @@ mosaic fleet doctor
mosaic fleet migrate-v1 preview --source <path> --decisions <path> --observations <path>
```
`get` is the read/show operation for one v2 agent. Full roster parsing and semantic validation occur on every v2 mutation/reconcile path; there is no separate mutable “config store.” The executable JSON Schema and validated example provide offline structural evidence.
`get` is the read/show operation for one v2 agent. Full roster parsing and semantic validation occur on every v2 mutation/reconcile path; there is no separate mutable “config store.” The executable JSON Schema and validated example provide offline structural evidence. The PRD requires an explicit programmatic `mosaic fleet validate` operation, but the current CLI does not expose one; do not substitute another command or claim that requirement is delivered. This remains an implementation gap for #758.
## JSON and exit behavior

View File

@@ -3,9 +3,9 @@
Roster-v2 `lifecycle.enabled` and `lifecycle.desired_state` are the only persisted lifecycle authority. Systemd, tmux, generated environment, and heartbeat state are derived or observed.
| Event | Desired-state write | Runtime effect | Safety boundary |
| --------------------------- | -------------------------------------------------------------------- | ------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
| --------------------------- | -------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `fleet create` | Adds enabled/stopped by default; `--persisted-start` records running | None | Generation-guarded; validates full roster/projections. |
| `fleet update` | May change enabled/desired state in complete payload | None | Generation-guarded; stable name immutable. |
| `fleet update` | Preserves the existing enabled/desired state; updates other mutable fields | None | Generation-guarded; stable name and lifecycle are immutable on this path. |
| `fleet delete` | Removes exact roster member | None | Removes only generated projection; retains local/quarantine evidence. |
| `fleet apply` / `reconcile` | Never | Rebuilds projections; starts only enabled/running; stops disabled or stopped roster members | Current generation, private lock/paths, semantic validity, holder ownership, no unmanaged named-socket sessions. |
| `fleet start <name>` | Never | One-shot exact service start | Exact enabled roster name and proven ownership. |
@@ -16,6 +16,6 @@ Roster-v2 `lifecycle.enabled` and `lifecycle.desired_state` are the only persist
| Cutover/canary | Held for FCM-M4-002 | Not implemented by preview | Must preserve every observed stopped state. |
| Rollback | Held for FCM-M4-002 | Not implemented | Must restore selected authority/projections without surprise starts or unmanaged targeting. |
Explicit apply/reconcile never starts a stopped roster agent. Direct lifecycle commands are explicit one-shot actions and do not persist intent. Change durable lifecycle only through generation-guarded CRUD and then review reconciliation. Reboot preservation for stopped/disabled agents is not yet guaranteed because current enabled units and launcher projections do not carry the persisted lifecycle fence; that acceptance evidence remains FCM-M3-002.
Explicit apply/reconcile never starts a stopped roster agent. Direct lifecycle commands are explicit one-shot actions and do not persist intent. The current update operation preserves `existing.lifecycle`; there is no delivered generation-guarded CRUD operation for changing durable lifecycle after creation. Reboot preservation for stopped/disabled agents is not yet guaranteed because current enabled units and launcher projections do not carry the persisted lifecycle fence; that acceptance evidence remains FCM-M3-002.
Missing/stale generation, concurrent writer, unsafe path, ownership mismatch, unmanaged session, unsupported runtime, invalid projection, and lifecycle precondition failures return stable redacted JSON and non-zero status. No command targets fuzzy names, arbitrary sockets/commands/channels/secrets, or generated files as authority. Legacy sensitive values are never printed.

View File

@@ -20,6 +20,15 @@ These are accepted existing DAG boundaries, not omissions silently claimed as de
M5 docs describe prerequisites and the preview boundary only. A ready preview is not migration or rollback evidence.
## Explicit validate-operation gap
- `FCM-REQ-03` requires a documented programmatic `mosaic fleet validate` operation.
- The current CLI does not expose that operation. Existing mutation/reconcile validation and the
documentation example test are not a replacement for the missing command.
- FCM-M5-001 documents this implementation gap without inventing syntax, JSON, exit behavior, or an
owning implementation card. Parent #758 must remain open until the requirement is implemented and
evidenced or the PRD/DAG is explicitly revised through the authoritative process.
## FCM-M5-002 hold
- Deterministic source-versus-installed asset revision detection and safe refresh implementation.

View File

@@ -23,14 +23,14 @@
| Complete DAG and artifact inventory | `docs/TASKS.md`; M0 inventory; executable disposition tests. |
| IA pages | Every path named by the M0 checklist exists and is linked from `docs/fleet/README.md`. |
| Examples | `docs/fleet/examples/roster-v2.yaml` validates through production v2 compiler/shared resolver; shipped artifact dispositions validate through declared production readers. |
| Links | Deterministic local Markdown link test covers the entire fleet book and sitemap. |
| Sensitive/example safety | Validator scans every fenced fleet-book example plus the canonical roster for sensitive-like keys, arbitrary command override, and hardcoded Tess/Ultron identities; docs consistently state value-free diagnostics. |
| Links | Deterministic local Markdown link test covers the entire fleet book and sitemap, including local heading-fragment resolution. |
| Sensitive/example safety | Validator scans every fenced fleet-book example plus the canonical roster for sensitive-looking keys, common credential formats, privileged commands, arbitrary command override, and hardcoded Tess/Ultron identities; docs consistently state value-free diagnostics. |
| Holds | `docs/reports/deferred/758-fleet-config-deferrals.md` records M3-002, M4-002, M5-002, compatibility, and repository-structure boundaries. |
## Documentation completion checklist
- [x] Root PRD exists and remains the #758 requirements authority.
- [x] Accepted project-specific fleet book is complete and indexed.
- [ ] The accepted project-specific fleet book is indexed, but it is not complete against `FCM-REQ-03`: the required explicit programmatic `mosaic fleet validate` operation is not implemented. The CLI reference and deferral report record this gap without inventing behavior.
- [x] Sitemap links the fleet entry point and operator-critical pages.
- [x] No HTTP/API/auth contract changed; OpenAPI/endpoint rows are not applicable.
- [x] Working evidence remains under `docs/scratchpads/`; closure and deferral evidence remains under `docs/reports/`.

View File

@@ -40,7 +40,9 @@ Deliver the accepted fleet documentation information architecture, operator work
- [x] Checklist mapped and docs authored.
- [x] Validation green.
- [x] Review/remediation complete.
- [ ] Commit, queue guard, push, PR.
- [x] Commit, queue guard, push, PR #789.
- [x] Rejected exact-head RoR findings repaired on a new descendant commit candidate.
- [ ] New exact-head review and CI after repair push.
## Tests and verification
@@ -61,6 +63,18 @@ Deliver the accepted fleet documentation information architecture, operator work
- Post-remediation `@mosaicstack/mosaic` lint/typecheck passed; package test passed 61 files / 1,045
tests; sanitization and resident-budget gates passed again.
- Post-PR exact-head RoR on rejected head `0aee2c09819fd06e28f927384ea56fa2ef374edf`
identified five blockers: update-lifecycle overclaim, missing explicit `fleet validate` gap,
fragment-blind link validation, unsupported checklist-evidence claim, and insufficient example safety
validation. Red-first regressions failed before implementation for missing-heading, privileged-command,
and credential-format fixtures. Repairs now preserve/document implementation truth, validate heading
fragments, narrow checklist claims, and scan fenced/canonical examples for common credential formats
and privileged commands without printing fixture values.
- Repair-focused fleet contracts: 7 files, 192 tests passed after review remediation; documentation
validator contributed 11 tests. Full gates passed: format; lint 23/23; typecheck 42/42; test 43/43
tasks with `@mosaicstack/mosaic` 61 files / 1,052 tests; sanitization; resident budget; and
`git diff --check`. New exact-head review/CI remain pending until the repair commit is pushed.
## Risks/blockers
- Checklist may include behavior intentionally deferred to M4-002/M5-002; such items must be recorded as approved-existing holds rather than claimed delivered.

View File

@@ -49,23 +49,157 @@ async function markdownFiles(root: string): Promise<string[]> {
}
function localMarkdownTargets(source: string): string[] {
return [...source.matchAll(/\[[^\]]*\]\(([^)]+)\)/g)]
.map((match): string => match[1] ?? '')
const link =
/\[[^\]]*\]\(\s*(?:<([^>]+)>|((?:\\.|[^()\s]|\([^()]*\))+))(?:\s+(?:"[^"]*"|'[^']*'|\([^)]*\)))?\s*\)/g;
return [...source.matchAll(link)]
.map((match): string => match[1] ?? match[2] ?? '')
.filter(
(target): boolean =>
target !== '' &&
!target.startsWith('#') &&
!target.startsWith('http://') &&
!target.startsWith('https://') &&
!target.startsWith('mailto:'),
)
.map((target): string => decodeURIComponent(target.split('#', 1)[0] ?? ''));
);
}
function markdownHeadingAnchors(source: string): Set<string> {
const anchors = new Set<string>();
let fence: { readonly marker: string; readonly length: number } | undefined;
for (const line of source.split('\n')) {
const fenceMatch = line.match(/^\s{0,3}(`{3,}|~{3,})(.*)$/);
if (fence === undefined && fenceMatch !== null) {
const run = fenceMatch[1] ?? '';
fence = { marker: run[0] ?? '', length: run.length };
continue;
}
if (fence !== undefined) {
const closingRun = line.match(/^\s{0,3}(`{3,}|~{3,})\s*$/)?.[1];
if (
closingRun !== undefined &&
closingRun[0] === fence.marker &&
closingRun.length >= fence.length
) {
fence = undefined;
}
continue;
}
const heading = line.match(/^\s{0,3}#{1,6}\s+(.+?)\s*#*\s*$/)?.[1];
if (heading === undefined) continue;
const base = heading
.replace(/!?\[([^\]]*)\]\([^)]*\)/g, '$1')
.replace(/<[^>]*>/g, '')
.replace(/[`*_~]/g, '')
.toLowerCase()
.trim()
.replace(/[^\p{L}\p{N}\s-]/gu, '')
.replace(/\s+/g, '-');
let anchor = base;
let duplicate = 0;
while (anchors.has(anchor)) {
duplicate += 1;
anchor = `${base}-${duplicate}`;
}
anchors.add(anchor);
}
return anchors;
}
function markdownLinkViolations(
sourcePath: string,
source: string,
documents: Readonly<Record<string, string>>,
): string[] {
const violations: string[] = [];
for (const target of localMarkdownTargets(source)) {
const [encodedPath = '', encodedFragment] = target.split('#', 2);
const targetPath = decodeURIComponent(encodedPath);
const normalizedTarget = resolve('/', dirname(sourcePath), targetPath).slice(1);
const targetSource = documents[normalizedTarget];
if (targetSource === undefined) {
violations.push(`${sourcePath} -> ${target}: missing file`);
continue;
}
if (encodedFragment !== undefined) {
const fragment = decodeURIComponent(encodedFragment);
if (fragment === '' || !markdownHeadingAnchors(targetSource).has(fragment)) {
violations.push(`${sourcePath} -> ${target}: missing heading`);
}
}
}
return violations;
}
function exampleSafetyViolationKinds(source: string): string[] {
const kinds = new Set<string>();
const sensitiveKey = /(?:secret|token|password|credential|MOSAIC_AGENT_COMMAND)/i;
const credentialFormat =
/(?:\bAKIA[0-9A-Z]{16}\b|\bAIza[0-9A-Za-z_-]{35}\b|\bgh[pousr]_[A-Za-z0-9]{20,}\b|\bgithub_pat_[A-Za-z0-9_]{20,}\b|\bglpat-[A-Za-z0-9_-]{20,}\b|\bnpm_[A-Za-z0-9]{20,}\b|\bsk_live_[A-Za-z0-9]{16,}\b|\bxox[baprs]-[A-Za-z0-9-]{10,}\b|\bBearer\s+[A-Za-z0-9._~+/=-]{16,}\b|\beyJ[A-Za-z0-9_-]{8,}\.[A-Za-z0-9_-]{8,}\.[A-Za-z0-9_-]{8,}\b|-----BEGIN [A-Z ]*PRIVATE KEY-----|\b[A-Za-z][A-Za-z0-9+.-]*:\/\/[^\s/:]+:[^\s/@]+@)/;
const privilegedCommand =
/(?:^|[\n;&|])\s*(?:[$#>]\s*)?(?:env\s+(?:[A-Za-z_][A-Za-z0-9_]*=[^\s]+\s+)*|command\s+)*(?:(?:sudo|doas|pkexec)\s+|su\s+(?:-|--command\b|-c\b)|(?:systemctl|service|mount|umount|reboot|shutdown|poweroff|halt|chown|chmod|useradd|usermod|groupadd|visudo)\s+)/m;
if (sensitiveKey.test(source)) kinds.add('sensitive-key');
if (credentialFormat.test(source)) kinds.add('credential-format');
if (privilegedCommand.test(source)) kinds.add('privileged-command');
if (/\b(?:Tess|Ultron)\b/.test(source)) kinds.add('identity');
return [...kinds].sort();
}
function fencedCodeBlocks(source: string): string[] {
return [...source.matchAll(/```[^\n]*\n(.*?)```/gs)].map((match): string => match[1] ?? '');
}
const UNSAFE_EXAMPLE_FIXTURES = [
{
expected: 'privileged-command',
source: ['cd /srv && su', 'do systemctl restart example'].join(''),
},
{ expected: 'credential-format', source: ['ghp_', 'a'.repeat(36)].join('') },
{
expected: 'credential-format',
source: ['eyJ', 'a'.repeat(12), '.', 'b'.repeat(12), '.', 'c'.repeat(12)].join(''),
},
] as const;
describe('documentation validation regressions', (): void => {
it('rejects a local Markdown link whose heading fragment does not exist', (): void => {
expect(
markdownLinkViolations('docs/fleet/source.md', '[broken](target.md#missing)', {
'docs/fleet/target.md': '# Present',
}),
).toEqual(['docs/fleet/source.md -> target.md#missing: missing heading']);
});
it('accepts balanced-parenthesis link destinations and visible-text heading anchors', (): void => {
expect(localMarkdownTargets('[guide](guide-(legacy).md#setup "Guide")')).toEqual([
'guide-(legacy).md#setup',
]);
expect(markdownHeadingAnchors('# [Fleet API](cli.md) behavior')).toContain(
'fleet-api-behavior',
);
});
it('keeps longer fenced blocks closed only by an equal-or-longer fence', (): void => {
expect(markdownHeadingAnchors('````markdown\n```\n# Not a heading\n````\n# Present')).toEqual(
new Set(['present']),
);
});
it('assigns a free suffix when a prior heading already occupies the next duplicate anchor', (): void => {
expect(markdownHeadingAnchors('# Foo\n# Foo-1\n# Foo')).toEqual(
new Set(['foo', 'foo-1', 'foo-2']),
);
});
it.each(UNSAFE_EXAMPLE_FIXTURES)(
'classifies unsafe example fixture as $expected',
({ expected, source }): void => {
expect(exampleSafetyViolationKinds(source)).toContain(expected);
},
);
});
describe('fleet operator documentation', (): void => {
it('ships every accepted information-architecture page', async (): Promise<void> => {
await expect(
@@ -73,20 +207,34 @@ describe('fleet operator documentation', (): void => {
).resolves.toHaveLength(REQUIRED_FLEET_PAGES.length);
});
it('resolves every local Markdown link in the fleet book and sitemap', async (): Promise<void> => {
it('resolves every local Markdown link and heading fragment in the fleet book and sitemap', async (): Promise<void> => {
const files = [...(await markdownFiles(fleetDocs)), join(repositoryRoot, 'docs', 'SITEMAP.md')];
const missing: string[] = [];
const documents: Record<string, string> = {};
for (const file of files) {
const source = await readFile(file, 'utf8');
const relative = file.slice(repositoryRoot.length + 1);
documents[relative] = await readFile(file, 'utf8');
}
const violations: string[] = [];
for (const [sourcePath, source] of Object.entries(documents)) {
for (const target of localMarkdownTargets(source)) {
const encodedPath = target.split('#', 1)[0] ?? '';
const targetPath = resolve(
dirname(join(repositoryRoot, sourcePath)),
decodeURIComponent(encodedPath),
);
const relativeTarget = targetPath.slice(repositoryRoot.length + 1);
if (documents[relativeTarget] === undefined) {
try {
await readFile(resolve(dirname(file), target), 'utf8');
documents[relativeTarget] = await readFile(targetPath, 'utf8');
} catch {
missing.push(`${file.slice(repositoryRoot.length + 1)} -> ${target}`);
// The deterministic validator below records the missing target without exposing content.
}
}
}
expect(missing).toEqual([]);
violations.push(...markdownLinkViolations(sourcePath, source, documents));
}
expect(violations).toEqual([]);
});
it('validates the canonical documentation example through the production compiler and resolver', async (): Promise<void> => {
@@ -105,26 +253,20 @@ describe('fleet operator documentation', (): void => {
]);
});
it('keeps every fenced fleet example free of sensitive values, arbitrary commands, and product-hardcoded identities', async (): Promise<void> => {
it('keeps every fenced fleet example free of sensitive values, privileged commands, arbitrary command overrides, and product-hardcoded identities', async (): Promise<void> => {
const violations: string[] = [];
for (const file of await markdownFiles(fleetDocs)) {
const source = await readFile(file, 'utf8');
for (const [index, block] of fencedCodeBlocks(source).entries()) {
if (/(?:secret|token|password|credential|MOSAIC_AGENT_COMMAND)/i.test(block)) {
violations.push(`${file.slice(repositoryRoot.length + 1)}#block-${index + 1}: sensitive`);
}
if (/\b(?:Tess|Ultron)\b/.test(block)) {
violations.push(`${file.slice(repositoryRoot.length + 1)}#block-${index + 1}: identity`);
for (const kind of exampleSafetyViolationKinds(block)) {
violations.push(`${file.slice(repositoryRoot.length + 1)}#block-${index + 1}: ${kind}`);
}
}
}
const rosterSource = await readFile(join(fleetDocs, 'examples', 'roster-v2.yaml'), 'utf8');
if (/(?:secret|token|password|credential|MOSAIC_AGENT_COMMAND)/i.test(rosterSource)) {
violations.push('docs/fleet/examples/roster-v2.yaml: sensitive');
}
if (/\b(?:Tess|Ultron)\b/.test(rosterSource)) {
violations.push('docs/fleet/examples/roster-v2.yaml: identity');
for (const kind of exampleSafetyViolationKinds(rosterSource)) {
violations.push(`docs/fleet/examples/roster-v2.yaml: ${kind}`);
}
expect(violations).toEqual([]);
});