jason.woltje
3b88ef399d
(cherry picked from commit367fd0d240) (cherry picked from commit068e1e0f1c)
337 lines
20 KiB
Markdown
337 lines
20 KiB
Markdown
# packages/library
|
|
|
|
Reads a writing repo by a mapping of folders to books, types, canon and playground. This is Case 2
|
|
row C2-01 (`docs/plans/2026-10-20_case2-rows.md`), built from design sections 1.1 to 1.3 and 7.4 in
|
|
`agents/gosalyn/work/case2/DESIGN.md` on `gosalyn/case2-design`.
|
|
It is pure code with no state. Business file loading (C2-04), sync (C2-03a), the index (C2-05a) and the
|
|
guard (C2-06) build on it.
|
|
|
|
## Mapping
|
|
|
|
`checkMapping(entry, where, { fail })` checks a `kind: "library"` project entry and returns a
|
|
deep-frozen copy. `where` names the entry in messages. `fail(code, message)` defaults to throwing a
|
|
`LibraryError`. C2-04 passes its own so a business file refusal carries the file name.
|
|
|
|
```json
|
|
{
|
|
"kind": "library",
|
|
"remote": "https://git.example.invalid/her/books.git",
|
|
"branch": "main",
|
|
"user": "books-bot",
|
|
"vaultPrefix": "books",
|
|
"types": { "Characters": "character", "Places": "place", "Research": "research" },
|
|
"manuscript": ["Chapters"],
|
|
"universes": {
|
|
"harbor": { "name": "Harbor", "books": ["Book One"], "playground": ["Book One/Playground/"] }
|
|
}
|
|
}
|
|
```
|
|
|
|
- `remote` is https with no credentials, query or fragment. `branch` is a plain ref name.
|
|
- `user` is the bot account name from design step 3, required for sync. It is optional in the
|
|
mapping and checked with row 199's account name pattern, `[A-Za-z0-9_.-]`, 1 to 100 characters.
|
|
- `vaultPrefix` is optional, one folder name.
|
|
- `editor` is optional, `{ "name", "email" }` plus an optional `"user"`: the ident her Console saves commit under
|
|
(C2-01d). Each value is text that is non-empty after trim, with no control character
|
|
(`U+0000`-`U+001F`, `U+007F`-`U+009F`), `<` or `>`, and no whitespace or git-stripped character at
|
|
either end. `user` is her Gitea login for the `library-human` slot (sage-2925), checked as the
|
|
entry's `user` is. `editor` and its `user` come back only when given; a save without them refuses
|
|
in C2-02.
|
|
- `types` maps a folder name to `character`, `place`, `research`, `tracker` or `summary`.
|
|
`manuscript` lists folder names and shares none with `types`.
|
|
- A universe has `name`, `books` and `playground`. A hybrid adds `kind: "hybrid"`, two or more
|
|
plain-universe `sources`, `constraints` (each one `{ "exclude": ... }` or
|
|
`{ "pin": { "entity", "field", "value" } }`) and `notes`, and may have `books: []`. A draft
|
|
universe has only `kind: "draft"`, `name` and one or more `playground` prefixes, and comes back as
|
|
`{ kind: "draft", name, books: [], playground, supporting }`. `kind` is absent, `"hybrid"` or `"draft"`.
|
|
- A playground prefix ends in `/` and has one of three levels (C2-01c). Book level lies inside one of
|
|
its own entry's book folders. Universe level, in a plain universe or a hybrid, lies outside every
|
|
book folder. A draft's prefixes all lie outside every book folder. No prefix holds another or is
|
|
listed twice, across every entry.
|
|
- Every universe kind may also have `supporting`, a list of prefixes where agents may write (C2-01d,
|
|
M12). It comes back on every universe, `[]` when absent. A supporting prefix follows the playground's
|
|
rules: it ends in `/`, has the same three levels, holds no other supporting prefix and is listed
|
|
once, across every entry.
|
|
- Between the two lists: a supporting prefix may equal a playground prefix of its own entry, and a
|
|
playground prefix may hold one of its own entry's supporting prefixes. A supporting prefix may not
|
|
hold a playground prefix, which scope would then never reach, and may not equal, hold or lie inside
|
|
another entry's playground prefix.
|
|
|
|
Refusals:
|
|
|
|
| Code | When |
|
|
|---|---|
|
|
| `library-playground-other-universe` | a playground prefix lies inside, or holds, a book folder of another entry |
|
|
| `library-supporting-other-universe` | a supporting prefix lies inside, or holds, a book folder of another entry |
|
|
| `library-mapping-invalid` | any other shape fault: unknown key, bad remote or branch, an unknown `kind`, a book folder in two universes, a book folder that holds another, a playground or supporting prefix that holds another of its list or is listed twice, a prefix equal to a book folder or holding its own entry's book, a supporting prefix that holds a playground prefix or overlaps another entry's, a bad `editor`, a draft with `books` or with no prefix, a hybrid source that is unknown, a hybrid or a draft |
|
|
|
|
## Scope
|
|
|
|
`classify(mapping, path)` decides scope from the path alone. Only paths ending in lowercase `.md` are
|
|
mapped. It checks prefixes first, supporting before playground. A path under any entry's supporting
|
|
prefix is `supporting`, and otherwise one under a playground prefix is `playground`. Either way that
|
|
entry is `universe` and the book folder holding the prefix is `book`, or `null` at universe level and
|
|
in a draft. Any other path under a book folder is `canon`. Everything else is `unmapped`. The type is
|
|
the first folder after the prefix, or after the book: a `types` name, `manuscript`, or `null`.
|
|
|
|
## Agent write paths
|
|
|
|
`supportingPath(mapping, universe, path, tree)` returns `path` when an agent may write it in that
|
|
universe, else it refuses `library-write-story-path` (C2-01d, M12). `tree` is the array `listTree`
|
|
returns for the commit the write lands on; anything else is a TypeError. It reads no git. The checks
|
|
run in this order, so a path with two faults always names the same one:
|
|
|
|
1. a string of at most 1,024 UTF-8 bytes;
|
|
2. no control character (the `editor` class), and the message leaves the path out;
|
|
3. no backslash;
|
|
4. not absolute;
|
|
5. no empty, `.` or `..` segment, which also refuses `//` and a trailing `/`;
|
|
6. `classify` gives `supporting` in that same universe, which refuses unmapped, `.canvas`, `.base`
|
|
and attachment paths, canon, a playground-only folder and another universe's prefix;
|
|
7. no folder on the way, and not the path itself, is a symlink (`120000`) or gitlink (`160000`) in
|
|
`tree`;
|
|
8. no folder on the way, and not the path itself, matches a tree path or tree folder after
|
|
`normalize('NFC').toLowerCase()` unless it is that exact name. APFS folds case and Unicode form,
|
|
so her checkout would put the file somewhere else. A tree that already holds case twins refuses a
|
|
write to either (R7).
|
|
|
|
A delete's further rule, that the path exists at the tip as a `100644` blob, belongs to the C2-02
|
|
broker.
|
|
|
|
## Reader
|
|
|
|
`readLibrary({ repo, commit, mapping })` reads one commit, never a working tree. `repo` may be a
|
|
clone or a bare mirror. `commit` must be a full 40-hex id that names a commit, else `library-commit`.
|
|
Git runs through `git()` and `gitBytes()` from `packages/project/src/git.mjs`, so the same safe config,
|
|
environment and ceiling directory apply. A failed git call refuses `library-git`.
|
|
|
|
It returns `{ commit, files, problems }`, deep-frozen and sorted. Each file has its path, scope,
|
|
universe, book, type, numbered lines, `bodyLine`, parsed `frontmatter` or `null`, `frontmatterRaw`
|
|
and `links`. Front matter `appears_in` and `relationships` wikilinks count as links. A link in a block
|
|
list carries its list item's line. One in a scalar or flow value carries its key's line.
|
|
|
|
Problems carry `{ code, path, line, message }`:
|
|
|
|
| Rule | Meaning | File indexed |
|
|
|---|---|---|
|
|
| R1 | not a regular `100644` blob of at most 1 MiB, or not UTF-8, or holds a NUL | no |
|
|
| R2 | path is unmapped | no |
|
|
| R3 | front matter not fully parsed, `line` is where parsing stopped | yes, front matter kept raw |
|
|
| R4 | link resolves to no file or to more than one | yes, link dropped |
|
|
| R5 | canon file links into the playground | yes |
|
|
| R6 | canon file links into another universe, a hybrid's sources excepted, or a hybrid's file links into a path its `exclude` covers | yes |
|
|
| R7 | two paths differ only in case | no, neither |
|
|
| R8 | pinned value changed: a hybrid's pinned entity is missing or its `field` differs from `value`, or a hybrid version of it sets `field` to another value | yes |
|
|
|
|
The front matter parser takes scalars, quoted strings, flow lists, block lists and wikilinks. Nested
|
|
maps, anchors, block scalars and duplicate keys stop it with R3.
|
|
|
|
## Links
|
|
|
|
`resolver(paths, { vaultPrefix })` tries four steps in order, each with the target as written and
|
|
then with `.md` added:
|
|
|
|
1. `root`: the path from the repo root.
|
|
2. `vault-prefix`: the same path with `vaultPrefix/` stripped.
|
|
3. `relative`: relative to the linking file, never above the repo root.
|
|
4. `shortest`: a target with no `/` that matches the base name of exactly one indexed file.
|
|
|
|
It resolves only against indexed files. A link to a `.canvas` file or an attachment is R4. A link into
|
|
another repo resolves to nothing. `extractLinks` skips fenced and inline code.
|
|
|
|
A `![[...]]` embed resolves like any link. A body embed that resolves to nothing and whose target ends
|
|
in an extension other than `.md`, in any case, embeds an attachment, and it is never R4. An extension is
|
|
a dot and up to 8 letters and digits, at least one a letter. Any other unresolved embed is R4, and so is
|
|
an unresolved front matter value written as an embed (C2-01b N3, sage-1900).
|
|
|
|
## renderReply
|
|
|
|
`renderReply(body, { check, changed, name })` turns a stored reply into what both channels show. It
|
|
replaces C2-01's items renderer and is pure. A body is free text, or the design 5.2 object
|
|
`{ "text", "citations" }` with library cites `{ path, commit, lines, quote, as? }` and web cites
|
|
`{ url, quote }`. Any other body, a JSON object with another key or a bad cite included, renders as
|
|
free text: the whole body, with no citations.
|
|
|
|
`check` is the stored check result (C2-06), `{ citations: [{ scope, agent }] }` in body order. A path
|
|
cite's badge (`canon`, `playground` or `supporting`) and its agent-written label come only from it. A
|
|
url cite is always `web`, and is a link only when its scheme is `https`. `changed({ path, commit }, i)`
|
|
says whether the file's last commit is newer than the cited one. `plain` is the rendering as text:
|
|
|
|
```
|
|
Mara's eyes are grey in chapter 3.
|
|
[canon] Book One/Chapters/Chapter 03.md:41 @aaaaaaa "her grey eyes"
|
|
[supporting] Notes/Eyes.md:9-10 @aaaaaaa "green eyes" (agent-written: acme/scribe) (changed since)
|
|
[web] https://example.invalid/tides "spring tides"
|
|
```
|
|
|
|
The broker's write notice, `{ "notice": "library-write", op, path, commit }`, gives the line
|
|
`<name> wrote <path> @<commit>` or `<name> deleted <path> @<commit>`. A notice is recognized only
|
|
when `check` is null or absent: a checked agent message with that shape stays whole-body free text
|
|
with no Revert control. Pre-217 history has no check, so a notice-shaped older body still renders as
|
|
a notice; the accepted legacy residual is recorded in the row225 BUILD.md.
|
|
|
|
## Survey
|
|
|
|
```sh
|
|
mosaic library survey <repo> [--mapping <file>]
|
|
```
|
|
|
|
It reads the repo's HEAD commit and prints folder names, file counts by extension, front matter key
|
|
sets, link-form counts and the count of body embeds whose target has an extension other than `.md`,
|
|
which still count as embeds. That count goes by the name alone, so an embed of `Dr.Quinn.md` written
|
|
`![[Dr.Quinn]]` counts there too. Folder names print as JSON strings, with DEL, C1 and bidi controls,
|
|
U+061C among them, also as `\uXXXX`. It never prints file text, front matter values, file names or
|
|
link targets.
|
|
A key that is not a short identifier prints as `(other key)`, since it could be her words. With
|
|
`--mapping` it adds scope counts, resolver step counts and problem counts by rule.
|
|
|
|
Exit codes follow `packages/project`: 0 done, 2 the repo can't be read (`library-git`,
|
|
`library-commit`), 3 a bad mapping, 4 usage. `scripts/mosaic` sends `library survey` here. Other
|
|
library verbs will be bus verbs.
|
|
|
|
## Sync
|
|
|
|
`sync(context, { slot, accept })` (C2-03a, design 7.2) fetches the entry's `branch` from its `remote`
|
|
into `refs/remotes/library/<branch>` of the C2-04 mirror, then publishes under the library lock:
|
|
|
|
| Fetched tip | `served` | State |
|
|
|---|---|---|
|
|
| a fast-forward, or the first sync | moves to the tip | `problems` rebuilt from the tip |
|
|
| the served commit | stays | `lastFetch` set |
|
|
| not a fast-forward | stays | `diverged` holds the tip, `library-diverged`, until `accept` |
|
|
| fetch failed, slot missing | stays | `lastError` holds `library-fetch-failed` or `library-credential-missing` |
|
|
|
|
`lastFetch` is the last fetch that succeeded. Once diverged, `served` stays put until a human runs
|
|
`library.sync` with `accept` set to the fetched tip, even if a later tip is a fast-forward again. A
|
|
tip back at the served commit clears it. A
|
|
stale `accept` refuses `library-accept-stale`. `accept` also starts the state over, so it is how a
|
|
changed `branch` gets a fresh record. An entry with no `user` refuses `library-remote-invalid` and
|
|
runs no git.
|
|
|
|
`syncCommand('fetch', url, { mirror, branch, user, slot })` gives the one argv and env. The argv is
|
|
`SAFE` from `packages/project/src/git.mjs`, then `-c protocol.https.allow=always -c
|
|
http.followRedirects=false -c credential.helper= -c core.createObject=rename`, then `-C <mirror>` and
|
|
the fetch. `core.createObject=rename` is there because git's default files an object by link then
|
|
unlink, and a reader's mirror check refuses a file with two links. The env is
|
|
`gitEnv()` with `LC_ALL=C`, `GIT_ASKPASS` naming `bin/askpass`, and the host, user and slot path. The
|
|
helper answers only `Username for 'https://<host>': ` and `Password for 'https://<user>@<host>': `,
|
|
the second from the slot's first line, and exits 1 on any other prompt. Sync checks that the slot is
|
|
a private regular file and never opens it. The token is in no argv, env, URL, config or log.
|
|
|
|
Sync never runs gc or prune. A file read takes `served` under the lock and reads its objects after it
|
|
lets go, so an object must outlive the next move of `served`. The bus host runs `sync` for every entry
|
|
with a `user`, once at start and then every 60 seconds; the slot is
|
|
`<configDir>/secrets/<business>/library-gitea`. `library.sync` is the human's bus verb for the same.
|
|
|
|
The human transport (`packages/bus/src/human-cli.mjs`) waits 130 seconds for `library.sync`, past the
|
|
server's 120-second hold, and 5 seconds for every other verb. A client that gives up before the answer
|
|
gets `outcome-unknown`, but the sync keeps running: a second call meanwhile gets `library-sync-busy`,
|
|
and `library.status` shows the result once it ends. A sync that outlasts the 120-second hold gets
|
|
`request-timeout` from the server and still finishes the same way. The Console has no library route yet: `cli.mjs`
|
|
passes `librarySync` to `startServer`, and `serve.mjs` doesn't read it.
|
|
|
|
### Push and credentials (C2-03b)
|
|
|
|
`push(context, { commit, credential, retried })` sends a full commit id, already in the mirror, to the
|
|
entry's `branch` as `push --porcelain --no-verify --no-follow-tags --end-of-options <url>
|
|
<commit>:refs/heads/<branch>`, through the same prefix, env and helper as the fetch. The refspec has no
|
|
`+`, so the remote takes a fast-forward only.
|
|
|
|
| git says | Answer |
|
|
|---|---|
|
|
| the push landed | `{ pushed: true }` |
|
|
| `[rejected] (fetch first)` or `(non-fast-forward)` | one fetch with the same credential, then `{ pushed: false, raced: true, tip }` |
|
|
| the same, with `retried: true` | `stale-base` |
|
|
| the same, while a sync of the mirror runs | `library-sync-busy`, with no fetch |
|
|
| anything else, `[remote rejected]` (a hook) included | `library-push-failed` |
|
|
|
|
Push moves no local ref; C2-02 re-applies and moves `served`. git's output is never kept.
|
|
|
|
A credential is `{ kind, user, file }`, with `kind` one of `slot`, `chief`, `seat` or `human`, or the
|
|
bus's `{ refused: code }`. The helper reads only `file`. It opens the file on fd 3 and refuses unless
|
|
the open file and the path have the same device and inode, and the file is regular, the bus's uid's,
|
|
mode 0600 and at most 16384 bytes, as `readSecretFile` in `packages/bus/src/gitea-grant.mjs` checks a
|
|
grant store file. A FIFO swapped in after `test -f` blocks the open, the same residual
|
|
`readSecretFile` has.
|
|
|
|
- A seat writes with its own Gitea user and grant store file, `<storeRoot>/<business>/<agent>/gitea`.
|
|
- `humanCredential(context, { user, configDir })` gives her save its credential: her Gitea login and
|
|
`<configDir>/secrets/<business>/library-human`. A missing login, or a slot that is missing, a link,
|
|
another uid's or not 0600, refuses `library-human-missing` before any git call.
|
|
- The fetch takes the broker's choice, `libraryFetchCredential(business)`. The chief is a `roles`
|
|
entry whose definition is `chief-of-staff`; a fire keeps it there. With no Gitea grant evidence for
|
|
a chief, the fetch uses slot `library-gitea` and the entry's `user`. Once there is some, only the one
|
|
listed chief's live, unheld grant answers, with that grant's user and store file. Anything else
|
|
refuses `library-fetch-no-grant`, and so does any Gitea grant evidence in a business with no chief
|
|
entry. That refusal opens no file and runs no git. The bus answers its code; the timer's log line
|
|
adds `fix: "add the chief of staff, or grant the library through C2-21"`.
|
|
|
|
The sync state adds `credential`, `{ kind, user }` of the last fetch, or `null` when none ran. It is
|
|
optional, so a state written before C2-03b still reads.
|
|
|
|
## Tests
|
|
|
|
```sh
|
|
node --test packages/library/tests/
|
|
```
|
|
|
|
Fixtures are made-up repos built per test in a temp dir. A marker string stands in for her text, and
|
|
the survey tests check that it never appears in output.
|
|
|
|
### C2-02 supporting writes and human saves
|
|
|
|
The host exposes `library.doc.write {universe,path,base,content}` and
|
|
`library.doc.delete {universe,path,base}` only to a creative session holding
|
|
`library.write` and a live, unheld Gitea grant for the library project. The
|
|
broker chooses the project from the universe mapping, the author/committer
|
|
from the instance identity, and the credential from its grant store. `base`
|
|
is the served-tree blob id, or `null` to create. Supporting-path checks run
|
|
before remote effects and against each fetched tip; deletes require an
|
|
existing `100644` blob. Content is UTF-8 without NUL and at most 1 MiB;
|
|
agent requests still have the ordinary 64 KiB socket envelope limit.
|
|
|
|
Each operation is one plumbing commit with one parent, a fixed
|
|
`Mosaic: write <path>` or `Mosaic: delete <path>` message, and the trailer
|
|
`Mosaic-Agent: <business>/<instance>`. Pushes are fast-forward only. A moved
|
|
remote tip retries once when the target blob still equals `base`; otherwise
|
|
it refuses `stale-base`. A failed push leaves no queued work or local ref
|
|
change. A successful push updates served/tracking, rebuilds the index and
|
|
records `lastPush: {commit,at}`. The broker inserts the exact fixed notice
|
|
`{notice:"library-write",op,path,commit}` from the writer to the human in
|
|
that thread, independently of the agent-message citation guard.
|
|
|
|
`library.file.save {project,path,base,content}` and
|
|
`library.revert {project,commit}` are human-only. Project is the business
|
|
project key. A save reaches any mapped `.md` file under the structural path
|
|
rules, uses the mapping's `editor` as author and committer, and pushes with
|
|
slot `library-human`. Revert accepts only a reachable, one-parent,
|
|
one-path commit with this business's agent trailer, and restores its
|
|
parent's blob with the agent commit's blob as `base`. The result is
|
|
`{project,path,reverted,commit}`. Revoking/firing the original writer does
|
|
not prevent a human revert. Missing editor refuses
|
|
`library-mapping-invalid`; missing/unsafe human slot refuses
|
|
`library-human-missing`, before Git. An agent calling either human verb
|
|
gets `human-required`. Only `library.file.save` joins the larger Console
|
|
save envelope exception; agent writes retain 64 KiB.
|
|
|
|
The write guard excludes concurrent sync and admits a race fetch only for
|
|
its opaque owner token. The library flock spans fetch/build/push as the
|
|
brief requires, so readers can receive `library-sync-busy` after the
|
|
existing five-second lock wait while network Git runs (MVP tradeoff,
|
|
sage-3628). Publication errors return `outcome-unknown`: served is restored
|
|
to match old state unless the new state was already published. The remote
|
|
commit remains; callers must inspect status rather than retry blindly.
|
|
|
|
The human push login is `editor.user` (optional in the mapping, required
|
|
for save/revert), distinct from `user`, which names the fetch bot.
|
|
An absent editor login refuses `library-human-missing`; no fallback uses
|
|
the bot's login. Once an agent push succeeds, the broker stores the exact
|
|
write notice with that pushed commit id before local publication. If
|
|
publication then fails, the agent receives `outcome-unknown` and the
|
|
notice remains. Revert can refuse `library-revert-not-agent` until a sync
|
|
makes the commit reachable from served. No notice outbox or retry is
|
|
needed. If the Console store itself refuses the notice insert, the caller
|
|
still receives `outcome-unknown` and a separate metadata-only host log
|
|
records `library-notice-store-failed` with the business, instance, operation
|
|
and pushed commit id; it contains no file text or credential.
|