mosaicstack/stack
25
0
Fork 1
Code Issues 344 Pull Requests 13 Actions Packages Projects Releases 15 Wiki Activity

git maintenance has a failure channel and no liveness channel — two hosts, two mechanisms, one blind spot #962

New Issue
Open
opened 2026-07-30 19:31:02 +00:00 by Ghost · 0 comments
Ghost commented 2026-07-30 19:31:02 +00:00
Copy Link
Copy Source

Summary

Git reports that maintenance ran and failed. Nothing reports that maintenance has not run.
Add a liveness check for git maintenance to the fleet health surface, and drive maintenance from a
scheduler rather than from git's automatic triggers.

Two hosts hit unbounded object growth this week by two different mechanisms. Both were invisible for
the same reason, and the observable we all reached for — .git/gc.log — cannot distinguish healthy
from dead.

The gap

git gc --auto writes .git/gc.log when it runs and fails, then suppresses further automatic runs
for gc.logExpiry (default 1 day), re-printing the stale error instead of retrying.

There is no counterpart on the success path. Git records no queryable "last successful maintenance"
timestamp. Therefore:

.git/gc.log possible states
PRESENT maintenance ran and failed — diagnostic
ABSENT maintenance is healthy · maintenance was never invoked · maintenance failed and the log was since removed — not diagnostic

We read that absent file on five repositories across two hosts and drew opposite conclusions from
it, correct in some cases and wrong in others, based entirely on what else each reader happened to
know. A file that is written only on failure cannot answer a liveness question.

Specimen A — suppression path

Repository on HOST_A reached ~38 GB across 131 pack files and filled the volume. Work stopped; the
message poller went blind (it fail-closed correctly and reported the blindness, which is the only
reason this surfaced as an incident rather than as a silent channel).

Configuration was pure git defaults on both the failing and the healthy host — gc.auto,
gc.autoPackLimit, gc.autoDetach, maintenance.auto all unset. Default gc.autoPackLimit is 50,
so 131 packs is 2.6x past the threshold that exists to prevent exactly this.

Most probable mechanism: a disk-pressure spiral. Auto-gc triggered at 50 packs, failed with ENOSPC,
wrote gc.log, self-suppressed for a day, and repeated while packs climbed and the volume filled.

Confirmation is pending and may be recoverable. The operator ran git gc --prune=now, which
deletes gc.log on success — erasing the artifact that would have named the cause. However, while
gc.log exists git copies its contents into the stderr of every subsequent porcelain command. On
a host where git runs under a scheduler with journald capture, hundreds to thousands of copies of
the original error survive the file's deletion. That log search is the outstanding confirmation
step.

Specimen B — never-invoked path, no failure record at all

Repository on HOST_B, measured read-only, no locks taken:

count-objects -v: count=57068  in-pack=105121  packs=16  prune-packable=6950  garbage=0
loose objects older than 14 days: 42,406 of 57,068  (74%)
age curve: <1d:100 · <7d:2,881 · <14d:12,900 · <30d:21,666 · <60d:54,792
newest packs: Jul 26 15:56 (141MB) · Jul 26 15:44 (800KB) · Jul 22 16:12 (577KB)
.git/gc.log: ABSENT

Loose count is ~8.5x past the default gc.auto threshold, and .git/gc.log is absent — so the
suppression trap cannot be the cause here. Same outcome, different path.

Two independent readings confirm maintenance is not completing:

  1. 42,406 unreachable objects past the default two-week gc.pruneExpire grace window. Any
    successful gc prune would have removed them.
  2. prune-packable=6950 — loose but reachable objects that a repack folds in. A completed repack
    takes this to ~0.

This falsified the benign reading we considered first, that a healthy gc was deliberately
declining to touch a large population of young unreachable objects. The population is old.

Instrument caveat worth carrying into any check built from this: the recent 141 MB pack is not
evidence that gc succeeded. Fetch writes packs too, and a large pack landing beside 15 undisturbed
siblings is a fetch artifact, not a consolidation — a repack would have folded the siblings and
zeroed prune-packable, and neither happened. A file's mtime tells you that something wrote it,
not what wrote it. Newest-pack-mtime is therefore a weak liveness proxy on repositories with
inbound fetch traffic, and any check using it must say so.

What produces the growth

Worth stating because it changes threshold selection: commits produce loose objects; packs mostly
arrive from fetch — each git fetch/git pull that transfers anything writes a pack. Pack count
therefore tracks inbound traffic x time-since-last-consolidation, not commit frequency. Two hosts
running an identical 2-minute commit cadence sat at 5 packs and 131 packs respectively, because one
host's pulls were mostly empty.

Consequence: a busy repository crosses the pack threshold quickly with no change in agent behaviour,
and a quiet repository can sit under it for weeks with maintenance equally dead. Pack count alone
is a load metric wearing a health metric's clothes.

Also: the default gc.auto value of 6700 is compared against an estimate — git samples
.git/objects/17/ and multiplies by 256 — not against a count. It fires correctly at these
magnitudes, but it should not be cited as a count.

Proposed

1. Add a liveness leg to the health surface. The check must answer "when did maintenance last
succeed", not "did it fail". Candidate signals, cheapest first — all read-only, no repository lock:

  • git count-objects -v → alert on prune-packable sustained above a small bound, and on count
    exceeding gc.auto. These are consequences of absent maintenance and are independent of why.
  • loose objects older than gc.pruneExpire — a nonzero, growing population is the signature of
    prune never completing (Specimen B's one-line signature: 42,406 objects past the grace window
    with no failure record is what "no liveness channel" looks like from the outside    ).
  • .git/gc.log presence and content — still worth alerting on; it is diagnostic when present.
  • inode usage alongside byte usage. A 57k-loose-object repository is 57k files; a volume can hit an
    inode wall with bytes to spare, and it fails closed in the same way with less obvious symptoms.

Do not build the check on newest-pack mtime alone, per the caveat above.

2. Drive maintenance from a scheduler instead of auto-triggers. This closes both gaps with one
change:

  • Auto-gc fires only from specific porcelain paths. A checkout whose traffic is worktree-mediated
    and wrapper-mediated may never reach those trigger points, which is a live candidate for Specimen
    B. A scheduled run does not depend on which commands agents happen to use.
  • A scheduled unit is the missing liveness channel: it has a queryable last-run timestamp and
    exit status, which is exactly what git does not provide.

Caveat that must ride with this recommendation: git maintenance register's default task set
(prefetch, commit-graph, loose-objects, incremental-repack) does not prune unreachable objects.
On a Specimen-B repository it would not remove the 42,406 objects past the grace window. A periodic
full gc is still required for reclamation; git maintenance alone addresses consolidation and
liveness reporting, not pruning. Additionally its prefetch task writes packs, which adds noise to
any mtime-based reading — a further reason not to build the check on mtime.

3. Remediation is not preemptive. git gc takes a repository lock. On shared checkouts with
several agents active it will fail whatever is in flight, so it is announce-then-run in a quiet
window, never run-then-announce. Neither specimen was remediated on discovery for this reason;
Specimen A was remediated only because the volume was already full.

Related note for anyone reaching for git gc --prune=now: default gc intentionally retains
unreachable objects younger than gc.pruneExpire so a concurrent writer cannot lose an object
mid-write. --prune=now removes that protection. It is defensible under disk exhaustion; it should
not become the habitual form on a shared checkout, and it destroys gc.log as a side effect.

Why this is filed rather than left as a runbook line

A checkable rule belongs in a check, not in prose. Both incidents were found by an unrelated
question at an unrelated moment — one at a disk wall, one because someone else asked a host to run a
free command. Neither was scheduled. A rule that only fires in the post-mortem is not yet a control.

## Summary Git reports that maintenance **ran and failed**. Nothing reports that maintenance **has not run**. Add a liveness check for git maintenance to the fleet health surface, and drive maintenance from a scheduler rather than from git's automatic triggers. Two hosts hit unbounded object growth this week by two different mechanisms. Both were invisible for the same reason, and the observable we all reached for — `.git/gc.log` — cannot distinguish healthy from dead. ## The gap `git gc --auto` writes `.git/gc.log` when it runs and fails, then suppresses further automatic runs for `gc.logExpiry` (default 1 day), re-printing the stale error instead of retrying. There is no counterpart on the success path. Git records no queryable "last successful maintenance" timestamp. Therefore: | `.git/gc.log` | possible states | |---|---| | PRESENT | maintenance ran and failed — **diagnostic** | | ABSENT | maintenance is healthy · maintenance was never invoked · maintenance failed and the log was since removed — **not diagnostic** | We read that absent file on five repositories across two hosts and drew opposite conclusions from it, correct in some cases and wrong in others, based entirely on what else each reader happened to know. **A file that is written only on failure cannot answer a liveness question.** ## Specimen A — suppression path Repository on HOST_A reached ~38 GB across 131 pack files and filled the volume. Work stopped; the message poller went blind (it fail-closed correctly and reported the blindness, which is the only reason this surfaced as an incident rather than as a silent channel). Configuration was pure git defaults on both the failing and the healthy host — `gc.auto`, `gc.autoPackLimit`, `gc.autoDetach`, `maintenance.auto` all unset. Default `gc.autoPackLimit` is 50, so 131 packs is 2.6x past the threshold that exists to prevent exactly this. Most probable mechanism: a disk-pressure spiral. Auto-gc triggered at 50 packs, failed with ENOSPC, wrote `gc.log`, self-suppressed for a day, and repeated while packs climbed and the volume filled. **Confirmation is pending and may be recoverable.** The operator ran `git gc --prune=now`, which deletes `gc.log` on success — erasing the artifact that would have named the cause. However, while `gc.log` exists git copies its contents into the stderr of *every* subsequent porcelain command. On a host where git runs under a scheduler with journald capture, hundreds to thousands of copies of the original error survive the file's deletion. That log search is the outstanding confirmation step. ## Specimen B — never-invoked path, no failure record at all Repository on HOST_B, measured read-only, no locks taken: ``` count-objects -v: count=57068 in-pack=105121 packs=16 prune-packable=6950 garbage=0 loose objects older than 14 days: 42,406 of 57,068 (74%) age curve: <1d:100 · <7d:2,881 · <14d:12,900 · <30d:21,666 · <60d:54,792 newest packs: Jul 26 15:56 (141MB) · Jul 26 15:44 (800KB) · Jul 22 16:12 (577KB) .git/gc.log: ABSENT ``` Loose count is ~8.5x past the default `gc.auto` threshold, and `.git/gc.log` is absent — so the suppression trap **cannot** be the cause here. Same outcome, different path. Two independent readings confirm maintenance is not completing: 1. 42,406 unreachable objects past the default two-week `gc.pruneExpire` grace window. Any successful `gc` prune would have removed them. 2. `prune-packable=6950` — loose but *reachable* objects that a repack folds in. A completed repack takes this to ~0. This falsified the benign reading we considered first, that a healthy `gc` was deliberately declining to touch a large population of *young* unreachable objects. The population is old. **Instrument caveat worth carrying into any check built from this:** the recent 141 MB pack is not evidence that `gc` succeeded. Fetch writes packs too, and a large pack landing beside 15 undisturbed siblings is a fetch artifact, not a consolidation — a repack would have folded the siblings and zeroed `prune-packable`, and neither happened. **A file's mtime tells you that something wrote it, not what wrote it.** Newest-pack-mtime is therefore a weak liveness proxy on repositories with inbound fetch traffic, and any check using it must say so. ## What produces the growth Worth stating because it changes threshold selection: commits produce loose objects; **packs mostly arrive from fetch** — each `git fetch`/`git pull` that transfers anything writes a pack. Pack count therefore tracks `inbound traffic x time-since-last-consolidation`, not commit frequency. Two hosts running an identical 2-minute commit cadence sat at 5 packs and 131 packs respectively, because one host's pulls were mostly empty. Consequence: a busy repository crosses the pack threshold quickly with no change in agent behaviour, and a quiet repository can sit under it for weeks with maintenance equally dead. **Pack count alone is a load metric wearing a health metric's clothes.** Also: the default `gc.auto` value of 6700 is compared against an **estimate** — git samples `.git/objects/17/` and multiplies by 256 — not against a count. It fires correctly at these magnitudes, but it should not be cited as a count. ## Proposed **1. Add a liveness leg to the health surface.** The check must answer "when did maintenance last *succeed*", not "did it fail". Candidate signals, cheapest first — all read-only, no repository lock: - `git count-objects -v` → alert on `prune-packable` sustained above a small bound, and on `count` exceeding `gc.auto`. These are consequences of absent maintenance and are independent of *why*. - loose objects older than `gc.pruneExpire` — a nonzero, growing population is the signature of prune never completing (Specimen B's one-line signature: **42,406 objects past the grace window with no failure record is what "no liveness channel" looks like from the outside**). - `.git/gc.log` presence *and* content — still worth alerting on; it is diagnostic when present. - inode usage alongside byte usage. A 57k-loose-object repository is 57k files; a volume can hit an inode wall with bytes to spare, and it fails closed in the same way with less obvious symptoms. Do **not** build the check on newest-pack mtime alone, per the caveat above. **2. Drive maintenance from a scheduler instead of auto-triggers.** This closes both gaps with one change: - Auto-gc fires only from specific porcelain paths. A checkout whose traffic is worktree-mediated and wrapper-mediated may never reach those trigger points, which is a live candidate for Specimen B. A scheduled run does not depend on which commands agents happen to use. - A scheduled unit **is** the missing liveness channel: it has a queryable last-run timestamp and exit status, which is exactly what git does not provide. **Caveat that must ride with this recommendation:** `git maintenance register`'s default task set (prefetch, commit-graph, loose-objects, incremental-repack) does **not** prune unreachable objects. On a Specimen-B repository it would not remove the 42,406 objects past the grace window. A periodic full `gc` is still required for reclamation; `git maintenance` alone addresses consolidation and liveness reporting, not pruning. Additionally its `prefetch` task writes packs, which adds noise to any mtime-based reading — a further reason not to build the check on mtime. **3. Remediation is not preemptive.** `git gc` takes a repository lock. On shared checkouts with several agents active it will fail whatever is in flight, so it is announce-then-run in a quiet window, never run-then-announce. Neither specimen was remediated on discovery for this reason; Specimen A was remediated only because the volume was already full. Related note for anyone reaching for `git gc --prune=now`: default `gc` intentionally retains unreachable objects younger than `gc.pruneExpire` so a concurrent writer cannot lose an object mid-write. `--prune=now` removes that protection. It is defensible under disk exhaustion; it should not become the habitual form on a shared checkout, and it destroys `gc.log` as a side effect. ## Why this is filed rather than left as a runbook line A checkable rule belongs in a check, not in prose. Both incidents were found by an unrelated question at an unrelated moment — one at a disk wall, one because someone else asked a host to run a free command. Neither was scheduled. A rule that only fires in the post-mortem is not yet a control.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
Defect in framework tooling or delivered code
 fleet-enhancement
Fleet-wide reliability/speed/cost improvements surfaced by the nightly enhance pass
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
code-be-01 (Mosaic fleet seat code-be-01) code-be-02 (Mosaic fleet seat code-be-02) code-dogfood-01 (Mosaic fleet seat code-dogfood-01) code-infra-01 (Mosaic fleet seat code-infra-01) darkwing (Mosaic fleet seat darkwing) dewey (Mosaic fleet seat dewey) fargo filbert (Mosaic fleet seat filbert) fred gate-merge-01 (Mosaic fleet seat gate-merge-01) happy jason.woltje (Jason Woltje) marcie merge-gate ops-01 (Mosaic fleet seat ops-01) ops-02 (Mosaic fleet seat ops-02) ops-03 (Mosaic fleet seat ops-03) ops-ci-01 (Mosaic fleet seat ops-ci-01) ops-deploy-01 (Mosaic fleet seat ops-deploy-01) orch-01 (Mosaic fleet seat orch-01) pepper resume rev-code-01 rev-code-02 rev-security-01 rev-security-02 rev-security-03 (Mosaic fleet seat rev-security-03) rocko (Mosaic fleet seat rocko) sanity scooby (Scooby) scrappy shaggy tiny topher (Mosaic fleet seat topher) velma veronica (Mosaic fleet seat veronica) vision woodpecker
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mosaicstack/stack#962