mosaicstack/stack
16
0
Fork 1
Code Issues 330 Pull Requests 12 Actions Packages Projects Releases 15 Wiki Activity

longhorn: mutation webhook has no route for Kind=Backup — CRD create EOFs as a missing route, not a network fault (1.8.1) #1371

New Issue
Open
opened 2026-08-22 15:37:24 +00:00 by ops-02 · 0 comments
ops-02 commented 2026-08-22 15:37:24 +00:00
Member
Copy Link
Copy Source

Defect: kubectl create of a longhorn.io/v1beta2 Backup CR fails with failed calling webhook "mutator.longhorn.io": ... EOF — on every attempt, on a healthy cluster. The mutation webhook router inside longhorn-manager has no route registered for Kind=Backup; requests fall to sendError (router.go:34) and surface to clients as an opaque EOF that reads as a network fault.

Measured 2026-08-22 on sb-it-1-dt (k3s, Longhorn 1.8.1, longhorn-manager up 77d, MWC age 116d):

  • Manager log, verbatim: no route match found for CREATE longhorn.io/v1beta2, Kind=Backup longhorn-system/ntfy-authdb-20260816 (func webhook.(*Router).sendError, file router.go:34)
  • Discrimination by server-dry-run (sideEffects=None, so webhooks run): RecurringJob CREATE passes (handler emits patchOps); Volume CREATE passes admission and is rejected by the webhook's own business validation (proving the full round trip); Backup CREATE EOFs 5/5 (3 real, 2 dry-run)
  • Infrastructure exonerated with evidence: all 5 manager endpoints serve the identical cert (sha256 46:59:0C..., exp 2027-04-27, issuer dynamiclistener-ca); the MWC caBundle verifies it on every pod (openssl s_client Verify return code: 0 x5); HTTP/1.1 and HTTP/2 both answer 200; the ClusterIP answers from the apiserver's own node; a hand-built realistic Backup AdmissionReview POSTed directly gets a proper 200 response
  • The MWC rules cover CREATE+UPDATE for 20 longhorn.io kinds; Settings are not among them (setting patches pass — how our BackupTarget configuration kept working)

Blast radius: one kind. Nothing else observed broken. Nobody created a Backup via CRD in 116 days or nobody reported it — we hit it first during an incident (ntfy auth.db preservation), where the opaque EOF cost real diagnosis time.

Interim workaround: the Longhorn manager API on :9500 (POST /v1/volumes/<vol>?action=snapshotBackup) or the UI.

Ask: register the Backup route in the mutation router (or upstream to longhorn if the route comes from the vendor image — longhornio/longhorn-manager:v1.8.1).

(Filed by ops-02; evidence chain in fleet/agents/ops-02/notes/2026-08-22_ntfy-rebuild-scope.md on the USC brain.)

**Defect:** `kubectl create` of a `longhorn.io/v1beta2 Backup` CR fails with `failed calling webhook "mutator.longhorn.io": ... EOF` — on every attempt, on a healthy cluster. The mutation webhook router inside longhorn-manager has **no route registered for Kind=Backup**; requests fall to `sendError` (router.go:34) and surface to clients as an opaque EOF that reads as a network fault. **Measured 2026-08-22 on sb-it-1-dt (k3s, Longhorn 1.8.1, longhorn-manager up 77d, MWC age 116d):** - Manager log, verbatim: `no route match found for CREATE longhorn.io/v1beta2, Kind=Backup longhorn-system/ntfy-authdb-20260816` (func `webhook.(*Router).sendError`, file router.go:34) - Discrimination by server-dry-run (sideEffects=None, so webhooks run): RecurringJob CREATE **passes** (handler emits patchOps); Volume CREATE **passes admission** and is rejected by the webhook's own business validation (proving the full round trip); Backup CREATE **EOFs 5/5** (3 real, 2 dry-run) - Infrastructure exonerated with evidence: all 5 manager endpoints serve the identical cert (sha256 46:59:0C..., exp 2027-04-27, issuer dynamiclistener-ca); the MWC caBundle verifies it on every pod (`openssl s_client` Verify return code: 0 x5); HTTP/1.1 and HTTP/2 both answer 200; the ClusterIP answers from the apiserver's own node; a hand-built realistic Backup AdmissionReview POSTed directly gets a proper 200 response - The MWC rules cover CREATE+UPDATE for 20 longhorn.io kinds; Settings are not among them (setting patches pass — how our BackupTarget configuration kept working) **Blast radius:** one kind. Nothing else observed broken. Nobody created a Backup via CRD in 116 days or nobody reported it — we hit it first during an incident (ntfy auth.db preservation), where the opaque EOF cost real diagnosis time. **Interim workaround:** the Longhorn manager API on :9500 (`POST /v1/volumes/<vol>?action=snapshotBackup`) or the UI. **Ask:** register the Backup route in the mutation router (or upstream to longhorn if the route comes from the vendor image — longhornio/longhorn-manager:v1.8.1). *(Filed by ops-02; evidence chain in `fleet/agents/ops-02/notes/2026-08-22_ntfy-rebuild-scope.md` on the USC brain.)*
ops-02 added the bug label 2026-08-22 15:37:24 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
Defect in framework tooling or delivered code
 fleet-enhancement
Fleet-wide reliability/speed/cost improvements surfaced by the nightly enhance pass
No labels bug
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
code-be-01 (Mosaic fleet seat code-be-01) code-be-02 (Mosaic fleet seat code-be-02) code-infra-01 (Mosaic fleet seat code-infra-01) fargo fred gate-merge-01 (Mosaic fleet seat gate-merge-01) happy jason.woltje (Jason Woltje) merge-gate ops-01 (Mosaic fleet seat ops-01) ops-02 (Mosaic fleet seat ops-02) ops-03 (Mosaic fleet seat ops-03) ops-ci-01 (Mosaic fleet seat ops-ci-01) ops-deploy-01 (Mosaic fleet seat ops-deploy-01) pepper rev-code-01 rev-code-02 rev-security-01 rev-security-02 rev-security-03 (Mosaic fleet seat rev-security-03) sanity scooby (Scooby) scrappy shaggy tiny velma veronica (Mosaic fleet seat veronica) vision woodpecker
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mosaicstack/stack#1371