stack/briefs/DECISION-BRIEF-KANBAN-SOT-D3.md

Decision Brief — Native Kanban SOT, Decision #3

Decision: Do not permit a writable file fallback. Adopt Option A: PostgreSQL as the sole writable source of truth (SOT), with fail-closed mutations.

Context and decision rule

The approved design already makes PostgreSQL the canonical writable store, generates a read-only TASKS.md view, uses a mechanical coordinator, and reserves the Certifier as a final gate without merge authority. The remaining question is whether a PostgreSQL outage should permit writes to a local file for later reconciliation.

This decision is not “database availability versus file availability.” It is whether the system preserves one authoritative ordering, identity, and audit trail during failure. Given Kubernetes deployment, Longhorn DiskPressure/replica-loss history, and GitOps recovery paths, the safer design is to make a database outage visible and operationally explicit, then recover the one authority. It is not to create an emergency second authority whose reconciliation semantics must be correct under the worst conditions.

Options assessed

Dimension	A — PostgreSQL only; mutations fail closed	B — writable local file fallback; reconcile later
1. Data integrity	5/5. Every accepted mutation is validated, ordered, transactionally committed, and constrained in one place. During DB loss, the system accepts no new state it cannot durably prove. PITR restores a known consistent point; subsequent replay is explicit rather than implicit.	2/5. A local file may be atomically written on one host, but it cannot preserve global transaction ordering, database constraints, cross-card invariants, or durable identity allocation without duplicating database behavior. A host crash, partial write, clock skew, or stale local copy can leave an apparently valid but semantically invalid queue of changes.
2. Split-brain / dual-writer risk	5/5. There is one writer and one failure mode: unavailable means refuse writes. Read-only exports are deliberately non-authoritative and cannot race the database.	1/5. The fallback is a second writer precisely while reachability is uncertain. “DB down” can be a network partition, a single pod failure, or a stale health signal while PostgreSQL is still writable elsewhere. Reconciliation then needs conflict policy for edits, transitions, assignments, approvals, idempotency, ordering, and deletes; choosing “file wins” or “DB wins” loses valid work in some cases.
3. Outage operability	3/5. Mutations stop, which is painful but honest. Operators can continue with read-only exports, incident handling, and a documented restoration clock; automation does not silently create divergent work. The coordinator should expose a clear degraded status and reject writes deterministically.	4/5 for immediate intake, 1/5 for total operational burden. Operators can keep entering work locally, but each outage becomes a reconciliation incident. Staff must know which host owns the file, whether its writes were imported, and whether the DB was actually unavailable. The apparent availability shifts complexity to a higher-risk, later moment when context is worse.
4. Disaster recovery	5/5. Backup plus WAL-based PITR restores the same authoritative data model to a selected point. Recovery is testable: restore PostgreSQL, validate, then re-enable one writer. Longhorn incidents are mitigated by backups stored outside the Longhorn failure domain.	2/5. A file fallback does not replace database recovery: the recovered DB still needs authoritative restoration, then uncertain import. If the fallback file shares the failed node/volume, it is not an independent recovery mechanism. If it is replicated, it becomes another distributed datastore that needs backup, encryption, retention, and restore testing.
5. Auditability	5/5. Database events can carry actor, correlation ID, timestamp, prior/new state, idempotency key, and approval reference in a transaction. Refused writes are also observable as outage evidence. The generated file is a reproducible view, not an editable audit source.	2/5. Git/file history can record text changes, but it cannot reliably bind a mutation to the same authenticated principal, authorization decision, transaction boundary, or approval consumption as PostgreSQL. Later import timestamps and commit order are not necessarily the original event order. Manual edits are difficult to distinguish from intended fallback entries.
6. Migration and rollback	4/5. Migration has one cutover: seed/validate PostgreSQL, generate the read-only file, and disable legacy writes. Rollback restores a database backup/PITR point and regenerates exports. A brief write freeze is understandable and testable.	1/5. Every migration and rollback must also define whether fallback files are enabled, which schema/version they target, how they are replayed, and how already-imported records are detected. A rollback after a fallback import can reintroduce records or lose conflict resolutions.

Tradeoff conclusion

Option B buys local write acceptance during an outage, but it does so by abandoning the property the SOT was selected to provide: one authoritative transaction history. A “fallback” that requires distributed ordering, conflict resolution, identity semantics, authorization replay, and exactly-once import is a second datastore, not a safety valve. It is less safe than a deliberately unavailable mutation path backed by independently recoverable PostgreSQL.

Recommendation

Choose Option A: PostgreSQL is the sole writable SOT. When PostgreSQL is unavailable or its write-health cannot be proven, all Kanban mutations fail closed; they are never redirected to files.

Read-only TASKS.md exports remain useful for situational awareness and incident continuity, but are explicitly marked generated/non-authoritative and are never accepted as an import source. If Jason needs to capture ideas while the system is unavailable, use an out-of-band human note or issue intake outside the Kanban mutation API; that note is a proposal to enter after recovery, not shadow Kanban state.

Minimum safeguards for Option A

Recovery objectives and backup design

• RPO: 15 minutes maximum for committed Kanban state. RTO: 4 hours maximum to restore the writable service after a regional/Longhorn-class storage incident; target 60 minutes for a single-pod or local volume incident.
• Continuous WAL archiving: archive PostgreSQL WAL at least every 5 minutes to encrypted object storage outside the Kubernetes cluster and outside Longhorn. Retain PITR capability for 35 days.
• Base backups: take a verified physical base backup daily; retain daily backups for 35 days, weekly backups for 13 weeks, and monthly backups for 12 months. Keep at least one copy in a separate failure domain/account where feasible.
• Exported evidence: generate the read-only TASKS.md plus a machine-readable signed/checksummed snapshot hourly and on every successful release. Retain exports for 90 days. Exports support visibility and reconciliation of human context; they are never writable recovery input.
• Restore proof: conduct a documented PITR restore test monthly and a full break-glass exercise quarterly, measuring actual RPO/RTO and verifying record counts, event/audit integrity, and generated export consistency.

Break-glass restore procedure

1. Declare write freeze. Put the Kanban mutation endpoint and coordinator in explicit maintenance mode; deny all writes with a stable outage code. Do not enable a file writer.
2. Preserve evidence. Record incident time, database/Longhorn symptoms, last healthy transaction/WAL archive, and the target recovery timestamp. Preserve affected volume and pod evidence before destructive actions when practical.
3. Restore outside the failed path. Provision a clean PostgreSQL instance/volume from a verified base backup and apply archived WAL to the approved target timestamp. Do not restore solely from a Longhorn replica after a replica-loss incident without validation.
4. Validate before reopening. Run automated integrity checks, verify schema version, audit/event continuity, key Kanban invariants, and compare a regenerated read-only export with the restored state. Obtain designated incident-owner approval to reopen writes.
5. Cut over one writer. Update GitOps/Kubernetes configuration to the validated database endpoint, verify a canary read and authorized write, then remove maintenance mode. Generate and publish a fresh read-only export.
6. Close and learn. Reconcile any human outage notes as new, attributable post-recovery entries; never bulk-import a local shadow file. Record achieved RPO/RTO and corrective actions.

Monitoring and alerting

• Alert on PostgreSQL write probe failure, replication/WAL archive failure, backup age exceeding 24 hours, PITR archive lag exceeding 10 minutes, backup verification failure, and restore-test failure.
• Alert on Longhorn DiskPressure, replica degradation/loss, volume robustness below healthy, node filesystem pressure, and sustained database latency/error-rate thresholds.
• Expose a single Kanban health state: healthy, read-only-degraded, or write-unavailable. Mutation clients must distinguish an intentional fail-closed denial from a retryable transport error.
• Alert on export generation/checksum failure and export age exceeding 75 minutes. This is visibility degradation, not permission to write the export.

Residual risks and mitigations

• Risk: an outage blocks legitimate priority work. Mitigation: publish the write-unavailable state, keep an incident contact/runbook, and permit human notes as proposals for attributable post-recovery entry—not as shadow state.
• Risk: backup/PITR is misconfigured or untested. Mitigation: independent off-cluster storage, archive/backup freshness alerts, monthly restore tests, quarterly break-glass drills, and RPO/RTO measurement.
• Risk: Longhorn loss exceeds local recovery assumptions. Mitigation: treat Longhorn as an availability layer, not the only recovery layer; restore from external PostgreSQL backups/WAL to clean storage.
• Risk: stale read-only exports mislead operators. Mitigation: include generated-at timestamp, source commit/checksum, and visible READ ONLY / NOT AUTHORITATIVE labeling; alert on export staleness.
• Risk: manual emergency database changes weaken the audit trail. Mitigation: time-box break-glass access, require incident ID and SQL/audit capture, use peer review after restoration, and regenerate exports immediately after validation.