Compare commits
5
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
78ec47cd97 | ||
|
|
37aae6506c | ||
|
|
241113e6fd | ||
|
|
44ffa99a15 | ||
|
|
b71750122a |
@@ -1,118 +0,0 @@
|
||||
# RM-61 — CI contract exemption for #1000 teardown artifact
|
||||
|
||||
**Tracking:** RM-61 / issue #1000
|
||||
|
||||
**Branch:** `fix/rm-61-ci-contract-exemption`
|
||||
**Owner:** `coder-mos1`
|
||||
|
||||
## Objective
|
||||
|
||||
Determine, by red-first provider controls, whether the `ci-postgres` pod-not-found teardown signature discriminates from a real PostgreSQL failure. Only if it discriminates may a named, bounded CI-contract exemption be implemented. The exemption must retire when #1000 is fixed; fixing #1000 is the closure path.
|
||||
|
||||
## Pre-registered kill criterion
|
||||
|
||||
If an injected real `ci-postgres` failure also yields `pods "wp-svc-<ULID>-ci-postgres" not found` as the service's provider-visible failure, the signature does not discriminate. Option B is unsafe; stop exemption implementation and fall to Option A (#1000).
|
||||
|
||||
## Plan
|
||||
|
||||
1. Capture full `-f json` records for the 11 supplied observations and state counts.
|
||||
2. Run one startup-failure control using the real pgvector/PostgreSQL image with an invalid `initdb` argument.
|
||||
3. Run one post-readiness crash control using real PostgreSQL, `pg_isready`, and a deliberate postmaster kill while a DB-dependent probe is active.
|
||||
4. Compare the raw `ci-postgres` service record independently of failures in dependent steps.
|
||||
5. Investigate runner/time/head clustering only as a hypothesis; never encode incidental correlates or retries into policy.
|
||||
6. If and only if the controls discriminate, implement and test the exact exemption, document its two-way boundary, and track retirement at #1000.
|
||||
|
||||
## Budget
|
||||
|
||||
No explicit token cap supplied. Working estimate: 20K–30K tokens. Limit provider controls to the two pre-registered runs; no retries or re-roll policy.
|
||||
|
||||
## Initial evidence
|
||||
|
||||
Historical JSON saved locally under `.evidence/rm-61/` (not for commit). Supplied pipelines: 11 total. Child-step counts: five pipelines with 9 children and six with 10 children. Seven contain the `ci-postgres` pod-not-found failure (#2170, #2175, #2180, #2181, #2182, #2187, #2188); four do not (#2158, #2167, #2184, #2186). Every observed workflow reports `agent_id=44`, so the available JSON does not separate clean and artifact runs by runner. This refutes runner identity as a discriminator in the sampled record.
|
||||
|
||||
## Progress
|
||||
|
||||
- [x] Requirements and kill criterion recorded before control implementation.
|
||||
- [x] Historical full-JSON records captured.
|
||||
- [x] Startup-failure control observed terminal.
|
||||
- [x] Post-readiness crash control observed terminal.
|
||||
- [x] Discrimination verdict recorded: Option B may proceed.
|
||||
- [x] Conditional exemption implementation.
|
||||
|
||||
## Tests / evidence
|
||||
|
||||
### Control 1 — real startup failure
|
||||
|
||||
- Commit: `3931b0e29eb834914f7b17e4db7e221481d436fa`
|
||||
- Pipeline: #2189, exact commit match.
|
||||
- Full JSON child scan: 9 total — 7 success, 2 failure, 0 skipped/pending/running.
|
||||
- `ci-postgres`: `state=failure`, `exit_code=1`, `error=null`, with a five-second execution window.
|
||||
- `test`: `state=failure`, `exit_code=1` after the readiness budget expired.
|
||||
- Pipeline/workflow: terminal `failure`.
|
||||
|
||||
This control is red and its service record differs from #1000 (`exit_code=0` plus pod-not-found). It proves the startup-failure direction only. It does not settle the dangerous post-readiness crash/garbage-collection path.
|
||||
|
||||
### Control 2 — real post-readiness crash
|
||||
|
||||
- Commit: `25ac59715a94dd1b52ef42577472eb44ecc4b446`
|
||||
- Pipeline: #2191, exact commit match.
|
||||
- Full JSON child scan: 9 total — 7 success, 2 failure, 0 skipped/pending/running.
|
||||
- Service log proves PostgreSQL reached `database system is ready to accept connections`, the test created the arm table, and the service then killed postmaster PID 7.
|
||||
- Test log proves a successful `SELECT 1` followed by `Connection refused`; it exited the pre-registered control code 61.
|
||||
- `ci-postgres`: `state=failure`, `exit_code=137`, `error=null`, with a 203-second execution window.
|
||||
- `test`: `state=failure`, `exit_code=61`.
|
||||
- Pipeline/workflow: terminal `failure`.
|
||||
|
||||
This is the dangerous post-readiness crash path. Its service record is not pod-not-found and therefore differs from #1000 independently of the dependent test failure.
|
||||
|
||||
### Discrimination verdict
|
||||
|
||||
Both real failures are provider-visible as process exits (`exit_code=1` startup; `exit_code=137` crash) with no pod-not-found error. The seven observed #1000 artifacts are provider reconciliation misses (`exit_code=0` plus the exact pod-not-found error). The declared kill criterion did not fire, so Option B may proceed with a matcher requiring the full conjunction. This evidence does **not** prove every future Kubernetes failure is distinguishable; it proves these two concrete real-failure classes remain blocking and bounds the exemption to the observed reconciliation shape.
|
||||
|
||||
### Unit red-first checkpoint
|
||||
|
||||
The nine-case contract harness was written before the verifier. First execution exited 1 because `verify-terminal-green.py` did not exist; no exemption implementation was live. Cases pre-register ordinary green, the exact artifact, both provider controls, near-miss signatures, an independent failure, and a skipped step.
|
||||
|
||||
### Control 2 setup attempt — invalid, excluded from evidence
|
||||
|
||||
- Commit: `9455cd6a2650b2b7e70f746c07933d96e5cb3d20`
|
||||
- Pipeline: #2190, exact commit match.
|
||||
- Full JSON child scan: 9 total — 7 success, 2 failure, 0 skipped/pending/running.
|
||||
- Service log: `/bin/sh: 0: -c requires an argument`.
|
||||
- Root cause: Woodpecker service `commands` did not become the third `sh -c` argument. PostgreSQL never started, so this run is **not** the post-readiness crash control and provides no discrimination evidence.
|
||||
- Focused remediation: place the script directly in the third `entrypoint` element and supply `PGPASSWORD` for the marker query. This is a control-fixture correction, not a retry of #1000 and not evidence for either verdict.
|
||||
|
||||
## Implementation evidence
|
||||
|
||||
- `verify-terminal-green.py` consumes only the full JSON/API record; it performs no fetch, retry, or trigger.
|
||||
- Exact #2188 record: exit 0, 10 children, 9 success + 1 named exemption.
|
||||
- Historical set: #2158/#2167/#2184/#2186 pass with no exemption; #2170/#2175/#2182/#2187/#2188 pass with one named exemption; #2180/#2181 remain red because independent failures exist.
|
||||
- Provider controls: #2189 and #2191 both exit 1 under the verifier; neither is exempted.
|
||||
- Unit harness: initial 9/9 cases passed after the red-first checkpoint; review remediation expands this to 12 cases with expected-head match/missing/mismatch coverage.
|
||||
- Test-membership guard: PASS, population 45; 26 enumerated, 19 signed exclusions; all 39 surface paths present.
|
||||
- Python compile: PASS.
|
||||
- `pnpm typecheck`: PASS, 45/45 tasks.
|
||||
- `pnpm lint`: PASS, 25/25 tasks.
|
||||
- `pnpm format:check`: PASS after moving local evidence outside the repository tree.
|
||||
- `test:framework-shell`: RM-61 and all preceding suites passed, then the pre-existing wake assertion aborted with exit 97 because this host's Bash 5.2.15 reports `BASH_LINENO [3 5]` where that suite requires `[3 4]`. RM-61 does not modify the wake suite; the command is not fully runnable on this host as written and no substitute result is claimed.
|
||||
|
||||
## Independent review
|
||||
|
||||
- Review 67 / comment 20403 at exact head `e7b29219e11efd0a19395156ac0b154bec0c3a73`: **REQUEST CHANGES**.
|
||||
- Blocker: the verifier echoed the pipeline commit but did not bind it to the current PR head; mutating only #2188's commit still returned terminal-green.
|
||||
- Remediation: require `--expect-commit <full-40>`, add a pipeline anomaly on missing/mismatched record commits, emit expected and observed values, wire both CI documentation and the merge-gate baseline to pass provider PR head, and add match/missing/mismatch tests.
|
||||
- This binding is not prohibited head-based clustering policy: it proves the evidence belongs to the commit under verdict. Runner/node/time/head correlation remains excluded from the teardown signature itself.
|
||||
- Review 69 later approved the commit-binding remediation at exact head `033b2ffb46674b2c0bcc5197273c109b461f62d9`; pipeline #2193 was 9/9 success. Before merge-gate, an independent adjudicator found that Python treats JSON `false == 0`, allowing a non-integer exit value to match. The prior gate-ready state was withdrawn. The type-strict set distinguishes genuine red-first controls (`false`, `0.0`, which wrongly exempted) from regression guards (`true`, `"0"`, `null`, which already blocked). Remediation requires the decoded type to be exactly `int` and excludes `bool` explicitly.
|
||||
|
||||
## Documentation checklist
|
||||
|
||||
- [x] CI contract documented in the canonical framework CI/CD guide.
|
||||
- [x] Operator command documented in the Woodpecker tool README.
|
||||
- [x] Merge-gate baseline points to the deterministic verifier and named retirement.
|
||||
- [x] Tracking and retirement cite issue #1000.
|
||||
- [x] Both positive and negative guarantee boundaries are stated.
|
||||
- [x] No API/auth/schema/user-facing navigation change; OpenAPI, user guide, and sitemap are not applicable.
|
||||
|
||||
## Risks
|
||||
|
||||
The controls establish discrimination for deterministic startup failure and an armed post-readiness postmaster crash on the current Woodpecker Kubernetes provider. They cannot prove that every future Kubernetes failure mode will preserve a non-zero exit before reconciliation. The exact matcher minimizes that residual risk, and issue #1000 remains the mandatory provider-seam closure and retirement trigger.
|
||||
@@ -1,71 +0,0 @@
|
||||
# #1019 — Zero-timeout queue-guard harness race
|
||||
|
||||
- **Issue:** #1019 (parent status remains `believed-fixed, pending jarvis validation`; do not close)
|
||||
- **Branch:** `fix/1019-ci-queue-timeout-harness`
|
||||
- **Owner:** `be-coder-08`
|
||||
- **Base:** `origin/main` at `5916aeefd6ed12bcac086c6834c7f6c4ae38e1bc`
|
||||
- **Charter:** `/home/hermes/agent-work/tl-mosaic/CHARTER-1019-HARNESS-FIX.md`
|
||||
|
||||
## Objective
|
||||
|
||||
Make `test-ci-queue-wait-tristate.sh` deterministic without changing any asserted outcome. Remove the indiscriminate zero-timeout race, require every status-classification case to prove the provider was observed, and prove the harness-controlled virtual clock is active.
|
||||
|
||||
## Scope
|
||||
|
||||
- In scope: `packages/mosaic/framework/tools/git/test-ci-queue-wait-tristate.sh` only, plus this evidence scratchpad.
|
||||
- Out of scope: guard parsers, D2/D3 behavior, installer/reseed staleness, PR #1060, and issue closure.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
1. RED deterministically reproduces deadline pre-emption before the provider call.
|
||||
2. Every case that intends status classification positively proves provider observation.
|
||||
3. Pending observes `pending` before deterministic virtual-time expiration.
|
||||
4. The virtual clock has a positive interception control; a broken-clock mutant makes the suite red.
|
||||
5. The exact CI-base image passes the final harness repeatedly with zero failures.
|
||||
6. Baseline gates, independent code/security review, exact-head CI, and coordinator-authorized squash merge pass.
|
||||
|
||||
## Plan
|
||||
|
||||
1. Add deterministic RED instrumentation for the known merge/provider-unreachable pre-emption.
|
||||
2. Replace global `-t 0` with a nonzero timeout interpreted under an event-driven virtual clock; stub sleep without wall waiting.
|
||||
3. Add provider-observation and virtual-clock positive controls without changing outcome assertions.
|
||||
4. Run focused shell checks, repeat in exact CI-base image, baseline gates, and independent reviews.
|
||||
5. Commit with both identity layers, queue-guard plus direct Woodpecker terminal-state verification, push, self-post PR, verify poster/head/CI, obtain coordinator merge authorization, then squash merge without closing #1019.
|
||||
|
||||
## Budget
|
||||
|
||||
- No explicit token cap supplied. Keep scope to one harness file and one scratchpad; stop/report at the charter's 60% context gate.
|
||||
|
||||
## Evidence
|
||||
|
||||
- RED, deterministic pre-provider expiry: `evidence/1019-harness-fix/red-pre-provider-expiry.log` — rc 1; merge/provider-unreachable got rc 124 instead of 75, omitted CANNOT_ASSERT, did not observe the status provider, and wrote no additional audit record (four named failures).
|
||||
- GREEN host focused harness: `evidence/1019-harness-fix/green-host.log` — rc 0, all outcome classes passed.
|
||||
- Load-bearing clock negative control: a temporary same-directory mutant replaced the virtual `date` body with `/bin/date`; `evidence/1019-harness-fix/red-clock-not-intercepted.log` — rc 1 with named `virtual clock interception did not run` failures. The mutant file was removed after the run.
|
||||
- Exact CI-base repeat: `git.mosaicstack.dev/mosaicstack/stack/ci-base:latest`, repository mounted read-only, harness work under container `/tmp`; `evidence/1019-harness-fix/ci-image-repeat/summary.log` — **100 pass / 0 fail / 100 total**.
|
||||
- Synchronization design: provider-status observation creates the event marker; virtual time is 1000 before the event and 1002 afterward. Pending alone reaches the stubbed no-op sleep and a post-observation deadline check. `-t 1` is uniquely load-bearing because removing it restores the 900-second default deadline at virtual time 1900, which 1002 does not cross. The numeric timeout is subject semantics under virtual time, not a wall-clock synchronization duration.
|
||||
|
||||
## Review remediation — semantic timeout vs. liveness bound
|
||||
|
||||
Security review found that virtual time remained at 1000 forever before provider observation and stubbed sleep never waited. A regression looping before the status endpoint—or blocking in the first provider call—therefore could prevent `run_guard` from returning, so the post-return provider assertion could never fire.
|
||||
|
||||
**General rule:** A timeout usually serves two purposes: semantics and liveness. Removing wall time from semantic synchronization can silently remove the only independent hang bound. Preserve deterministic virtual time for subject semantics, but provide a separately implemented real-clock liveness watchdog and prove that watchdog fires.
|
||||
|
||||
Remediation:
|
||||
|
||||
- Every guard subject invocation is launched by absolute `/usr/bin/python3` in a new session. Python's internal monotonic `wait(timeout=...)` provides real-clock liveness independently of PATH; expiry kills the entire isolated process group, so neither PATH-front shims nor a blocked provider descendant can retain the capture pipe.
|
||||
- Watchdog expiry returns distinct harness rc 90 plus `FAIL HANG watchdog`, separate from subject timeout rc 124.
|
||||
- A first attempt using absolute `/usr/bin/timeout -s KILL` passed on GNU coreutils but failed in the exact Alpine CI-base image: BusyBox killed the immediate wrapper while the guard/provider descendants survived and retained the command-substitution pipe. The process-group kill is therefore required behavior, not portability polish.
|
||||
- A committed positive control hangs the branch-provider stub before the status endpoint. It must terminate through the watchdog, emit the hang-specific diagnostic, return rc 90, and prove the status provider was never reached.
|
||||
- RED before remediation: a temporary ordinary-success mutant hung before provider observation; only an external control could kill the suite (rc 137), and there was no internal hang-specific diagnostic (`red-watchdog-absent.log`).
|
||||
- The watchdog mutant/control is load-bearing: removing the internal watchdog leaves the control unable to produce its required rc 90 and diagnostic.
|
||||
|
||||
Post-review evidence:
|
||||
|
||||
- Host focused harness with process-group watchdog: rc 0 (`green-watchdog-process-group-host.log`).
|
||||
- Exact Alpine CI-base focused harness with process-group watchdog: rc 0 (`green-watchdog-ci-image.log`).
|
||||
- Hanging ordinary-success mutant: suite rc 1; success returned rc 90, emitted `FAIL HANG watchdog`, and loudly reported that provider/clock observation did not occur (`red-watchdog-fires.log`).
|
||||
- Removed-`-t 1` mutant: suite rc 1; pending was terminated by the watchdog instead of producing `ASSERTED_NOT_READY`, proving the explicit timeout is load-bearing (`red-timeout-argument-removed.log`).
|
||||
|
||||
## 60% context hold
|
||||
|
||||
Stopped before baseline/review/commit as required by the charter. Remaining: inspect final diff, shell/static/baseline gates, independent code/security review, remediation if any, identity-bound commit/trailer verification, mandatory queue guard plus direct terminal Woodpecker `mosaic` enumeration, push, self-posted PR/provider poster read-back, exact-head terminal-green CI, coordinator merge authorization, squash merge, main CI verification, and leave #1019 unclosed as `believed-fixed, pending jarvis validation`.
|
||||
@@ -13,14 +13,7 @@ It is a **gate** role: the one and only merge path.
|
||||
2. **Use the wrapped scripts as the ONLY merge path** — the merge-gate merges
|
||||
**exclusively** by calling **`pr-merge.sh`** (the merge action, which carries the
|
||||
authoritative forbidden-path guard) and **`pr-ci-wait.sh`** (to wait for green
|
||||
CI before merging). Before issuing a verdict, scan the full JSON/API child-step
|
||||
record (including `clone`) with **`verify-terminal-green.py --expect-commit
|
||||
<current-provider-PR-head>`** and record the equal expected/observed full-40
|
||||
commits, exact step count, anomalies, and named exemptions. Missing or mismatched
|
||||
commit binding is a hard refusal. The verifier's sole interim
|
||||
exemption is `WP-K8S-1000-CI-POSTGRES-TEARDOWN`; it is signature-scoped, tracked
|
||||
by #1000, and retires when #1000 is fixed. These scripts are the _only_
|
||||
sanctioned merge path.
|
||||
CI before merging). These two scripts are the _only_ sanctioned merge path.
|
||||
3. **Never call the raw API** — the merge-gate **does NOT** call `tea`, the raw
|
||||
Gitea/forge HTTP API, or any other merge mechanism directly. Only `pr-merge.sh`
|
||||
and `pr-ci-wait.sh`.
|
||||
|
||||
@@ -868,38 +868,6 @@ steps:
|
||||
7. **Test on a short-lived non-main branch first** — open a PR and verify quality gates before merging to `main`
|
||||
8. **Verify images appear** in Gitea Packages tab after successful pipeline
|
||||
|
||||
## Terminal-Green Full-Step Contract
|
||||
|
||||
A successful pipeline summary is not sufficient: verification MUST consume the full JSON/API child-step record, including `clone`.
|
||||
|
||||
```bash
|
||||
PR_HEAD=<full-40-hex-provider-head>
|
||||
~/.config/mosaic/tools/woodpecker/pipeline-status.sh \
|
||||
-r mosaicstack/stack -n <pipeline-number> -f json \
|
||||
| ~/.config/mosaic/tools/woodpecker/verify-terminal-green.py \
|
||||
--expect-commit "$PR_HEAD" -
|
||||
```
|
||||
|
||||
`PR_HEAD` MUST come from the current provider PR metadata and MUST be the full 40-hex head, not a local branch guess. The verifier fails if the argument is missing, malformed, absent from the pipeline record, or differs from that record.
|
||||
|
||||
The verifier reports the expected and observed commits, total step count, state counts, anomalies, and any applied exemption. Exit `0` means the record satisfies the contract; exit `1` means the commit binding or at least one pipeline, workflow, or child-step state blocks terminal-green; exit `2` means the invocation or JSON input could not be verified.
|
||||
|
||||
### Named interim exemption: `WP-K8S-1000-CI-POSTGRES-TEARDOWN`
|
||||
|
||||
Only this exact conjunction is exempted:
|
||||
|
||||
- pipeline and workflow state are `success`;
|
||||
- exactly one non-success child exists;
|
||||
- its name is `ci-postgres` and type is `service`;
|
||||
- its state is `failure`, exit code is the JSON integer `0` (not boolean, float, string, or null); and
|
||||
- its error exactly matches `pods "wp-svc-<ULID>-ci-postgres" not found`.
|
||||
|
||||
Every near miss remains blocking, including non-zero service exits, startup failures, post-readiness crashes, connection errors, image-pull errors, skipped steps, another failed child, malformed pod names, duplicate matches, or a non-success pipeline/workflow.
|
||||
|
||||
**Boundary in both directions:** this exemption recognizes the observed Woodpecker Kubernetes reconciliation miss after an otherwise-successful run. It does not prove that every future PostgreSQL or Kubernetes failure is distinguishable. It does prove, through provider controls, that a deterministic startup failure (`exit_code=1`) and an armed post-readiness postmaster crash (`exit_code=137`, dependent probe `Connection refused`) do not match and remain red.
|
||||
|
||||
**Tracking and retirement:** [mosaicstack/stack#1000](https://git.mosaicstack.dev/mosaicstack/stack/issues/1000) owns the provider-seam fix. This exemption MUST be removed when #1000 is fixed. It is not authority to retry or re-trigger a pipeline, and no per-PR re-roll is part of the contract.
|
||||
|
||||
## Post-Merge CI Monitoring (Hard Rule)
|
||||
|
||||
For source-code delivery, completion is not allowed at "PR opened" stage.
|
||||
|
||||
@@ -9,51 +9,10 @@ WORK_DIR="${MOSAIC_TEST_WORK_DIR:-$PWD/.mosaic-test-work/ci-queue-wait-tristate}
|
||||
REPO_DIR="$WORK_DIR/repo"
|
||||
STUB_DIR="$WORK_DIR/stubs"
|
||||
AUDIT_LOG="$WORK_DIR/audit/ci-queue-wait.jsonl"
|
||||
STATUS_OBSERVED="$WORK_DIR/status-observed"
|
||||
CLOCK_LOG="$WORK_DIR/clock.log"
|
||||
WATCHDOG_PYTHON="/usr/bin/python3"
|
||||
WATCHDOG_SCRIPT="$WORK_DIR/real-clock-watchdog.py"
|
||||
WATCHDOG_TIMEOUT_SEC=5
|
||||
WATCHDOG_EXIT=90
|
||||
FEATURE_BRANCH="fix/rm-03-fixture"
|
||||
|
||||
if [[ ! -x "$WATCHDOG_PYTHON" ]]; then
|
||||
echo "FAIL setup: required real-clock watchdog runtime is unavailable at $WATCHDOG_PYTHON" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
rm -rf "$WORK_DIR"
|
||||
mkdir -p "$REPO_DIR" "$STUB_DIR"
|
||||
cat > "$WATCHDOG_SCRIPT" <<'PY'
|
||||
import os
|
||||
import signal
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
if len(sys.argv) < 3:
|
||||
raise SystemExit(2)
|
||||
|
||||
timeout_seconds = float(sys.argv[1])
|
||||
process = subprocess.Popen(sys.argv[2:], start_new_session=True)
|
||||
try:
|
||||
return_code = process.wait(timeout=timeout_seconds)
|
||||
except subprocess.TimeoutExpired:
|
||||
try:
|
||||
os.killpg(process.pid, signal.SIGKILL)
|
||||
except ProcessLookupError:
|
||||
pass
|
||||
process.wait()
|
||||
print(
|
||||
f"FAIL HANG watchdog: subject exceeded {timeout_seconds:g}s "
|
||||
"before completing its intended path",
|
||||
file=sys.stderr,
|
||||
)
|
||||
raise SystemExit(90)
|
||||
|
||||
if return_code < 0:
|
||||
raise SystemExit(128 - return_code)
|
||||
raise SystemExit(return_code)
|
||||
PY
|
||||
git -C "$REPO_DIR" init -q
|
||||
git -C "$REPO_DIR" checkout -q -b "$FEATURE_BRANCH"
|
||||
git -C "$REPO_DIR" remote add origin https://git.example.test/acme/widgets.git
|
||||
@@ -74,9 +33,6 @@ printf '%s\n' "$url" >> "${MOSAIC_STUB_URL_LOG:?}"
|
||||
|
||||
case "$url" in
|
||||
*/branches/*)
|
||||
if [[ "${MOSAIC_STUB_BRANCH_MODE:-ok}" == "hang-before-provider" ]]; then
|
||||
while :; do :; done
|
||||
fi
|
||||
if [[ "${MOSAIC_STUB_BRANCH_MODE:-ok}" == "unreachable" ]]; then
|
||||
exit 7
|
||||
fi
|
||||
@@ -88,7 +44,6 @@ case "$url" in
|
||||
fi
|
||||
;;
|
||||
*/status)
|
||||
: > "${MOSAIC_STUB_STATUS_OBSERVED:?}"
|
||||
case "${MOSAIC_STUB_STATUS_MODE:?}" in
|
||||
success) printf '%s' '{"state":"success","statuses":[{"status":"success"}]}' ;;
|
||||
pending) printf '%s' '{"state":"pending","statuses":[{"status":"pending","context":"ci/test"}]}' ;;
|
||||
@@ -108,31 +63,7 @@ case "$url" in
|
||||
*) echo "unexpected curl URL: $url" >&2; exit 2 ;;
|
||||
esac
|
||||
SH
|
||||
|
||||
cat > "$STUB_DIR/date" <<'SH'
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
if [[ "$#" -ne 1 || "$1" != "+%s" ]]; then
|
||||
echo "unexpected date invocation: $*" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
if [[ -e "${MOSAIC_STUB_STATUS_OBSERVED:?}" ]]; then
|
||||
printf 'date-phase=after-status\n' >> "${MOSAIC_STUB_CLOCK_LOG:?}"
|
||||
printf '1002\n'
|
||||
else
|
||||
printf 'date-phase=before-status\n' >> "${MOSAIC_STUB_CLOCK_LOG:?}"
|
||||
printf '1000\n'
|
||||
fi
|
||||
SH
|
||||
|
||||
cat > "$STUB_DIR/sleep" <<'SH'
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
printf 'sleep-after-status=%s\n' "$*" >> "${MOSAIC_STUB_CLOCK_LOG:?}"
|
||||
SH
|
||||
chmod +x "$STUB_DIR/curl" "$STUB_DIR/date" "$STUB_DIR/sleep"
|
||||
chmod +x "$STUB_DIR/curl"
|
||||
|
||||
run_guard() {
|
||||
local status_mode="$1"
|
||||
@@ -152,46 +83,13 @@ run_guard() {
|
||||
export GITEA_URL=https://git.example.test
|
||||
export MOSAIC_STUB_STATUS_MODE="$status_mode"
|
||||
fi
|
||||
rm -f "$STATUS_OBSERVED" "$CLOCK_LOG"
|
||||
export MOSAIC_STUB_URL_LOG="$WORK_DIR/urls.log"
|
||||
export MOSAIC_STUB_STATUS_OBSERVED="$STATUS_OBSERVED"
|
||||
export MOSAIC_STUB_CLOCK_LOG="$CLOCK_LOG"
|
||||
export MOSAIC_CI_QUEUE_AUDIT_LOG="$audit_log"
|
||||
# Provider observation is the synchronization event. The one-second
|
||||
# timeout is subject semantics under virtual time, never a wall wait.
|
||||
# The absolute Python runtime uses an internal monotonic wait and kills
|
||||
# the subject's isolated process group. Neither operation can resolve
|
||||
# to the virtual date/sleep stubs at the front of PATH.
|
||||
local subject_rc
|
||||
if "$WATCHDOG_PYTHON" "$WATCHDOG_SCRIPT" "$WATCHDOG_TIMEOUT_SEC" \
|
||||
"$SCRIPT_DIR/ci-queue-wait.sh" --purpose "${MOSAIC_TEST_PURPOSE:-push}" -t 1 -i 1 "$@"; then
|
||||
subject_rc=0
|
||||
else
|
||||
subject_rc=$?
|
||||
fi
|
||||
return "$subject_rc"
|
||||
"$SCRIPT_DIR/ci-queue-wait.sh" --purpose "${MOSAIC_TEST_PURPOSE:-push}" -t 0 -i 0 "$@"
|
||||
)
|
||||
}
|
||||
|
||||
failures=0
|
||||
assert_provider_observed() {
|
||||
local name="$1" require_expiration="${2:-0}"
|
||||
if [[ ! -e "$STATUS_OBSERVED" ]]; then
|
||||
echo "FAIL $name: status provider was not observed" >&2
|
||||
failures=$((failures + 1))
|
||||
fi
|
||||
if [[ ! -s "$CLOCK_LOG" ]] || ! grep -q '^date-phase=before-status$' "$CLOCK_LOG"; then
|
||||
echo "FAIL $name: virtual clock interception did not run before provider observation" >&2
|
||||
failures=$((failures + 1))
|
||||
fi
|
||||
if [[ "$require_expiration" -eq 1 ]]; then
|
||||
if ! grep -q '^sleep-after-status=' "$CLOCK_LOG" || ! grep -q '^date-phase=after-status$' "$CLOCK_LOG"; then
|
||||
echo "FAIL $name: pending path did not expire after provider observation" >&2
|
||||
failures=$((failures + 1))
|
||||
fi
|
||||
fi
|
||||
}
|
||||
|
||||
run_assertion() {
|
||||
local name="$1" expected_rc="$2" status_mode="$3" required_text="$4"
|
||||
local output rc
|
||||
@@ -226,13 +124,6 @@ run_assertion() {
|
||||
printf '%s\n' "$output" >&2
|
||||
failures=$((failures + 1))
|
||||
fi
|
||||
if [[ "$status_mode" != "credential-unresolvable" ]]; then
|
||||
if [[ "$status_mode" == "pending" ]]; then
|
||||
assert_provider_observed "$name" 1
|
||||
else
|
||||
assert_provider_observed "$name"
|
||||
fi
|
||||
fi
|
||||
}
|
||||
|
||||
set -e
|
||||
@@ -249,27 +140,6 @@ run_assertion large-payload not126 large-success 'state=terminal-success'
|
||||
run_assertion credential-unresolvable zero credential-unresolvable 'CANNOT_ASSERT'
|
||||
run_assertion provider-unreachable zero unreachable 'CANNOT_ASSERT'
|
||||
|
||||
# Positive liveness control: a subject mutant hangs before the branch lookup
|
||||
# can reach the status provider. Only the independent real-clock watchdog may
|
||||
# terminate it, and its failure must be distinct from subject timeout rc=124.
|
||||
set +e
|
||||
watchdog_output=$(MOSAIC_STUB_BRANCH_MODE=hang-before-provider run_guard success "$AUDIT_LOG" 2>&1)
|
||||
watchdog_rc=$?
|
||||
set -e
|
||||
if [[ "$watchdog_rc" -ne "$WATCHDOG_EXIT" ]]; then
|
||||
echo "FAIL watchdog-control: expected hang-specific rc=$WATCHDOG_EXIT, got rc=$watchdog_rc" >&2
|
||||
failures=$((failures + 1))
|
||||
fi
|
||||
if [[ "$watchdog_output" != *"FAIL HANG watchdog:"* ]]; then
|
||||
echo "FAIL watchdog-control: expected distinct hang-specific diagnostic" >&2
|
||||
printf '%s\n' "$watchdog_output" >&2
|
||||
failures=$((failures + 1))
|
||||
fi
|
||||
if [[ -e "$STATUS_OBSERVED" ]]; then
|
||||
echo "FAIL watchdog-control: hanging mutant unexpectedly reached the status provider" >&2
|
||||
failures=$((failures + 1))
|
||||
fi
|
||||
|
||||
if [[ ! -s "$AUDIT_LOG" ]] || ! grep -q '"outcome":"CANNOT_ASSERT"' "$AUDIT_LOG"; then
|
||||
echo "FAIL provider-unreachable-audit: expected durable CANNOT_ASSERT JSONL record" >&2
|
||||
failures=$((failures + 1))
|
||||
@@ -290,7 +160,6 @@ if [[ "$merge_unreachable_output" != *"CANNOT_ASSERT"* ]]; then
|
||||
echo "FAIL merge-provider-unreachable: expected loud CANNOT_ASSERT diagnostic" >&2
|
||||
failures=$((failures + 1))
|
||||
fi
|
||||
assert_provider_observed merge-provider-unreachable
|
||||
merge_audit_lines_after=$(wc -l < "$AUDIT_LOG")
|
||||
if [[ "$merge_audit_lines_after" -le "$merge_audit_lines_before" ]]; then
|
||||
echo "FAIL merge-provider-unreachable: expected an additional audit record" >&2
|
||||
@@ -354,7 +223,6 @@ if [[ "$audit_failure_output" != *"audit"* ]]; then
|
||||
echo "FAIL audit-unavailable: expected loud audit failure diagnostic" >&2
|
||||
failures=$((failures + 1))
|
||||
fi
|
||||
assert_provider_observed audit-unavailable
|
||||
|
||||
if [[ "$failures" -ne 0 ]]; then
|
||||
echo "ci-queue-wait tri-state regression failed ($failures assertions)" >&2
|
||||
|
||||
@@ -39,12 +39,11 @@ ORIG_PATH="$PATH"
|
||||
# loop — which would make the control a false negative. A root dotfile is
|
||||
# operator-owned (unknown→operator), so the sync loop skips it. Clean up on exit.
|
||||
STRIPPED="$FW/.install-rollback-control.tmp.sh"
|
||||
SIGNALED="$FW/.install-signal-control.tmp.sh"
|
||||
NOEXIT="$FW/.install-noexit-control.tmp.sh"
|
||||
D1CTRL="$FW/.install-d1guard-control.tmp.sh"
|
||||
D2CTRL="$FW/.install-d2guard-control.tmp.sh"
|
||||
rm -f "$STRIPPED" "$SIGNALED" "$NOEXIT" "$D1CTRL" "$D2CTRL"
|
||||
trap 'rm -f "$STRIPPED" "$SIGNALED" "$NOEXIT" "$D1CTRL" "$D2CTRL"' EXIT
|
||||
rm -f "$STRIPPED" "$NOEXIT" "$D1CTRL" "$D2CTRL"
|
||||
trap 'rm -f "$STRIPPED" "$NOEXIT" "$D1CTRL" "$D2CTRL"' EXIT
|
||||
|
||||
pass=0; fail=0
|
||||
chk() { if eval "$2"; then echo " ✓ $1"; pass=$((pass + 1)); else echo " ✗ $1"; fail=$((fail + 1)); fi; }
|
||||
@@ -181,86 +180,41 @@ chk "[control] without -E the mid-sync corruption survives (no rollback)" \
|
||||
# ── Part C: an INT/TERM interrupt must terminate, not resume (blocker-A) ──────
|
||||
# A bash signal trap that merely returns lets the script continue past the
|
||||
# interrupt — restoring the snapshot, then resuming the sync and reporting
|
||||
# success. The earlier test used a child cp shim to signal its parent, making
|
||||
# child completion race Bash's interrupted wait. Concurrency is not part of the
|
||||
# guarded property: sync_framework_keep() runs in the installer's own Bash
|
||||
# process, and `kill` is a builtin. Generate two installer fixtures that signal
|
||||
# themselves at the same known mid-sync point. Their TERM handlers emit the same
|
||||
# observable before diverging, so missing signal delivery fails BOTH arms rather
|
||||
# than manufacturing a pass. The only semantic difference between fixtures is
|
||||
# the explicit `exit 1` whose load-bearing behavior this control proves.
|
||||
TERM_MARKER='[test-control] TERM handler entered'
|
||||
HANDLER_WITH_EXIT="trap 'echo \"$TERM_MARKER\" >&2; restore_snapshot; exit 1' TERM # TEST-TERM-HANDLER"
|
||||
HANDLER_WITHOUT_EXIT="trap 'echo \"$TERM_MARKER\" >&2; restore_snapshot' TERM # TEST-TERM-HANDLER"
|
||||
|
||||
make_signal_installer() {
|
||||
local output="$1" handler="$2"
|
||||
local target_trap="trap 'restore_snapshot; exit 1' ERR INT TERM"
|
||||
local target_cp=' cp "$abs" "$dst/$rel"'
|
||||
local inject_open=" if [[ \"\$rel\" == \"$POISON_REL\" ]]; then"
|
||||
local inject_kill=' kill -TERM "$$" # TEST-TERM-INJECTION'
|
||||
local inject_close=' fi'
|
||||
|
||||
if ! awk \
|
||||
-v target_trap="$target_trap" -v target_cp="$target_cp" \
|
||||
-v handler="$handler" -v inject_open="$inject_open" \
|
||||
-v inject_kill="$inject_kill" -v inject_close="$inject_close" '
|
||||
$0 == target_cp {
|
||||
print inject_open
|
||||
print inject_kill
|
||||
print inject_close
|
||||
injection_sites++
|
||||
}
|
||||
{ print }
|
||||
$0 == target_trap {
|
||||
print handler
|
||||
handler_sites++
|
||||
}
|
||||
END {
|
||||
if (handler_sites != 1 || injection_sites != 1) exit 42
|
||||
}
|
||||
' "$INSTALL" > "$output"; then
|
||||
rm -f "$output"
|
||||
fail "Could not construct the self-TERM control installer at the exact trap/copy sites"
|
||||
exit 1
|
||||
fi
|
||||
chmod +x "$output"
|
||||
# success. We inject a SIGTERM mid-sync with a cp that SUCCEEDS (so set -e never
|
||||
# fires and ONLY the signal path governs), and assert the shipped installer
|
||||
# restores AND exits without reporting success. The control strips `exit 1` from
|
||||
# the trap and shows the buggy resume-to-success.
|
||||
make_term_shim() {
|
||||
local dir="$1"
|
||||
cat > "$dir/cp" <<SHIM
|
||||
#!/usr/bin/env bash
|
||||
dest="\${@: -1}"
|
||||
case "\$dest" in
|
||||
*/$POISON_REL)
|
||||
kill -TERM "\$PPID" 2>/dev/null # signal install.sh; the copy still succeeds
|
||||
exec env PATH="$ORIG_PATH" cp "\$@" ;;
|
||||
esac
|
||||
exec env PATH="$ORIG_PATH" cp "\$@"
|
||||
SHIM
|
||||
chmod +x "$dir/cp"
|
||||
}
|
||||
|
||||
make_signal_installer "$SIGNALED" "$HANDLER_WITH_EXIT"
|
||||
make_signal_installer "$NOEXIT" "$HANDLER_WITHOUT_EXIT"
|
||||
signal_fixture_ready() {
|
||||
local fixture="$1" expected_handler="$2"
|
||||
[[ "$(grep -cF '# TEST-TERM-INJECTION' "$fixture")" -eq 1 ]] \
|
||||
&& [[ "$(grep -cF '# TEST-TERM-HANDLER' "$fixture")" -eq 1 ]] \
|
||||
&& grep -Fqx "$expected_handler" "$fixture"
|
||||
}
|
||||
signaled_fixture_ready() { signal_fixture_ready "$SIGNALED" "$HANDLER_WITH_EXIT"; }
|
||||
noexit_fixture_ready() { signal_fixture_ready "$NOEXIT" "$HANDLER_WITHOUT_EXIT"; }
|
||||
chk "[signal] shipped fixture has exactly one self-TERM injection and marked handler" \
|
||||
"signaled_fixture_ready"
|
||||
chk "[control] no-exit fixture has exactly one self-TERM injection and marked handler" \
|
||||
"noexit_fixture_ready"
|
||||
chk "[control] removing the explicit TERM exit changes the fixture" \
|
||||
"! cmp -s '$SIGNALED' '$NOEXIT'"
|
||||
|
||||
# Run one keep-mode upgrade whose own shell delivers SIGTERM synchronously at
|
||||
# the selected copy. Echoes "<exit>\t<out>\t<home>".
|
||||
# Run one keep-mode upgrade with the SIGTERM shim. Echoes "<exit>\t<out>\t<home>".
|
||||
run_signal_upgrade() {
|
||||
local installer="$1" H OUT rc
|
||||
H=$(mktemp -d); OUT=$(mktemp)
|
||||
local installer="$1" H OUT SHIM rc
|
||||
H=$(mktemp -d); OUT=$(mktemp); SHIM=$(mktemp -d)
|
||||
seed_home "$H"
|
||||
make_term_shim "$SHIM"
|
||||
set +e
|
||||
PATH="$ORIG_PATH" \
|
||||
PATH="$SHIM:$ORIG_PATH" \
|
||||
MOSAIC_HOME="$H" MOSAIC_INSTALL_MODE=keep MOSAIC_SYNC_ONLY=1 bash "$installer" >"$OUT" 2>&1
|
||||
rc=$?
|
||||
set -e 2>/dev/null || true
|
||||
rm -rf "$SHIM"
|
||||
printf '%s\t%s\t%s\n' "$rc" "$OUT" "$H"
|
||||
}
|
||||
|
||||
IFS=$'\t' read -r rcC OUTC HC < <(run_signal_upgrade "$SIGNALED")
|
||||
chk "[signal] TERM handler observable fires exactly once" \
|
||||
"[ \"\$(grep -cF '$TERM_MARKER' '$OUTC')\" -eq 1 ]"
|
||||
IFS=$'\t' read -r rcC OUTC HC < <(run_signal_upgrade "$INSTALL")
|
||||
chk "[signal] SIGTERM mid-sync aborts non-zero (trap exits, does not resume)" \
|
||||
"[ '$rcC' -ne 0 ]"
|
||||
chk "[signal] restore_snapshot fires on the interrupt" \
|
||||
@@ -268,13 +222,13 @@ chk "[signal] restore_snapshot fires on the interrupt" \
|
||||
chk "[signal] does NOT resume to report sync success after the interrupt" \
|
||||
"! grep -q 'file phase complete' '$OUTC'"
|
||||
|
||||
IFS=$'\t' read -r rcD OUTD HD < <(run_signal_upgrade "$NOEXIT")
|
||||
chk "[control] TERM handler observable fires exactly once" \
|
||||
"[ \"\$(grep -cF '$TERM_MARKER' '$OUTD')\" -eq 1 ]"
|
||||
chk "[control] without 'exit 1' the handler restores before returning" \
|
||||
"grep -q 'restoring previous state from snapshot' '$OUTD'"
|
||||
chk "[control] without 'exit 1' the installer exits zero after resuming" \
|
||||
"[ '$rcD' -eq 0 ]"
|
||||
# Control: strip `exit 1` from the signal trap → the handler returns, the script
|
||||
# resumes past the interrupt and wrongly reports success. In $FW so SOURCE_DIR resolves.
|
||||
sed "s/trap 'restore_snapshot; exit 1' ERR INT TERM/trap 'restore_snapshot' ERR INT TERM/" \
|
||||
"$INSTALL" > "$NOEXIT"
|
||||
chk "[control] the exit-strip actually changed the installer" \
|
||||
"! cmp -s '$INSTALL' '$NOEXIT'"
|
||||
IFS=$'\t' read -r _rcD OUTD HD < <(run_signal_upgrade "$NOEXIT")
|
||||
chk "[control] without 'exit 1' the trap resumes and reports sync success (the bug)" \
|
||||
"grep -q 'file phase complete' '$OUTD'"
|
||||
|
||||
@@ -355,10 +309,10 @@ chk "[control] without the D2 recovery line the operator gets no snapshot pointe
|
||||
# Reap any snapshot the reset-fail runs left in /tmp (reset failed → never cleaned).
|
||||
grep -o '/[^ ]*mosaic-snapshot[^ ]*' "$OUTH" 2>/dev/null | head -1 | while read -r s; do rm -rf "$s"; done
|
||||
|
||||
# Cleanup (generated installer controls are also removed by the EXIT trap).
|
||||
# Cleanup ($STRIPPED / $NOEXIT / $D1CTRL / $D2CTRL are also removed by the EXIT trap).
|
||||
for d in "$HA" "$REFA" "$HB" "$REFB" "$HC" "$HD" "$HE" "$REFE" "$HF" "$REFF" "$HG" "$HH"; do rm -rf "$d"; done
|
||||
rm -f "$OUTA" "$OUTB" "$OUTC" "$OUTD" "$OUTE" "$OUTF" "$OUTG" "$OUTH" \
|
||||
"$STRIPPED" "$SIGNALED" "$NOEXIT" "$D1CTRL" "$D2CTRL"
|
||||
"$STRIPPED" "$NOEXIT" "$D1CTRL" "$D2CTRL"
|
||||
|
||||
echo
|
||||
echo "RESULT: $pass passed, $fail failed"
|
||||
|
||||
@@ -26,13 +26,12 @@ A Woodpecker API token is required. To configure:
|
||||
|
||||
## Scripts
|
||||
|
||||
| Script | Purpose |
|
||||
| -------------------------- | -------------------------------------------------------------- |
|
||||
| `pipeline-list.sh` | List recent pipelines for a repo |
|
||||
| `pipeline-status.sh` | Get status of a specific or latest pipeline |
|
||||
| `pipeline-trigger.sh` | Trigger a new pipeline build |
|
||||
| `ci-wait.sh` | Block until pipeline(s) reach terminal state |
|
||||
| `verify-terminal-green.py` | Verify every JSON/API child step under the bounded CI contract |
|
||||
| Script | Purpose |
|
||||
| --------------------- | -------------------------------------------- |
|
||||
| `pipeline-list.sh` | List recent pipelines for a repo |
|
||||
| `pipeline-status.sh` | Get status of a specific or latest pipeline |
|
||||
| `pipeline-trigger.sh` | Trigger a new pipeline build |
|
||||
| `ci-wait.sh` | Block until pipeline(s) reach terminal state |
|
||||
|
||||
## Common Options
|
||||
|
||||
@@ -60,9 +59,4 @@ A Woodpecker API token is required. To configure:
|
||||
|
||||
# Block until one or more pipelines finish (event-driven CI wait)
|
||||
~/.config/mosaic/tools/woodpecker/ci-wait.sh -r usc/uconnect -n 3917 -n 3918
|
||||
|
||||
# Verify the full JSON child-step record; do not use the text summary for this gate
|
||||
PR_HEAD=<full-40-hex-provider-head>
|
||||
~/.config/mosaic/tools/woodpecker/pipeline-status.sh -r mosaicstack/stack -n 2188 -f json \
|
||||
| ~/.config/mosaic/tools/woodpecker/verify-terminal-green.py --expect-commit "$PR_HEAD" -
|
||||
```
|
||||
|
||||
@@ -1,109 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Red-first contract harness for RM-61 / #1000.
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
VERIFIER="$SCRIPT_DIR/verify-terminal-green.py"
|
||||
EXPECTED_COMMIT=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
|
||||
TMP=$(mktemp -d)
|
||||
trap 'rm -rf "$TMP"' EXIT
|
||||
|
||||
write_fixture() {
|
||||
local file="$1" pipeline_status="$2" postgres_state="$3" postgres_exit="$4" postgres_error="$5" test_state="$6"
|
||||
python3 - "$file" "$pipeline_status" "$postgres_state" "$postgres_exit" "$postgres_error" "$test_state" <<'PY'
|
||||
import json, sys
|
||||
path, pipeline_status, pg_state, pg_exit, pg_error, test_state = sys.argv[1:]
|
||||
steps = [
|
||||
{"name": "clone", "type": "clone", "state": "success", "exit_code": 0, "error": None},
|
||||
{"name": "ci-postgres", "type": "service", "state": pg_state, "exit_code": int(pg_exit), "error": pg_error or None},
|
||||
{"name": "test", "type": "commands", "state": test_state, "exit_code": 0 if test_state == "success" else 1, "error": None},
|
||||
]
|
||||
json.dump({
|
||||
"number": 9999,
|
||||
"status": pipeline_status,
|
||||
"commit": "a" * 40,
|
||||
"workflows": [{"name": "ci", "state": pipeline_status, "children": steps}],
|
||||
}, open(path, "w"))
|
||||
PY
|
||||
}
|
||||
|
||||
expect_exit() {
|
||||
local expected_exit="$1" label="$2" file="$3" expected_commit="${4:-$EXPECTED_COMMIT}"
|
||||
set +e
|
||||
output=$(python3 "$VERIFIER" --expect-commit "$expected_commit" "$file" 2>&1)
|
||||
actual=$?
|
||||
set -e
|
||||
if [[ "$actual" -ne "$expected_exit" ]]; then
|
||||
printf 'FAIL %s: expected exit %s, got %s\n%s\n' "$label" "$expected_exit" "$actual" "$output" >&2
|
||||
exit 1
|
||||
fi
|
||||
printf 'PASS %s\n' "$label"
|
||||
printf '%s' "$output"
|
||||
}
|
||||
|
||||
# Ordinary terminal green.
|
||||
write_fixture "$TMP/green.json" success success 0 '' success
|
||||
out=$(expect_exit 0 green "$TMP/green.json")
|
||||
grep -q '"total_steps": 3' <<<"$out"
|
||||
grep -q '"exempted_steps": 0' <<<"$out"
|
||||
|
||||
# Exact, named #1000 teardown artifact: the only permitted non-success child.
|
||||
artifact='pods "wp-svc-01kyxzjhdf6w81swsnbfzh85z9-ci-postgres" not found'
|
||||
write_fixture "$TMP/artifact.json" success failure 0 "$artifact" success
|
||||
out=$(expect_exit 0 exact-artifact "$TMP/artifact.json")
|
||||
grep -q '"exemption_id": "WP-K8S-1000-CI-POSTGRES-TEARDOWN"' <<<"$out"
|
||||
grep -q '"exempted_steps": 1' <<<"$out"
|
||||
|
||||
# Negative controls: both real PostgreSQL failures must remain red.
|
||||
write_fixture "$TMP/startup.json" failure failure 1 '' failure
|
||||
expect_exit 1 startup-failure "$TMP/startup.json" >/dev/null
|
||||
write_fixture "$TMP/crash.json" failure failure 137 '' failure
|
||||
expect_exit 1 post-readiness-crash "$TMP/crash.json" >/dev/null
|
||||
|
||||
# The exemption is signature-scoped, not step-scoped.
|
||||
write_fixture "$TMP/wrong-error.json" success failure 0 'connection refused' success
|
||||
expect_exit 1 other-postgres-error "$TMP/wrong-error.json" >/dev/null
|
||||
write_fixture "$TMP/wrong-pod.json" success failure 0 'pods "other-ci-postgres" not found' success
|
||||
expect_exit 1 wrong-pod-signature "$TMP/wrong-pod.json" >/dev/null
|
||||
write_fixture "$TMP/nonzero-artifact.json" success failure 137 "$artifact" success
|
||||
expect_exit 1 nonzero-with-artifact-text "$TMP/nonzero-artifact.json" >/dev/null
|
||||
|
||||
# JSON booleans and non-integer zero look equal to 0 in Python but are not exit codes.
|
||||
python3 - "$TMP/artifact.json" "$TMP" <<'PY'
|
||||
import json, os, sys
|
||||
record = json.load(open(sys.argv[1]))
|
||||
for label, value in (("false", False), ("true", True), ("float", 0.0), ("string", "0"), ("null", None)):
|
||||
changed = json.loads(json.dumps(record))
|
||||
changed["workflows"][0]["children"][1]["exit_code"] = value
|
||||
json.dump(changed, open(os.path.join(sys.argv[2], f"exit-{label}.json"), "w"))
|
||||
PY
|
||||
for label in false true float string null; do
|
||||
expect_exit 1 "non-integer-exit-$label" "$TMP/exit-$label.json" >/dev/null
|
||||
done
|
||||
|
||||
# Exact artifact cannot mask any independent failure or non-success pipeline.
|
||||
write_fixture "$TMP/artifact-plus-failure.json" failure failure 0 "$artifact" failure
|
||||
expect_exit 1 artifact-plus-real-failure "$TMP/artifact-plus-failure.json" >/dev/null
|
||||
write_fixture "$TMP/skipped.json" success success 0 '' skipped
|
||||
expect_exit 1 skipped-step "$TMP/skipped.json" >/dev/null
|
||||
|
||||
# The scanned pipeline must be bound to an explicit, full PR-head commit.
|
||||
set +e
|
||||
missing_output=$(python3 "$VERIFIER" "$TMP/artifact.json" 2>&1)
|
||||
missing_rc=$?
|
||||
set -e
|
||||
if [[ "$missing_rc" -ne 2 ]] || ! grep -q -- '--expect-commit' <<<"$missing_output"; then
|
||||
printf 'FAIL missing-expected-commit: expected usage exit 2\n%s\n' "$missing_output" >&2
|
||||
exit 1
|
||||
fi
|
||||
expect_exit 1 mismatched-expected-commit "$TMP/artifact.json" bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb >/dev/null
|
||||
|
||||
python3 - "$TMP/artifact.json" "$TMP/missing-record-commit.json" <<'PY'
|
||||
import json, sys
|
||||
record = json.load(open(sys.argv[1]))
|
||||
record.pop("commit")
|
||||
json.dump(record, open(sys.argv[2], "w"))
|
||||
PY
|
||||
expect_exit 1 missing-record-commit "$TMP/missing-record-commit.json" >/dev/null
|
||||
|
||||
printf 'terminal-green contract harness: PASS (17 cases)\n'
|
||||
@@ -1,230 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Verify Mosaic's full-step Woodpecker terminal-green contract.
|
||||
|
||||
RM-61 permits one named, signature-scoped exception for issue #1000. The
|
||||
exception retires when #1000 is fixed; all other non-success states block.
|
||||
This program consumes the JSON/API record emitted by pipeline-status.sh -f json.
|
||||
It does not fetch, retry, or re-trigger pipelines.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
from collections import Counter
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
EXEMPTION_ID = "WP-K8S-1000-CI-POSTGRES-TEARDOWN"
|
||||
EXEMPTION_ISSUE = "https://git.mosaicstack.dev/mosaicstack/stack/issues/1000"
|
||||
POD_NOT_FOUND = re.compile(
|
||||
r'^pods "wp-svc-[0-9a-hjkmnp-tv-z]{26}-ci-postgres" not found$'
|
||||
)
|
||||
|
||||
|
||||
def fail_usage(message: str) -> int:
|
||||
print(f"terminal-green contract input error: {message}", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
|
||||
def load_record(argument: str | None) -> dict[str, Any]:
|
||||
if argument in (None, "-"):
|
||||
value = json.load(sys.stdin)
|
||||
else:
|
||||
with Path(argument).open(encoding="utf-8") as handle:
|
||||
value = json.load(handle)
|
||||
if not isinstance(value, dict):
|
||||
raise ValueError("pipeline record must be a JSON object")
|
||||
return value
|
||||
|
||||
|
||||
def is_issue_1000_artifact(step: dict[str, Any]) -> bool:
|
||||
error = step.get("error")
|
||||
exit_code = step.get("exit_code")
|
||||
return (
|
||||
step.get("name") == "ci-postgres"
|
||||
and step.get("type") == "service"
|
||||
and step.get("state") == "failure"
|
||||
and type(exit_code) is int
|
||||
and not isinstance(exit_code, bool)
|
||||
and exit_code == 0
|
||||
and isinstance(error, str)
|
||||
and POD_NOT_FOUND.fullmatch(error) is not None
|
||||
)
|
||||
|
||||
|
||||
def verify(record: dict[str, Any], expected_commit: str) -> tuple[int, dict[str, Any]]:
|
||||
anomalies: list[dict[str, Any]] = []
|
||||
candidates: list[dict[str, Any]] = []
|
||||
steps: list[dict[str, Any]] = []
|
||||
|
||||
pipeline_status = record.get("status")
|
||||
actual_commit = record.get("commit")
|
||||
if actual_commit != expected_commit:
|
||||
anomalies.append(
|
||||
{
|
||||
"scope": "pipeline",
|
||||
"name": str(record.get("number", "unknown")),
|
||||
"state": pipeline_status,
|
||||
"reason": "pipeline commit does not equal the expected PR head",
|
||||
"expected_commit": expected_commit,
|
||||
"actual_commit": actual_commit,
|
||||
}
|
||||
)
|
||||
if pipeline_status != "success":
|
||||
anomalies.append(
|
||||
{
|
||||
"scope": "pipeline",
|
||||
"name": str(record.get("number", "unknown")),
|
||||
"state": pipeline_status,
|
||||
"reason": "pipeline status is not success",
|
||||
}
|
||||
)
|
||||
|
||||
workflows = record.get("workflows")
|
||||
if not isinstance(workflows, list) or not workflows:
|
||||
anomalies.append(
|
||||
{
|
||||
"scope": "pipeline",
|
||||
"name": str(record.get("number", "unknown")),
|
||||
"state": pipeline_status,
|
||||
"reason": "workflows are missing or empty",
|
||||
}
|
||||
)
|
||||
workflows = []
|
||||
|
||||
for workflow_index, workflow in enumerate(workflows):
|
||||
if not isinstance(workflow, dict):
|
||||
anomalies.append(
|
||||
{
|
||||
"scope": "workflow",
|
||||
"name": str(workflow_index),
|
||||
"state": None,
|
||||
"reason": "workflow is not an object",
|
||||
}
|
||||
)
|
||||
continue
|
||||
workflow_name = str(workflow.get("name", workflow_index))
|
||||
if workflow.get("state") != "success":
|
||||
anomalies.append(
|
||||
{
|
||||
"scope": "workflow",
|
||||
"name": workflow_name,
|
||||
"state": workflow.get("state"),
|
||||
"reason": "workflow state is not success",
|
||||
}
|
||||
)
|
||||
children = workflow.get("children")
|
||||
if not isinstance(children, list) or not children:
|
||||
anomalies.append(
|
||||
{
|
||||
"scope": "workflow",
|
||||
"name": workflow_name,
|
||||
"state": workflow.get("state"),
|
||||
"reason": "child-step list is missing or empty",
|
||||
}
|
||||
)
|
||||
continue
|
||||
for child_index, child in enumerate(children):
|
||||
if not isinstance(child, dict):
|
||||
anomalies.append(
|
||||
{
|
||||
"scope": "step",
|
||||
"name": f"{workflow_name}[{child_index}]",
|
||||
"state": None,
|
||||
"reason": "step is not an object",
|
||||
}
|
||||
)
|
||||
continue
|
||||
steps.append(child)
|
||||
if child.get("state") == "success":
|
||||
continue
|
||||
if is_issue_1000_artifact(child):
|
||||
candidates.append(child)
|
||||
continue
|
||||
anomalies.append(
|
||||
{
|
||||
"scope": "step",
|
||||
"name": child.get("name"),
|
||||
"type": child.get("type"),
|
||||
"state": child.get("state"),
|
||||
"exit_code": child.get("exit_code"),
|
||||
"error": child.get("error"),
|
||||
"reason": "non-success step does not match the #1000 teardown signature",
|
||||
}
|
||||
)
|
||||
|
||||
if len(candidates) > 1:
|
||||
anomalies.append(
|
||||
{
|
||||
"scope": "exemption",
|
||||
"name": EXEMPTION_ID,
|
||||
"state": "invalid",
|
||||
"reason": "the #1000 exemption may apply to exactly one step",
|
||||
}
|
||||
)
|
||||
|
||||
exemption_applies = len(candidates) == 1 and not anomalies
|
||||
state_counts = Counter(str(step.get("state", "missing")) for step in steps)
|
||||
result: dict[str, Any] = {
|
||||
"schema_version": "mosaic-terminal-green/v1",
|
||||
"verdict": "terminal-green" if not anomalies else "not-terminal-green",
|
||||
"pipeline_number": record.get("number"),
|
||||
"commit": actual_commit,
|
||||
"expected_commit": expected_commit,
|
||||
"pipeline_status": pipeline_status,
|
||||
"total_steps": len(steps),
|
||||
"state_counts": dict(sorted(state_counts.items())),
|
||||
"exempted_steps": 1 if exemption_applies else 0,
|
||||
"anomalies": anomalies,
|
||||
}
|
||||
if exemption_applies:
|
||||
candidate = candidates[0]
|
||||
result["exemptions"] = [
|
||||
{
|
||||
"exemption_id": EXEMPTION_ID,
|
||||
"step": candidate.get("name"),
|
||||
"signature": candidate.get("error"),
|
||||
"tracking_issue": EXEMPTION_ISSUE,
|
||||
"retires_when": "issue #1000 is fixed",
|
||||
}
|
||||
]
|
||||
else:
|
||||
result["exemptions"] = []
|
||||
|
||||
return (0 if not anomalies else 1), result
|
||||
|
||||
|
||||
def parse_arguments() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="verify the full Woodpecker terminal-green child-step contract"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--expect-commit",
|
||||
required=True,
|
||||
metavar="FULL_SHA",
|
||||
help="full 40-hex PR-head commit that the pipeline record must match",
|
||||
)
|
||||
parser.add_argument("record", nargs="?", default="-", help="pipeline JSON file or -")
|
||||
arguments = parser.parse_args()
|
||||
if re.fullmatch(r"[0-9a-fA-F]{40}", arguments.expect_commit) is None:
|
||||
parser.error("--expect-commit must be a full 40-hex commit")
|
||||
arguments.expect_commit = arguments.expect_commit.lower()
|
||||
return arguments
|
||||
|
||||
|
||||
def main() -> int:
|
||||
arguments = parse_arguments()
|
||||
try:
|
||||
record = load_record(arguments.record)
|
||||
except (OSError, ValueError, json.JSONDecodeError) as error:
|
||||
return fail_usage(str(error))
|
||||
code, result = verify(record, arguments.expect_commit)
|
||||
print(json.dumps(result, indent=2, sort_keys=True))
|
||||
return code
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@mosaicstack/mosaic",
|
||||
"version": "0.0.49",
|
||||
"version": "0.0.48",
|
||||
"repository": {
|
||||
"type": "git",
|
||||
"url": "https://git.mosaicstack.dev/mosaicstack/stack.git",
|
||||
@@ -25,7 +25,7 @@
|
||||
"lint": "eslint src",
|
||||
"typecheck": "tsc --noEmit",
|
||||
"test": "vitest run --passWithNoTests && pnpm run test:framework-shell",
|
||||
"test:framework-shell": "bash framework/tools/quality/scripts/check-test-enumeration.sh && bash framework/tools/quality/scripts/test-check-test-enumeration.sh && python3 src/lease-broker/daemon_deadline_unittest.py && python3 src/lease-broker/normative_fragments_unittest.py && python3 src/lease-broker/receipt_challenge_unittest.py && python3 src/lease-broker/context_recovery_unittest.py && python3 src/lease-broker/recovery_runtime_unittest.py && python3 src/lease-broker/recovery_b1_adversarial_unittest.py && python3 src/lease-broker/framework_skill_portability_unittest.py && python3 src/mutator-gate/runtime_tools_unittest.py && python3 src/mutator-gate/runtime_launch_guard_unittest.py && python3 src/mutator-gate/version_coupling_unittest.py && python3 framework/tools/lease-broker/check-runtime-launches.py --root ../.. && bash framework/tools/codex/test-pr-diff-context.sh && bash framework/tools/qa/test-deps-preflight.sh && bash framework/tools/git/test-pr-review-gitea-comment.sh && bash framework/tools/git/test-pr-review-repo-host-override.sh && bash framework/tools/git/test-ci-queue-wait-branch-absent.sh && bash framework/tools/git/test-ci-queue-wait-tristate.sh && bash framework/tools/git/test-ci-queue-wait-github-checks.sh && bash framework/tools/git/test-pr-merge-queue-branch.sh && bash framework/tools/git/test-pr-merge-head-pin.sh && bash framework/tools/git/test-git-credential-mosaic.sh && bash framework/tools/git/test-gitea-token-identity.sh && bash framework/tools/woodpecker/test-terminal-green-contract.sh && bash framework/tools/_scripts/test-install-ordering-guard.sh && bash framework/tools/tmux/agent-send.test.sh && bash framework/tools/wake/test-wake-store-ack.sh && bash framework/tools/wake/test-wake-store-enqueue-race.sh && bash framework/tools/wake/test-wake-digest-hmac.sh && bash framework/tools/wake/test-wake-digest-quarantine.sh && bash framework/tools/wake/test-wake-detector.sh && bash framework/tools/wake/test-wake-fn-oracle.sh && bash framework/tools/wake/test-wake-reconcile.sh && bash framework/tools/wake/test-wake-beacon.sh && bash framework/tools/wake/test-wake-preimage.sh && bash framework/tools/wake/test-wake-install.sh"
|
||||
"test:framework-shell": "bash framework/tools/quality/scripts/check-test-enumeration.sh && bash framework/tools/quality/scripts/test-check-test-enumeration.sh && python3 src/lease-broker/daemon_deadline_unittest.py && python3 src/lease-broker/normative_fragments_unittest.py && python3 src/lease-broker/receipt_challenge_unittest.py && python3 src/lease-broker/context_recovery_unittest.py && python3 src/lease-broker/recovery_runtime_unittest.py && python3 src/lease-broker/recovery_b1_adversarial_unittest.py && python3 src/lease-broker/framework_skill_portability_unittest.py && python3 src/mutator-gate/runtime_tools_unittest.py && python3 src/mutator-gate/runtime_launch_guard_unittest.py && python3 src/mutator-gate/version_coupling_unittest.py && python3 framework/tools/lease-broker/check-runtime-launches.py --root ../.. && bash framework/tools/codex/test-pr-diff-context.sh && bash framework/tools/qa/test-deps-preflight.sh && bash framework/tools/git/test-pr-review-gitea-comment.sh && bash framework/tools/git/test-pr-review-repo-host-override.sh && bash framework/tools/git/test-ci-queue-wait-branch-absent.sh && bash framework/tools/git/test-ci-queue-wait-tristate.sh && bash framework/tools/git/test-ci-queue-wait-github-checks.sh && bash framework/tools/git/test-pr-merge-queue-branch.sh && bash framework/tools/git/test-pr-merge-head-pin.sh && bash framework/tools/git/test-git-credential-mosaic.sh && bash framework/tools/git/test-gitea-token-identity.sh && bash framework/tools/_scripts/test-install-ordering-guard.sh && bash framework/tools/tmux/agent-send.test.sh && bash framework/tools/wake/test-wake-store-ack.sh && bash framework/tools/wake/test-wake-store-enqueue-race.sh && bash framework/tools/wake/test-wake-digest-hmac.sh && bash framework/tools/wake/test-wake-digest-quarantine.sh && bash framework/tools/wake/test-wake-detector.sh && bash framework/tools/wake/test-wake-fn-oracle.sh && bash framework/tools/wake/test-wake-reconcile.sh && bash framework/tools/wake/test-wake-beacon.sh && bash framework/tools/wake/test-wake-preimage.sh && bash framework/tools/wake/test-wake-install.sh"
|
||||
},
|
||||
"dependencies": {
|
||||
"@mosaicstack/brain": "workspace:*",
|
||||
|
||||
Reference in New Issue
Block a user