Compare commits

..

2 Commits

Author SHA1 Message Date
Hermes Agent
e88a89f34d fix(framework): remediate Codex review findings in VAULT-SECRETS.md
Some checks failed
ci/woodpecker/push/ci Pipeline failed
ci/woodpecker/pr/ci Pipeline failed
Two should-fix findings from automated Codex review:

1. Vault KV v2 policy path — add explicit path for exact top-level
   `secret/data/k3s/<app>` entry alongside the wildcard `/*` sub-path
   rule. Without the exact path, apps reading the top-level secret get
   permission denied from Vault KV v2 even with the wildcard.

2. Go envconfig example — remove unused `os` import from config.go
   snippet (os was only referenced in a comment). Move the main() usage
   to a separate clearly-labelled main.go block to make both snippets
   copy-paste compilable.

Both fixes mirrored to duplicate path:
  guides/ <-> packages/mosaic/framework/guides/

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-22 12:01:29 -05:00
Hermes Agent
373e4558a3 chore(framework): canonize Vault-as-SSOT + ESO-default secrets policy
Some checks failed
ci/woodpecker/push/ci Pipeline failed
ci/woodpecker/pr/ci Pipeline failed
Encodes operator-approved (Jason, 2026-05-22) secrets policy as binding
framework rules across all Mosaic agent sessions and projects.

Changes:
- STANDARDS.md: add "Secrets handling (HARD RULE)" subsection under
  Non-Negotiables — Vault as SSOT, ESO bridge as default, Direct-Vault
  opt-in only, forbidden ${VAR:-default} for required values, forbidden
  .env in prod, required startup schema validation
- VAULT-SECRETS.md: add four new sections — architecture decision matrix
  (ESO vs Direct-Vault), full ESO bridge worked example (Vault path +
  ExternalSecret + Deployment YAML + zod/pydantic/Go validators),
  Direct-Vault opt-in pattern (AppRole provisioning + ESO bootstrap
  for chicken-and-egg), and forbidden patterns CI lint targets
- BOOTSTRAP.md: add "Secrets Bootstrap" required subsection with
  checklist for new apps (Vault path, README docs, ExternalSecret,
  secretKeyRef, schema validator, Direct-Vault justification)

All duplicate file paths kept in sync (md5-equal pairs):
  guides/ <-> packages/mosaic/framework/guides/
  packages/mosaic/framework/defaults/STANDARDS.md (single copy in repo)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-22 11:58:27 -05:00
11 changed files with 843 additions and 454 deletions

View File

@@ -30,7 +30,6 @@ These are MVP-level checks that don't belong to any single workstream. Updated b
| MVP-T04 | not-started | Sync `.mosaic/orchestrator/mission.json` MVP slot with this manifest (milestone enumeration, etc.) | Coord state file; consider whether to repopulate via `mosaic coord` or accept hand-edit |
| MVP-T05 | in-progress | Kick off W1 / FED-M1 — federated tier infrastructure | Session 16 (2026-04-19): FED-M1-01 in-progress on `feat/federation-m1-tier-config` |
| MVP-T06 | not-started | Declare additional workstreams (web dashboard, TUI/CLI parity, remote control, etc.) as scope solidifies | Track each new workstream by adding a row to the Workstream Rollup |
| T-A292E96F | in-progress | Fix Mosaic Gitea PR metadata/login wrapper regression for U-Connect merge preflight | Kanban `t_a292e96f`; branch `fix/t-a292e96f-gitea-pr-metadata`; scratchpad `docs/scratchpads/t-a292e96f-gitea-pr-metadata.md` |
## Pointer to Active Workstream

View File

@@ -1,51 +0,0 @@
# t_a292e96f — Gitea PR metadata wrapper fix
## Objective
Repair Mosaic git wrappers so Gitea PR metadata and merge preflight work for U-Connect PRs on `git.uscllc.com` without selecting the unrelated `git.mosaicstack.dev` tea login.
## Findings
- Reproduced the failure from `/src/uconnect-worktrees/t_39ce717c-authentik-smoke-gate` with the current `pr-metadata.sh`:
- PR #1905 returned JSON with `number=null`, `baseRefName=""`, `headRefName=""`.
- PR #1908 returned JSON with `number=null`, `baseRefName=""`, `headRefName=""`.
- Root cause: the wrapper treated HTTP/API error payloads as PR payloads and normalized missing fields to empty strings.
- The credential loader can return a non-working `git.uscllc.com` API token in this environment, while host-specific `~/.git-credentials` basic auth succeeds. The wrapper now falls back by host before normalization.
- `tea login list` has only `git.mosaicstack.dev` configured here; `pr-merge.sh` previously forced `--login mosaicstack`, which is invalid for `git.uscllc.com` and caused `Login name mosaicstack does not exist`.
## Changes
- `packages/mosaic/framework/tools/git/detect-platform.sh`
- Added `get_gitea_basic_auth <host>` to retrieve host-specific HTTPS credentials from `~/.git-credentials` without printing secrets.
- `packages/mosaic/framework/tools/git/pr-metadata.sh`
- Uses strict bash mode.
- Checks Gitea HTTP status and fails nonzero on API errors/non-JSON instead of emitting empty branch fields.
- Falls back from token auth to host-specific basic auth.
- Normalizes standard `head.ref`/`base.ref` and fallback branch fields.
- Requires non-empty `headRefName` and `baseRefName`.
- Preserves GitHub `gh pr view` behavior.
- `packages/mosaic/framework/tools/git/pr-merge.sh`
- Reads metadata once for base-branch policy preflight.
- Selects a `tea` login only when its configured URL matches the repo host.
- Falls back to authenticated Gitea merge API when no matching `tea` login exists, avoiding the wrong `mosaicstack` login for USC repos.
- Keeps squash-only and main-only merge policy.
- `packages/mosaic/framework/tools/git/test-pr-metadata-gitea.sh`
- Added fixture-based regression harness for standard Gitea fields, fallback branch fields, `refs/pull/<n>/head` plus `head.label` normalization, and API error payloads.
## Documentation / changelog note
This repository currently has no root `CHANGELOG.md`; the scratchpad and `docs/TASKS.md` carry the task-level change record for this wrapper fix.
## Verification log
- Red regression check: copied the new `test-pr-metadata-gitea.sh` harness next to `origin/main` wrapper scripts and ran it with `MOSAIC_TEST_WORK_DIR=$PWD/.mosaic-test-work/pr-metadata-gitea-red`; it failed as expected with `headRefName=''` and `baseRefName=''` on the fixture API-error path.
- `bash -n packages/mosaic/framework/tools/git/{detect-platform.sh,pr-metadata.sh,pr-merge.sh,test-pr-metadata-gitea.sh}`: passed.
- `shellcheck -x -P . -e SC1090 packages/mosaic/framework/tools/git/{detect-platform.sh,pr-metadata.sh,pr-merge.sh,test-pr-metadata-gitea.sh}`: passed.
- `MOSAIC_TEST_WORK_DIR=$PWD/.mosaic-test-work/pr-metadata-gitea packages/mosaic/framework/tools/git/test-pr-metadata-gitea.sh`: passed; verifies standard Gitea fields, fallback branch fields, `refs/pull/<n>/head` label normalization, and nonzero API-error handling.
- Installed wrapper parity: `/home/hermes/.config/mosaic/tools/git/{detect-platform.sh,pr-metadata.sh,pr-merge.sh}` byte-match the PR source copies after validation, so active U-Connect wrapper invocations use the same fix while source PR review runs.
- Live sanitized U-Connect metadata from `/src/uconnect` with `MOSAIC_CREDENTIALS_FILE=/src/jarvis-brain/credentials.json`:
- PR #1905: `number=1905`, `baseRefName=main`, `headRefName=edith/t_39ce717c-authentik-smoke-gate`, `state=open`, `host=git.uscllc.com`.
- PR #1908: `number=1908`, `baseRefName=main`, `headRefName=fix/t_23fa9e1d-portal-health-backend`, `state=closed`, `host=git.uscllc.com`.
- Merge preflight dry runs from installed wrappers:
- PR #1905: `Dry run: would merge PR #1905 on git.uscllc.com with authenticated Gitea API fallback (base=main, method=squash).`
- PR #1908: `Dry run: would merge PR #1908 on git.uscllc.com with authenticated Gitea API fallback (base=main, method=squash).`

View File

@@ -453,6 +453,26 @@ Initialize standard labels and the first pre-MVP milestone:
---
## Secrets Bootstrap (Required for Every New App)
Every new application MUST complete the following secrets bootstrap before deploying to any non-local environment. This is a hard gate — deployment without completed secrets bootstrap is forbidden.
### Secrets bootstrap checklist
- [ ] Vault path created: `vault kv put secret/k3s/<app>/ ...` with all required secret fields
- [ ] Required secrets listed in project README under a "Secrets architecture" section, including:
- Vault path(s) used
- All required secret keys and their purpose
- Whether the app uses ESO bridge (default) or Direct-Vault (opt-in, with justification)
- [ ] `external-secret.yaml` manifest committed to repo's `deploy/` or `k8s/` directory
- [ ] Deployment YAML references the synced k8s Secret via `secretKeyRef` (not raw env vars or `.env` files)
- [ ] App startup has schema-based validation for all required env vars (zod / pydantic / envconfig equivalent) that exits non-zero on missing required values
- [ ] Direct-Vault opt-in (if applicable): justification documented in README + AppRole provisioned + bootstrap credentials stored in Vault and synced via a separate `ExternalSecret`
See `~/.config/mosaic/guides/VAULT-SECRETS.md` for full worked examples of the ESO bridge pattern, the Direct-Vault opt-in pattern, and the forbidden antipatterns.
---
## Checklist
After bootstrapping, verify:

View File

@@ -203,3 +203,374 @@ Error: token expired
3. **Audit logging** - All access is logged; act accordingly
4. **No local copies** - Don't store secrets in files or env vars long-term
5. **Rotate on compromise** - Immediately rotate any exposed secrets
---
## Secrets Architecture Decision Matrix
Use this table to choose between the ESO bridge (default) and Direct-Vault (opt-in) patterns for every new app or integration.
| Factor | ESO Bridge (default) | Direct-Vault (opt-in) |
| --- | --- | --- |
| **Use-case** | All static secrets (DB creds, API keys, signing keys, OAuth secrets) | Dynamic creds with short TTLs (DB rotation, AWS STS, PKI), per-request audit trails, or lease renewal mid-pod-lifecycle |
| **App code change** | None — reads standard env vars via `secretKeyRef` | Requires Vault client (`hvac`, `node-vault`, `vault/api`) in application code |
| **Secret rotation** | ESO re-syncs on Vault write; pod restart or secret refresh picks up new value | App manages lease renewal or re-auth within the running process |
| **Audit granularity** | Access logged at Vault when ESO syncs; no per-request app audit | Every app request to Vault is a separate audit log entry |
| **Operational burden** | Low — ESO handles polling, sync, and k8s Secret lifecycle | Higher — app must handle auth, lease renewal, error paths, and token rotation |
| **Justification required?** | No — this is the default | Yes — document in project README under "Secrets architecture" |
| **Example use cases** | Web app DB password, OAuth client secret, JWT signing key, API token | HashiCorp DB secrets engine with 15-min TTL leases, AWS STS assume-role, Vault PKI short-lived certs |
**Decision rule:** If you are unsure, use ESO. Only justify Direct-Vault when the secret cannot be safely stored in a k8s Secret (too short-lived, per-request TTL required, or mid-lifecycle renewal needed).
---
## ESO Bridge Pattern (Default)
This is the required default for all k8s workloads. Follow this exact pattern unless a documented dynamic-secrets requirement justifies Direct-Vault.
### 1. Provision Vault path
```bash
# Write the secrets for the app (run once; use IaC/Terraform for repeatable provisioning)
vault kv put secret/k3s/<app> \
db_password="..." \
api_key="..." \
jwt_secret="..."
```
Use the canonical path structure: `secret/k3s/<app>` for k3s cluster workloads.
### 2. ExternalSecret manifest
Commit this to the repo's `deploy/` or `k8s/` directory:
```yaml
# deploy/external-secret.yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: <app>-secrets
namespace: <namespace>
spec:
refreshInterval: 1h
secretStoreRef:
name: vault-backend # ClusterSecretStore name — verify with cluster admin
kind: ClusterSecretStore
target:
name: <app>-secrets # k8s Secret name that will be created
creationPolicy: Owner
data:
- secretKey: DB_PASSWORD # key in the k8s Secret
remoteRef:
key: secret/k3s/<app> # Vault path
property: db_password # field within the Vault secret
- secretKey: API_KEY
remoteRef:
key: secret/k3s/<app>
property: api_key
- secretKey: JWT_SECRET
remoteRef:
key: secret/k3s/<app>
property: jwt_secret
```
### 3. Deployment manifest — reference synced k8s Secret
```yaml
# deploy/deployment.yaml (env section)
env:
- name: DB_PASSWORD
valueFrom:
secretKeyRef:
name: <app>-secrets # matches ExternalSecret target.name
key: DB_PASSWORD
- name: API_KEY
valueFrom:
secretKeyRef:
name: <app>-secrets
key: API_KEY
- name: JWT_SECRET
valueFrom:
secretKeyRef:
name: <app>-secrets
key: JWT_SECRET
- name: PORT
value: "3000" # safe-default: non-secret, no Vault needed
```
### 4. App-side schema validation — TypeScript (zod)
Validate all required env vars at startup. Exit non-zero on missing values.
```typescript
// src/env.ts
import { z } from 'zod';
const envSchema = z.object({
DB_PASSWORD: z.string().min(1, 'DB_PASSWORD is required'),
API_KEY: z.string().min(1, 'API_KEY is required'),
JWT_SECRET: z.string().min(32, 'JWT_SECRET must be at least 32 chars'),
PORT: z.coerce.number().default(3000),
NODE_ENV: z.enum(['development', 'production', 'test']).default('production'),
});
const result = envSchema.safeParse(process.env);
if (!result.success) {
console.error('Missing or invalid environment variables:');
console.error(result.error.flatten().fieldErrors);
process.exit(1);
}
export const env = result.data;
```
### 4b. App-side schema validation — Python (pydantic)
```python
# src/config.py
from pydantic_settings import BaseSettings, SettingsConfigDict
class Settings(BaseSettings):
db_password: str
api_key: str
jwt_secret: str
port: int = 3000
node_env: str = "production"
model_config = SettingsConfigDict(env_file=None) # no .env in prod
try:
settings = Settings()
except Exception as e:
import sys
print(f"Missing or invalid environment variables: {e}", file=sys.stderr)
sys.exit(1)
```
### 4c. App-side schema validation — Go (envconfig)
```go
// config/config.go
package config
import (
"fmt"
"github.com/kelseyhightower/envconfig"
)
type Config struct {
DBPassword string `envconfig:"DB_PASSWORD" required:"true"`
APIKey string `envconfig:"API_KEY" required:"true"`
JWTSecret string `envconfig:"JWT_SECRET" required:"true"`
Port int `envconfig:"PORT" default:"3000"`
}
func Load() (*Config, error) {
var cfg Config
if err := envconfig.Process("", &cfg); err != nil {
return nil, fmt.Errorf("invalid environment: %w", err)
}
return &cfg, nil
}
```
In your `main.go`:
```go
cfg, err := config.Load()
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
```
---
## Direct-Vault Opt-In Pattern
Use this pattern ONLY when a documented dynamic-secrets requirement applies (DB rotation with short TTLs, AWS STS, PKI, per-request audit). Document the justification in the project README under "Secrets architecture" before implementing.
### When it is justified
- Vault DB secrets engine with lease TTLs shorter than a typical pod lifecycle (< 1 hour)
- AWS STS assume-role tokens generated per-request
- Vault PKI short-lived certificates (< 24 hours) that must be renewed within a running pod
- Per-request audit trail requirement (each app call must appear separately in Vault audit log)
### Provision an AppRole for the app
```bash
# Enable AppRole auth (if not already enabled)
vault auth enable approle
# Create a Vault policy for the app
# Note: KV v2 paths require both the exact path (for the top-level secret) and the
# wildcard (for sub-paths). Always include both to avoid permission denied errors.
vault policy write <app>-policy - <<EOF
path "secret/data/k3s/<app>" {
capabilities = ["read"]
}
path "secret/data/k3s/<app>/*" {
capabilities = ["read"]
}
path "database/creds/<app>-role" {
capabilities = ["read"]
}
EOF
# Create the AppRole
vault write auth/approle/role/<app>-role \
token_policies="<app>-policy" \
token_ttl=1h \
token_max_ttl=4h \
secret_id_ttl=0
# Retrieve role-id and secret-id
vault read auth/approle/role/<app>-role/role-id
vault write -f auth/approle/role/<app>-role/secret-id
```
### Bootstrap AppRole credentials via ESO (solving the chicken-and-egg problem)
The AppRole `role-id` and `secret-id` are themselves secrets. Store them in Vault at a bootstrap path, then use ESO to sync them into a k8s Secret. The app reads that k8s Secret at startup to authenticate with Vault directly.
```bash
# Store the bootstrap credentials in Vault
vault kv put secret/k3s/<app>-bootstrap \
role_id="<role-id>" \
secret_id="<secret-id>"
```
```yaml
# deploy/external-secret-bootstrap.yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: <app>-vault-auth
namespace: <namespace>
spec:
refreshInterval: 24h
secretStoreRef:
name: vault-backend
kind: ClusterSecretStore
target:
name: <app>-vault-auth
creationPolicy: Owner
data:
- secretKey: VAULT_ROLE_ID
remoteRef:
key: secret/k3s/<app>-bootstrap
property: role_id
- secretKey: VAULT_SECRET_ID
remoteRef:
key: secret/k3s/<app>-bootstrap
property: secret_id
```
```yaml
# deploy/deployment.yaml (env section for Direct-Vault app)
env:
- name: VAULT_ADDR
value: "https://vault.example.com" # safe-default: non-secret cluster address
- name: VAULT_ROLE_ID
valueFrom:
secretKeyRef:
name: <app>-vault-auth
key: VAULT_ROLE_ID
- name: VAULT_SECRET_ID
valueFrom:
secretKeyRef:
name: <app>-vault-auth
key: VAULT_SECRET_ID
```
### App-side Vault client pattern
```typescript
// src/vault-client.ts — only exists in Direct-Vault apps
import vault from 'node-vault';
import { z } from 'zod';
const bootstrapSchema = z.object({
VAULT_ADDR: z.string().url(),
VAULT_ROLE_ID: z.string().min(1),
VAULT_SECRET_ID: z.string().min(1),
});
const bootstrap = bootstrapSchema.parse(process.env);
const client = vault({ endpoint: bootstrap.VAULT_ADDR });
export async function getVaultClient() {
const { auth } = await client.approleLogin({
role_id: bootstrap.VAULT_ROLE_ID,
secret_id: bootstrap.VAULT_SECRET_ID,
});
client.token = auth.client_token;
return client;
}
```
Document in README under "Secrets architecture": the Vault path, why Direct-Vault is required, and the lease/renewal strategy.
---
## Forbidden Patterns (CI Lint Targets)
The following patterns are forbidden in all Mosaic projects. CI lint SHOULD catch these automatically (implementation tracked separately). Agents MUST NOT introduce these patterns.
### 1. Untagged fallback defaults for required values
```yaml
# FORBIDDEN — required secret with silent fallback
environment:
- DB_PASSWORD=${DB_PASSWORD:-changeme}
- API_KEY=${API_KEY:-}
# REQUIRED — fast-fail on missing required values
environment:
- DB_PASSWORD=${DB_PASSWORD:?DB_PASSWORD is required}
- API_KEY=${API_KEY:?API_KEY is required}
# ALLOWED — true convenience default, tagged
environment:
- PORT=${PORT:-3000} # safe-default: non-secret, app works at any port
```
This applies to: `docker-compose.yml`, k8s manifests, Helm `values.yaml`, any env file committed to git.
### 2. Vault KV calls in application source code (ESO-default projects)
```python
# FORBIDDEN in ESO-default apps — direct Vault client in app source
import hvac
client = hvac.Client(url=os.environ['VAULT_ADDR'])
secret = client.secrets.kv.v2.read_secret_version(path='myapp/db')
```
ESO-default apps read env vars only. Direct-Vault clients belong only in apps with a documented dynamic-secrets justification in README.
### 3. Hardcoded secrets or API keys in committed files
```python
# FORBIDDEN — hardcoded credential
DB_PASSWORD = "supersecret123"
API_KEY = "sk-live-abc123"
```
No exceptions. CI lint must flag any string matching common secret patterns (`password`, `secret`, `api_key`, `token` assigned a literal non-env-var value).
### 4. `.env` files in production deployment paths
```
# FORBIDDEN — .env file in a production deploy path
deploy/.env
k8s/.env
docker/.env
# ALLOWED — local dev only
.env.example # template only, no real values
.env # local dev, must be in .gitignore
```
`.env` files are acceptable in local-dev contexts only and MUST be in `.gitignore`. They are forbidden in any path that a CI pipeline or production deployment process reads directly.

View File

@@ -27,6 +27,16 @@ Master/slave model:
- Do not perform destructive git/file actions without explicit instruction.
- Browser automation (Playwright, Cypress, Puppeteer) MUST run in headless mode. Never launch a visible browser — it collides with the user's display and active session.
### Secrets handling (HARD RULE)
- Vault is the canonical source-of-truth for every secret in every environment. No exceptions.
- For k8s workloads, the default read path is **External Secrets Operator → k8s Secret → env var** (`secretKeyRef`). The app reads standard env vars; no Vault client in app code.
- Direct-Vault clients in application code are **opt-in only**, justified per-app by a documented dynamic-secrets requirement (e.g., DB rotation, AWS STS). Default to ESO. Document the justification in the project's README under "Secrets architecture".
- `${VAR:-default}` fallback syntax in any deployment configuration (compose, k8s manifests, Helm values, env files committed to git) is **forbidden** for required values. Use `${VAR:?VAR is required}` to fast-fail. Defaults are allowed only for true conveniences (e.g. `${PORT:-3000}`) and MUST be tagged `# safe-default: <reason>` so a reviewer can confirm the intent.
- `.env` files in production deployment paths are **forbidden**. `.env.example` and `.env` in local-dev paths are fine.
- App startup MUST validate required secrets against a schema (zod / pydantic / equivalent) and exit non-zero on missing required values. Never run with defaulted weak fallbacks.
- New apps: bootstrap checklist (see `~/.config/mosaic/guides/BOOTSTRAP.md`) MUST include Vault path provisioning + `ExternalSecret` manifest + README declaring the Vault path and required keys.
## Session Lifecycle Contract
- Start: `scripts/agent/session-start.sh`

View File

@@ -453,6 +453,26 @@ Initialize standard labels and the first pre-MVP milestone:
---
## Secrets Bootstrap (Required for Every New App)
Every new application MUST complete the following secrets bootstrap before deploying to any non-local environment. This is a hard gate — deployment without completed secrets bootstrap is forbidden.
### Secrets bootstrap checklist
- [ ] Vault path created: `vault kv put secret/k3s/<app>/ ...` with all required secret fields
- [ ] Required secrets listed in project README under a "Secrets architecture" section, including:
- Vault path(s) used
- All required secret keys and their purpose
- Whether the app uses ESO bridge (default) or Direct-Vault (opt-in, with justification)
- [ ] `external-secret.yaml` manifest committed to repo's `deploy/` or `k8s/` directory
- [ ] Deployment YAML references the synced k8s Secret via `secretKeyRef` (not raw env vars or `.env` files)
- [ ] App startup has schema-based validation for all required env vars (zod / pydantic / envconfig equivalent) that exits non-zero on missing required values
- [ ] Direct-Vault opt-in (if applicable): justification documented in README + AppRole provisioned + bootstrap credentials stored in Vault and synced via a separate `ExternalSecret`
See `~/.config/mosaic/guides/VAULT-SECRETS.md` for full worked examples of the ESO bridge pattern, the Direct-Vault opt-in pattern, and the forbidden antipatterns.
---
## Checklist
After bootstrapping, verify:

View File

@@ -203,3 +203,374 @@ Error: token expired
3. **Audit logging** - All access is logged; act accordingly
4. **No local copies** - Don't store secrets in files or env vars long-term
5. **Rotate on compromise** - Immediately rotate any exposed secrets
---
## Secrets Architecture Decision Matrix
Use this table to choose between the ESO bridge (default) and Direct-Vault (opt-in) patterns for every new app or integration.
| Factor | ESO Bridge (default) | Direct-Vault (opt-in) |
| --- | --- | --- |
| **Use-case** | All static secrets (DB creds, API keys, signing keys, OAuth secrets) | Dynamic creds with short TTLs (DB rotation, AWS STS, PKI), per-request audit trails, or lease renewal mid-pod-lifecycle |
| **App code change** | None — reads standard env vars via `secretKeyRef` | Requires Vault client (`hvac`, `node-vault`, `vault/api`) in application code |
| **Secret rotation** | ESO re-syncs on Vault write; pod restart or secret refresh picks up new value | App manages lease renewal or re-auth within the running process |
| **Audit granularity** | Access logged at Vault when ESO syncs; no per-request app audit | Every app request to Vault is a separate audit log entry |
| **Operational burden** | Low — ESO handles polling, sync, and k8s Secret lifecycle | Higher — app must handle auth, lease renewal, error paths, and token rotation |
| **Justification required?** | No — this is the default | Yes — document in project README under "Secrets architecture" |
| **Example use cases** | Web app DB password, OAuth client secret, JWT signing key, API token | HashiCorp DB secrets engine with 15-min TTL leases, AWS STS assume-role, Vault PKI short-lived certs |
**Decision rule:** If you are unsure, use ESO. Only justify Direct-Vault when the secret cannot be safely stored in a k8s Secret (too short-lived, per-request TTL required, or mid-lifecycle renewal needed).
---
## ESO Bridge Pattern (Default)
This is the required default for all k8s workloads. Follow this exact pattern unless a documented dynamic-secrets requirement justifies Direct-Vault.
### 1. Provision Vault path
```bash
# Write the secrets for the app (run once; use IaC/Terraform for repeatable provisioning)
vault kv put secret/k3s/<app> \
db_password="..." \
api_key="..." \
jwt_secret="..."
```
Use the canonical path structure: `secret/k3s/<app>` for k3s cluster workloads.
### 2. ExternalSecret manifest
Commit this to the repo's `deploy/` or `k8s/` directory:
```yaml
# deploy/external-secret.yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: <app>-secrets
namespace: <namespace>
spec:
refreshInterval: 1h
secretStoreRef:
name: vault-backend # ClusterSecretStore name — verify with cluster admin
kind: ClusterSecretStore
target:
name: <app>-secrets # k8s Secret name that will be created
creationPolicy: Owner
data:
- secretKey: DB_PASSWORD # key in the k8s Secret
remoteRef:
key: secret/k3s/<app> # Vault path
property: db_password # field within the Vault secret
- secretKey: API_KEY
remoteRef:
key: secret/k3s/<app>
property: api_key
- secretKey: JWT_SECRET
remoteRef:
key: secret/k3s/<app>
property: jwt_secret
```
### 3. Deployment manifest — reference synced k8s Secret
```yaml
# deploy/deployment.yaml (env section)
env:
- name: DB_PASSWORD
valueFrom:
secretKeyRef:
name: <app>-secrets # matches ExternalSecret target.name
key: DB_PASSWORD
- name: API_KEY
valueFrom:
secretKeyRef:
name: <app>-secrets
key: API_KEY
- name: JWT_SECRET
valueFrom:
secretKeyRef:
name: <app>-secrets
key: JWT_SECRET
- name: PORT
value: "3000" # safe-default: non-secret, no Vault needed
```
### 4. App-side schema validation — TypeScript (zod)
Validate all required env vars at startup. Exit non-zero on missing values.
```typescript
// src/env.ts
import { z } from 'zod';
const envSchema = z.object({
DB_PASSWORD: z.string().min(1, 'DB_PASSWORD is required'),
API_KEY: z.string().min(1, 'API_KEY is required'),
JWT_SECRET: z.string().min(32, 'JWT_SECRET must be at least 32 chars'),
PORT: z.coerce.number().default(3000),
NODE_ENV: z.enum(['development', 'production', 'test']).default('production'),
});
const result = envSchema.safeParse(process.env);
if (!result.success) {
console.error('Missing or invalid environment variables:');
console.error(result.error.flatten().fieldErrors);
process.exit(1);
}
export const env = result.data;
```
### 4b. App-side schema validation — Python (pydantic)
```python
# src/config.py
from pydantic_settings import BaseSettings, SettingsConfigDict
class Settings(BaseSettings):
db_password: str
api_key: str
jwt_secret: str
port: int = 3000
node_env: str = "production"
model_config = SettingsConfigDict(env_file=None) # no .env in prod
try:
settings = Settings()
except Exception as e:
import sys
print(f"Missing or invalid environment variables: {e}", file=sys.stderr)
sys.exit(1)
```
### 4c. App-side schema validation — Go (envconfig)
```go
// config/config.go
package config
import (
"fmt"
"github.com/kelseyhightower/envconfig"
)
type Config struct {
DBPassword string `envconfig:"DB_PASSWORD" required:"true"`
APIKey string `envconfig:"API_KEY" required:"true"`
JWTSecret string `envconfig:"JWT_SECRET" required:"true"`
Port int `envconfig:"PORT" default:"3000"`
}
func Load() (*Config, error) {
var cfg Config
if err := envconfig.Process("", &cfg); err != nil {
return nil, fmt.Errorf("invalid environment: %w", err)
}
return &cfg, nil
}
```
In your `main.go`:
```go
cfg, err := config.Load()
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
```
---
## Direct-Vault Opt-In Pattern
Use this pattern ONLY when a documented dynamic-secrets requirement applies (DB rotation with short TTLs, AWS STS, PKI, per-request audit). Document the justification in the project README under "Secrets architecture" before implementing.
### When it is justified
- Vault DB secrets engine with lease TTLs shorter than a typical pod lifecycle (< 1 hour)
- AWS STS assume-role tokens generated per-request
- Vault PKI short-lived certificates (< 24 hours) that must be renewed within a running pod
- Per-request audit trail requirement (each app call must appear separately in Vault audit log)
### Provision an AppRole for the app
```bash
# Enable AppRole auth (if not already enabled)
vault auth enable approle
# Create a Vault policy for the app
# Note: KV v2 paths require both the exact path (for the top-level secret) and the
# wildcard (for sub-paths). Always include both to avoid permission denied errors.
vault policy write <app>-policy - <<EOF
path "secret/data/k3s/<app>" {
capabilities = ["read"]
}
path "secret/data/k3s/<app>/*" {
capabilities = ["read"]
}
path "database/creds/<app>-role" {
capabilities = ["read"]
}
EOF
# Create the AppRole
vault write auth/approle/role/<app>-role \
token_policies="<app>-policy" \
token_ttl=1h \
token_max_ttl=4h \
secret_id_ttl=0
# Retrieve role-id and secret-id
vault read auth/approle/role/<app>-role/role-id
vault write -f auth/approle/role/<app>-role/secret-id
```
### Bootstrap AppRole credentials via ESO (solving the chicken-and-egg problem)
The AppRole `role-id` and `secret-id` are themselves secrets. Store them in Vault at a bootstrap path, then use ESO to sync them into a k8s Secret. The app reads that k8s Secret at startup to authenticate with Vault directly.
```bash
# Store the bootstrap credentials in Vault
vault kv put secret/k3s/<app>-bootstrap \
role_id="<role-id>" \
secret_id="<secret-id>"
```
```yaml
# deploy/external-secret-bootstrap.yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: <app>-vault-auth
namespace: <namespace>
spec:
refreshInterval: 24h
secretStoreRef:
name: vault-backend
kind: ClusterSecretStore
target:
name: <app>-vault-auth
creationPolicy: Owner
data:
- secretKey: VAULT_ROLE_ID
remoteRef:
key: secret/k3s/<app>-bootstrap
property: role_id
- secretKey: VAULT_SECRET_ID
remoteRef:
key: secret/k3s/<app>-bootstrap
property: secret_id
```
```yaml
# deploy/deployment.yaml (env section for Direct-Vault app)
env:
- name: VAULT_ADDR
value: "https://vault.example.com" # safe-default: non-secret cluster address
- name: VAULT_ROLE_ID
valueFrom:
secretKeyRef:
name: <app>-vault-auth
key: VAULT_ROLE_ID
- name: VAULT_SECRET_ID
valueFrom:
secretKeyRef:
name: <app>-vault-auth
key: VAULT_SECRET_ID
```
### App-side Vault client pattern
```typescript
// src/vault-client.ts — only exists in Direct-Vault apps
import vault from 'node-vault';
import { z } from 'zod';
const bootstrapSchema = z.object({
VAULT_ADDR: z.string().url(),
VAULT_ROLE_ID: z.string().min(1),
VAULT_SECRET_ID: z.string().min(1),
});
const bootstrap = bootstrapSchema.parse(process.env);
const client = vault({ endpoint: bootstrap.VAULT_ADDR });
export async function getVaultClient() {
const { auth } = await client.approleLogin({
role_id: bootstrap.VAULT_ROLE_ID,
secret_id: bootstrap.VAULT_SECRET_ID,
});
client.token = auth.client_token;
return client;
}
```
Document in README under "Secrets architecture": the Vault path, why Direct-Vault is required, and the lease/renewal strategy.
---
## Forbidden Patterns (CI Lint Targets)
The following patterns are forbidden in all Mosaic projects. CI lint SHOULD catch these automatically (implementation tracked separately). Agents MUST NOT introduce these patterns.
### 1. Untagged fallback defaults for required values
```yaml
# FORBIDDEN — required secret with silent fallback
environment:
- DB_PASSWORD=${DB_PASSWORD:-changeme}
- API_KEY=${API_KEY:-}
# REQUIRED — fast-fail on missing required values
environment:
- DB_PASSWORD=${DB_PASSWORD:?DB_PASSWORD is required}
- API_KEY=${API_KEY:?API_KEY is required}
# ALLOWED — true convenience default, tagged
environment:
- PORT=${PORT:-3000} # safe-default: non-secret, app works at any port
```
This applies to: `docker-compose.yml`, k8s manifests, Helm `values.yaml`, any env file committed to git.
### 2. Vault KV calls in application source code (ESO-default projects)
```python
# FORBIDDEN in ESO-default apps — direct Vault client in app source
import hvac
client = hvac.Client(url=os.environ['VAULT_ADDR'])
secret = client.secrets.kv.v2.read_secret_version(path='myapp/db')
```
ESO-default apps read env vars only. Direct-Vault clients belong only in apps with a documented dynamic-secrets justification in README.
### 3. Hardcoded secrets or API keys in committed files
```python
# FORBIDDEN — hardcoded credential
DB_PASSWORD = "supersecret123"
API_KEY = "sk-live-abc123"
```
No exceptions. CI lint must flag any string matching common secret patterns (`password`, `secret`, `api_key`, `token` assigned a literal non-env-var value).
### 4. `.env` files in production deployment paths
```
# FORBIDDEN — .env file in a production deploy path
deploy/.env
k8s/.env
docker/.env
# ALLOWED — local dev only
.env.example # template only, no real values
.env # local dev, must be in .gitignore
```
`.env` files are acceptable in local-dev contexts only and MUST be in `.gitignore`. They are forbidden in any path that a CI pipeline or production deployment process reads directly.

View File

@@ -103,28 +103,16 @@ get_gitea_token() {
if [[ -f "$cred_loader" ]]; then
local token
token=$(
# shellcheck source=/dev/null
source "$cred_loader"
# Host-specific wrapper resolution must not inherit caller/global GITEA_*.
# load_credentials intentionally preserves existing env vars for interactive use,
# but metadata/merge wrappers need credentials matching the remote host.
unset GITEA_TOKEN GITEA_URL
case "$host" in
git.mosaicstack.dev) load_credentials gitea-mosaicstack 2>/dev/null ;;
git.uscllc.com) load_credentials gitea-usc 2>/dev/null ;;
*)
local matched=false
for svc in gitea-mosaicstack gitea-usc; do
unset GITEA_TOKEN GITEA_URL
load_credentials "$svc" 2>/dev/null || continue
if [[ "${GITEA_URL:-}" == "https://$host" || "${GITEA_URL:-}" == "http://$host" || "${GITEA_URL:-}" == *"//$host" ]]; then
matched=true
break
fi
done
if [[ "$matched" != true ]]; then
[[ "${GITEA_URL:-}" == *"$host"* ]] && break
unset GITEA_TOKEN GITEA_URL
fi
done
;;
esac
echo "${GITEA_TOKEN:-}"
@@ -135,12 +123,10 @@ get_gitea_token() {
fi
fi
# 2. GITEA_TOKEN env var (only when GITEA_URL, if present, matches the remote host)
# 2. GITEA_TOKEN env var (may be set by caller)
if [[ -n "${GITEA_TOKEN:-}" ]]; then
if [[ -z "${GITEA_URL:-}" || "${GITEA_URL:-}" == "https://$host" || "${GITEA_URL:-}" == "http://$host" || "${GITEA_URL:-}" == *"//$host" ]]; then
echo "$GITEA_TOKEN"
return 0
fi
echo "$GITEA_TOKEN"
return 0
fi
# 3. ~/.git-credentials file
@@ -157,37 +143,6 @@ get_gitea_token() {
return 1
}
# Resolve HTTPS basic auth credentials for a Gitea host from ~/.git-credentials.
# Prints "username:password" for direct curl -u consumption. Callers must not log it.
get_gitea_basic_auth() {
local host="$1"
local creds="$HOME/.git-credentials"
if [[ ! -f "$creds" ]]; then
return 1
fi
python3 - "$host" "$creds" <<'PY'
import sys
from pathlib import Path
from urllib.parse import unquote, urlparse
host = sys.argv[1]
creds = Path(sys.argv[2])
for line in creds.read_text(encoding="utf-8").splitlines():
parsed = urlparse(line.strip())
if parsed.hostname != host:
continue
username = unquote(parsed.username or "")
password = unquote(parsed.password or "")
if username and password:
print(f"{username}:{password}")
raise SystemExit(0)
raise SystemExit(1)
PY
}
# If script is run directly (not sourced), output the platform
if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
detect_platform

View File

@@ -2,10 +2,9 @@
# pr-merge.sh - Merge pull requests on Gitea or GitHub
# Usage: pr-merge.sh -n PR_NUMBER [-m squash] [-d] [--skip-queue-guard]
set -euo pipefail
set -e
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# shellcheck source=packages/mosaic/framework/tools/git/detect-platform.sh
source "$SCRIPT_DIR/detect-platform.sh"
# Default values
@@ -13,7 +12,6 @@ PR_NUMBER=""
MERGE_METHOD="squash"
DELETE_BRANCH=false
SKIP_QUEUE_GUARD=false
DRY_RUN=false
usage() {
cat <<EOF
@@ -26,7 +24,6 @@ Options:
-m, --method METHOD Merge method: squash only (default: squash)
-d, --delete-branch Delete the head branch after merge
--skip-queue-guard Skip CI queue guard wait before merge
--dry-run Run metadata/login preflight without merging
-h, --help Show this help message
Examples:
@@ -57,11 +54,6 @@ while [[ $# -gt 0 ]]; do
SKIP_QUEUE_GUARD=true
shift
;;
--dry-run)
DRY_RUN=true
SKIP_QUEUE_GUARD=true
shift
;;
-h|--help)
usage
;;
@@ -82,8 +74,7 @@ if [[ "$MERGE_METHOD" != "squash" ]]; then
exit 1
fi
PR_METADATA="$("$SCRIPT_DIR/pr-metadata.sh" -n "$PR_NUMBER")"
BASE_BRANCH="$(printf '%s' "$PR_METADATA" | python3 -c 'import json, sys; print((json.load(sys.stdin).get("baseRefName") or "").strip())')"
BASE_BRANCH="$("$SCRIPT_DIR/pr-metadata.sh" -n "$PR_NUMBER" | python3 -c 'import json, sys; print((json.load(sys.stdin).get("baseRefName") or "").strip())')"
if [[ "$BASE_BRANCH" != "main" ]]; then
echo "Error: Mosaic policy allows merges only for PRs targeting 'main' (found '$BASE_BRANCH')." >&2
exit 1
@@ -101,122 +92,21 @@ PLATFORM=$(detect_platform)
OWNER=$(get_repo_owner)
REPO=$(get_repo_name)
find_tea_login_for_host() {
local host="$1"
local logins_json
command -v tea >/dev/null 2>&1 || return 1
logins_json=$(tea login list --output json 2>/dev/null) || return 1
TEA_LOGINS_JSON="$logins_json" python3 - "$host" <<'PY'
import json
import os
import sys
host = sys.argv[1]
try:
logins = json.loads(os.environ.get("TEA_LOGINS_JSON", "[]"))
except Exception:
raise SystemExit(1)
for login in logins if isinstance(logins, list) else []:
url = str(login.get("url") or login.get("URL") or "")
name = str(login.get("name") or login.get("Name") or "")
if url.rstrip("/").endswith(host) and name:
print(name)
raise SystemExit(0)
raise SystemExit(1)
PY
}
merge_gitea_with_api() {
local host="$1" api_url token basic_auth body_file raw_code payload
body_file=$(mktemp)
payload='{"Do":"squash"}'
token=$(get_gitea_token "$host" || true)
if [[ -n "$token" ]]; then
raw_code=$(curl -sS -w '%{http_code}' -o "$body_file" \
-X POST \
-H "Authorization: token $token" \
-H 'Content-Type: application/json' \
-d "$payload" \
"$api_url" || true)
if [[ "$raw_code" =~ ^2 ]]; then
rm -f "$body_file"
return 0
fi
fi
basic_auth=$(get_gitea_basic_auth "$host" || true)
if [[ -n "$basic_auth" ]]; then
raw_code=$(curl -sS -w '%{http_code}' -o "$body_file" \
-X POST \
-u "$basic_auth" \
-H 'Content-Type: application/json' \
-d "$payload" \
"$api_url" || true)
if [[ "$raw_code" =~ ^2 ]]; then
rm -f "$body_file"
return 0
fi
fi
python3 - "${raw_code:-000}" "$body_file" <<'PY' >&2
import json
import sys
code, path = sys.argv[1], sys.argv[2]
try:
data = json.load(open(path, encoding="utf-8"))
message = data.get("message") or data.get("error") or "unknown API error"
except Exception:
message = open(path, encoding="utf-8", errors="replace").read()[:200] or "empty response"
print(f"Error: Gitea API merge failed with HTTP {code}: {message}")
PY
rm -f "$body_file"
return 1
}
if [[ "$DRY_RUN" == true ]]; then
if [[ "$PLATFORM" == "gitea" ]]; then
HOST=$(get_remote_host) || {
echo "Error: Cannot determine host from origin remote URL" >&2
exit 1
}
TEA_LOGIN="${GITEA_LOGIN:-$(find_tea_login_for_host "$HOST" || true)}"
if [[ -n "$TEA_LOGIN" ]]; then
echo "Dry run: would merge PR #$PR_NUMBER on $HOST with tea login '$TEA_LOGIN' (base=$BASE_BRANCH, method=squash)."
else
echo "Dry run: would merge PR #$PR_NUMBER on $HOST with authenticated Gitea API fallback (base=$BASE_BRANCH, method=squash)."
fi
else
echo "Dry run: would merge PR #$PR_NUMBER on $PLATFORM (base=$BASE_BRANCH, method=squash)."
fi
exit 0
fi
case "$PLATFORM" in
github)
GH_ARGS=(pr merge "$PR_NUMBER" --squash)
[[ "$DELETE_BRANCH" == true ]] && GH_ARGS+=(--delete-branch)
gh "${GH_ARGS[@]}"
CMD="gh pr merge $PR_NUMBER --squash"
[[ "$DELETE_BRANCH" == true ]] && CMD="$CMD --delete-branch"
eval "$CMD"
;;
gitea)
HOST=$(get_remote_host) || {
echo "Error: Cannot determine host from origin remote URL" >&2
exit 1
}
TEA_LOGIN="${GITEA_LOGIN:-$(find_tea_login_for_host "$HOST" || true)}"
if [[ -n "$TEA_LOGIN" ]]; then
tea pr merge "$PR_NUMBER" --style squash --repo "$OWNER/$REPO" --login "$TEA_LOGIN"
else
echo "No tea login configured for $HOST; using authenticated Gitea API merge fallback." >&2
merge_gitea_with_api "$HOST" "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/pulls/${PR_NUMBER}/merge"
fi
CMD="tea pr merge $PR_NUMBER --style squash --repo $OWNER/$REPO --login ${GITEA_LOGIN:-mosaicstack}"
# Delete branch after merge if requested
if [[ "$DELETE_BRANCH" == true ]]; then
echo "Note: Branch deletion after merge may need to be done separately with tea" >&2
fi
eval "$CMD"
;;
*)
echo "Error: Could not detect git platform" >&2

View File

@@ -2,10 +2,9 @@
# pr-metadata.sh - Get PR metadata as JSON on GitHub or Gitea
# Usage: pr-metadata.sh -n <pr_number> [-o <output_file>]
set -euo pipefail
set -e
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# shellcheck source=packages/mosaic/framework/tools/git/detect-platform.sh
source "$SCRIPT_DIR/detect-platform.sh"
# Parse arguments
@@ -32,7 +31,7 @@ while [[ $# -gt 0 ]]; do
exit 0
;;
*)
echo "Unknown option: $1" >&2
echo "Unknown option: $1"
exit 1
;;
esac
@@ -43,168 +42,56 @@ if [[ -z "$PR_NUMBER" ]]; then
exit 1
fi
write_metadata() {
local metadata="$1"
if [[ -n "$OUTPUT_FILE" ]]; then
printf '%s\n' "$metadata" > "$OUTPUT_FILE"
else
printf '%s\n' "$metadata"
fi
}
curl_gitea_pull() {
local api_url="$1"
local token basic_auth raw_code body_file http_code
body_file=$(mktemp)
token=$(get_gitea_token "$HOST" || true)
if [[ -n "$token" ]]; then
raw_code=$(curl -sS -w '%{http_code}' -o "$body_file" -H "Authorization: token $token" "$api_url" || true)
if [[ "$raw_code" =~ ^2 ]]; then
cat "$body_file"
rm -f "$body_file"
return 0
fi
http_code="$raw_code"
fi
basic_auth=$(get_gitea_basic_auth "$HOST" || true)
if [[ -n "$basic_auth" ]]; then
raw_code=$(curl -sS -w '%{http_code}' -o "$body_file" -u "$basic_auth" "$api_url" || true)
if [[ "$raw_code" =~ ^2 ]]; then
cat "$body_file"
rm -f "$body_file"
return 0
fi
http_code="$raw_code"
fi
if [[ -z "${http_code:-}" ]]; then
raw_code=$(curl -sS -w '%{http_code}' -o "$body_file" "$api_url" || true)
http_code="$raw_code"
fi
python3 - "$http_code" "$body_file" <<'PY' >&2
import json
import sys
code, path = sys.argv[1], sys.argv[2]
try:
data = json.load(open(path, encoding="utf-8"))
message = data.get("message") or data.get("error") or "unknown API error"
except Exception:
message = open(path, encoding="utf-8", errors="replace").read()[:200] or "empty response"
print(f"Error: Gitea pull request API request failed with HTTP {code}: {message}")
PY
rm -f "$body_file"
return 1
}
detect_platform > /dev/null
if [[ "$PLATFORM" == "github" ]]; then
METADATA=$(gh pr view "$PR_NUMBER" --json number,title,body,state,author,headRefName,baseRefName,files,labels,assignees,milestone,createdAt,updatedAt,url,isDraft)
write_metadata "$METADATA"
if [[ -n "$OUTPUT_FILE" ]]; then
echo "$METADATA" > "$OUTPUT_FILE"
else
echo "$METADATA"
fi
elif [[ "$PLATFORM" == "gitea" ]]; then
OWNER=$(get_repo_owner)
REPO=$(get_repo_name)
HOST=$(get_remote_host) || {
echo "Error: Cannot determine host from origin remote URL" >&2
exit 1
}
REMOTE_URL=$(git remote get-url origin 2>/dev/null)
API_URL="https://${HOST}/api/v1/repos/${OWNER}/${REPO}/pulls/${PR_NUMBER}"
if [[ -n "${MOSAIC_GITEA_PR_METADATA_RAW_FILE:-}" ]]; then
RAW=$(cat "$MOSAIC_GITEA_PR_METADATA_RAW_FILE")
# Extract host from remote URL
if [[ "$REMOTE_URL" == https://* ]]; then
HOST=$(echo "$REMOTE_URL" | sed -E 's|https://([^/]+)/.*|\1|')
elif [[ "$REMOTE_URL" == git@* ]]; then
HOST=$(echo "$REMOTE_URL" | sed -E 's|git@([^:]+):.*|\1|')
else
RAW=$(curl_gitea_pull "$API_URL")
echo "Error: Cannot determine host from remote URL" >&2
exit 1
fi
# Normalize Gitea response to match GitHub's expected metadata schema.
METADATA=$(printf '%s' "$RAW" | python3 -c "
import json
import sys
API_URL="https://${HOST}/api/v1/repos/${OWNER}/${REPO}/pulls/${PR_NUMBER}"
def first_non_empty(*values):
for value in values:
if value is None:
continue
if isinstance(value, str):
value = value.strip()
if value:
return value
return ''
GITEA_API_TOKEN=$(get_gitea_token "$HOST" || true)
def nested(data, *keys):
current = data
for key in keys:
if not isinstance(current, dict):
return None
current = current.get(key)
return current
try:
data = json.load(sys.stdin)
except json.JSONDecodeError as exc:
print(f'Error: Gitea API returned non-JSON response: {exc}', file=sys.stderr)
sys.exit(1)
if not isinstance(data, dict):
print('Error: Gitea API returned an unexpected non-object response', file=sys.stderr)
sys.exit(1)
if data.get('message') and not data.get('number'):
print(f\"Error: Gitea API error: {data.get('message')}\", file=sys.stderr)
sys.exit(1)
head_ref = first_non_empty(
nested(data, 'head', 'ref'),
nested(data, 'head', 'name'),
nested(data, 'head', 'branch'),
data.get('head_branch'),
data.get('head_ref'),
nested(data, 'head', 'label'),
data.get('head_label'),
)
if isinstance(head_ref, str) and head_ref.startswith('refs/pull/'):
head_ref = first_non_empty(
nested(data, 'head', 'label'),
data.get('head_label'),
nested(data, 'head', 'name'),
nested(data, 'head', 'branch'),
data.get('head_branch'),
data.get('head_ref'),
head_ref,
)
base_ref = first_non_empty(
nested(data, 'base', 'ref'),
nested(data, 'base', 'name'),
nested(data, 'base', 'branch'),
data.get('base_branch'),
data.get('base_ref'),
data.get('base_label'),
)
if not head_ref or not base_ref:
available = ', '.join(sorted(data.keys()))
print(
'Error: Unable to resolve non-empty Gitea PR head/base refs '
f'(headRefName={head_ref!r}, baseRefName={base_ref!r}; keys={available})',
file=sys.stderr,
)
sys.exit(1)
if [[ -n "$GITEA_API_TOKEN" ]]; then
RAW=$(curl -sS -H "Authorization: token $GITEA_API_TOKEN" "$API_URL")
else
RAW=$(curl -sS "$API_URL")
fi
# Normalize Gitea response to match our expected schema
METADATA=$(echo "$RAW" | python3 -c "
import json, sys
data = json.load(sys.stdin)
normalized = {
'number': data.get('number'),
'title': data.get('title'),
'body': data.get('body', ''),
'state': data.get('state'),
'author': nested(data, 'user', 'login') or '',
'headRefName': head_ref,
'baseRefName': base_ref,
'labels': [l.get('name', '') for l in data.get('labels', []) if isinstance(l, dict)],
'assignees': [a.get('login', '') for a in data.get('assignees', []) if isinstance(a, dict)],
'milestone': nested(data, 'milestone', 'title') or '',
'author': data.get('user', {}).get('login', ''),
'headRefName': data.get('head', {}).get('ref', ''),
'baseRefName': data.get('base', {}).get('ref', ''),
'labels': [l.get('name', '') for l in data.get('labels', [])],
'assignees': [a.get('login', '') for a in data.get('assignees', [])],
'milestone': data.get('milestone', {}).get('title', '') if data.get('milestone') else '',
'createdAt': data.get('created_at', ''),
'updatedAt': data.get('updated_at', ''),
'url': data.get('html_url', ''),
@@ -215,7 +102,11 @@ normalized = {
json.dump(normalized, sys.stdout, indent=2)
")
write_metadata "$METADATA"
if [[ -n "$OUTPUT_FILE" ]]; then
echo "$METADATA" > "$OUTPUT_FILE"
else
echo "$METADATA"
fi
else
echo "Error: Unknown platform" >&2
exit 1

View File

@@ -1,87 +0,0 @@
#!/usr/bin/env bash
# Regression harness for Gitea PR metadata normalization.
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
WORK_DIR="${MOSAIC_TEST_WORK_DIR:-$PWD/.mosaic-test-work/pr-metadata-gitea}"
REPO_DIR="$WORK_DIR/repo"
FIXTURE_DIR="$WORK_DIR/fixtures"
rm -rf "$WORK_DIR"
mkdir -p "$REPO_DIR" "$FIXTURE_DIR"
git -C "$REPO_DIR" init -q
git -C "$REPO_DIR" remote add origin https://git.uscllc.com/USC/uconnect.git
cat > "$FIXTURE_DIR/gitea-standard.json" <<'JSON'
{
"number": 1905,
"title": "Smoke gate fix",
"state": "open",
"user": {"login": "edith"},
"head": {"ref": "edith/t_39ce717c-authentik-smoke-gate"},
"base": {"ref": "main"},
"labels": [{"name": "ci"}],
"assignees": [{"login": "edith"}],
"html_url": "https://git.uscllc.com/USC/uconnect/pulls/1905"
}
JSON
cat > "$FIXTURE_DIR/gitea-fallback.json" <<'JSON'
{
"number": 1908,
"title": "Fallback branch fields",
"state": "open",
"user": {"login": "edith"},
"head_branch": "fix/fallback-head",
"base_branch": "main",
"html_url": "https://git.uscllc.com/USC/uconnect/pulls/1908"
}
JSON
cat > "$FIXTURE_DIR/gitea-refs-pull-label.json" <<'JSON'
{
"number": 1908,
"title": "Closed merged PR with synthetic pull ref",
"state": "closed",
"user": {"login": "edith"},
"head": {"ref": "refs/pull/1908/head", "label": "fix/t_23fa9e1d-portal-health-backend"},
"base": {"ref": "main"},
"html_url": "https://git.uscllc.com/USC/uconnect/pulls/1908"
}
JSON
cat > "$FIXTURE_DIR/gitea-error.json" <<'JSON'
{"message": "user does not exist [uid: 0, name: ]", "url": "https://git.uscllc.com/api/swagger"}
JSON
run_case() {
local fixture="$1" expected_number="$2" expected_head="$3"
local output
output=$(cd "$REPO_DIR" && MOSAIC_GITEA_PR_METADATA_RAW_FILE="$fixture" "$SCRIPT_DIR/pr-metadata.sh" -n "$expected_number")
PR_METADATA_OUTPUT="$output" python3 - "$expected_number" "$expected_head" <<'PY'
import json
import os
import sys
data = json.loads(os.environ["PR_METADATA_OUTPUT"])
expected_number = int(sys.argv[1])
expected_head = sys.argv[2]
assert data["number"] == expected_number, data
assert data["baseRefName"] == "main", data
assert data["headRefName"] == expected_head, data
PY
}
run_case "$FIXTURE_DIR/gitea-standard.json" 1905 edith/t_39ce717c-authentik-smoke-gate
run_case "$FIXTURE_DIR/gitea-fallback.json" 1908 fix/fallback-head
run_case "$FIXTURE_DIR/gitea-refs-pull-label.json" 1908 fix/t_23fa9e1d-portal-health-backend
if cd "$REPO_DIR" && MOSAIC_GITEA_PR_METADATA_RAW_FILE="$FIXTURE_DIR/gitea-error.json" "$SCRIPT_DIR/pr-metadata.sh" -n 1909 >/dev/null 2>"$WORK_DIR/error.log"; then
echo "Expected API error fixture to fail" >&2
exit 1
fi
grep -q "Gitea API error" "$WORK_DIR/error.log"
echo "Gitea PR metadata regression harness passed"