mosaic/bootstrap
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

chore: sync local Mosaic changes

Browse Source
This commit is contained in:
Jason Woltje
2026-02-21 09:55:34 -06:00
parent 1e4eefeca3
commit e3ec3e32e5
82 changed files with 5398 additions and 1969 deletions
Download Patch File Download Diff File Expand all files Collapse all files

253
guides/ci-cd-pipelines.md
View File

@@ -12,7 +12,7 @@ GIT PUSH
QUALITY GATES (lint, typecheck, test, audit)
↓ all pass
BUILD (compile all packages)
↓ only on main/develop/tags
↓ only on main/tags
DOCKER BUILD & PUSH (Kaniko → Gitea Container Registry)
↓ all images pushed
PACKAGE LINKING (associate images with repository in Gitea)
@@ -123,10 +123,10 @@ when:
# Top-level: run quality gates on everything
- event: [push, pull_request, manual]
# Per-step: only build Docker images on main/develop/tags
# Per-step: only build Docker images on main/tags
docker-build-api:
when:
- branch: [main, develop]
- branch: [main]
event: [push, manual, tag]
```
@@ -150,24 +150,29 @@ docker-build-SERVICE:
from_secret: gitea_username
GITEA_TOKEN:
from_secret: gitea_token
RELEASE_BASE_VERSION: ${RELEASE_BASE_VERSION}
CI_COMMIT_BRANCH: ${CI_COMMIT_BRANCH}
CI_COMMIT_TAG: ${CI_COMMIT_TAG}
CI_COMMIT_SHA: ${CI_COMMIT_SHA}
CI_PIPELINE_NUMBER: ${CI_PIPELINE_NUMBER}
commands:
- *kaniko_setup
- |
DESTINATIONS="--destination REGISTRY/ORG/IMAGE_NAME:${CI_COMMIT_SHA:0:8}"
SHORT_SHA="${CI_COMMIT_SHA:0:8}"
BUILD_ID="${CI_PIPELINE_NUMBER:-$SHORT_SHA}"
BASE_VERSION="${RELEASE_BASE_VERSION:?RELEASE_BASE_VERSION is required (example: 0.0.1)}"
DESTINATIONS="--destination REGISTRY/ORG/IMAGE_NAME:sha-$SHORT_SHA"
if [ "$CI_COMMIT_BRANCH" = "main" ]; then
DESTINATIONS="$DESTINATIONS --destination REGISTRY/ORG/IMAGE_NAME:latest"
elif [ "$CI_COMMIT_BRANCH" = "develop" ]; then
DESTINATIONS="$DESTINATIONS --destination REGISTRY/ORG/IMAGE_NAME:dev"
DESTINATIONS="$DESTINATIONS --destination REGISTRY/ORG/IMAGE_NAME:v${BASE_VERSION}-rc.${BUILD_ID}"
DESTINATIONS="$DESTINATIONS --destination REGISTRY/ORG/IMAGE_NAME:testing"
fi
if [ -n "$CI_COMMIT_TAG" ]; then
DESTINATIONS="$DESTINATIONS --destination REGISTRY/ORG/IMAGE_NAME:$CI_COMMIT_TAG"
fi
/kaniko/executor --context . --dockerfile PATH/TO/Dockerfile $DESTINATIONS
when:
- branch: [main, develop]
- branch: [main]
event: [push, manual, tag]
depends_on:
- build
@@ -184,15 +189,69 @@ docker-build-SERVICE:
### Image Tagging Strategy
Every build produces multiple tags:
Tagging MUST follow a two-layer model: immutable identity tags + mutable environment tags.
Immutable tags:
| Condition | Tag | Purpose |
|-----------|-----|---------|
| Always | `${CI_COMMIT_SHA:0:8}` | Immutable reference to exact commit |
| `main` branch | `latest` | Current production release |
| `develop` branch | `dev` | Current development build |
| Always | `sha-${CI_COMMIT_SHA:0:8}` | Immutable reference to exact commit |
| `main` branch | `v{BASE_VERSION}-rc.{BUILD_ID}` | Intermediate release candidate for the active milestone |
| Git tag (e.g., `v1.0.0`) | `v1.0.0` | Semantic version release |
Mutable environment tags:
| Tag | Purpose |
|-----|---------|
| `testing` | Current candidate under situational validation |
| `staging` (optional) | Pre-production validation target |
| `prod` | Current production pointer |
Hard rules:
- Do NOT use `latest` for deployment.
- Do NOT use `dev` as the primary deployment tag.
- Deployments MUST resolve to an immutable image digest.
### Digest-First Promotion (Hard Rule)
Deploy and promote by digest, not by mutable tag:
1. Build and push candidate tags (`sha-*`, `vX.Y.Z-rc.N`, `testing`).
2. Resolve the digest from `sha-*` tag.
3. Deploy that digest to testing and run situational tests.
4. If green, promote the same digest to `staging`/`prod` tags.
5. Create final semantic release tag (`vX.Y.Z`) only at milestone completion.
Example with `crane`:
```bash
DIGEST=$(crane digest REGISTRY/ORG/IMAGE:sha-${CI_COMMIT_SHA:0:8})
crane tag REGISTRY/ORG/IMAGE@${DIGEST} testing
# after situational tests pass:
crane tag REGISTRY/ORG/IMAGE@${DIGEST} prod
```
### Deployment Strategy: Blue-Green Default
- Blue-green is the default release strategy for lights-out operation.
- Canary is OPTIONAL and allowed only when automated SLO/error-rate monitoring and rollback triggers are configured.
- If canary guardrails are missing, you MUST use blue-green.
### Image Retention and Cleanup (Hard Rule)
Registry cleanup MUST be automated (daily or weekly job).
Retention policy:
- Keep all final release tags (`vX.Y.Z`) indefinitely.
- Keep digests currently referenced by `prod` and `testing` tags.
- Keep the most recent 20 RC tags (`vX.Y.Z-rc.N`) per service.
- Delete RC and `sha-*` tags older than 30 days when they are not referenced by active environments/releases.
Before deleting any image/tag:
- Verify digest is not currently deployed.
- Verify digest is not referenced by any active release/tag notes.
- Log cleanup actions in CI job output.
### Kaniko Options
Common flags for `/kaniko/executor`:
@@ -252,7 +311,10 @@ In `docker-compose.yml`:
```yaml
services:
api:
image: git.example.com/org/image:${IMAGE_TAG:-dev}
# Preferred: pin digest produced by CI and promoted by environment
image: git.example.com/org/image@${IMAGE_DIGEST}
# Optional channel pointer for non-prod:
# image: git.example.com/org/image:${IMAGE_TAG:-testing}
```
## Package Linking
@@ -315,7 +377,7 @@ link-packages:
link_package "image-name-1"
link_package "image-name-2"
when:
- branch: [main, develop]
- branch: [main]
event: [push, manual, tag]
depends_on:
- docker-build-image-1
@@ -350,6 +412,12 @@ Configure these in the Woodpecker UI (Settings > Secrets) or via CLI:
| `gitea_username` | Gitea username or service account | `push`, `manual`, `tag` |
| `gitea_token` | Gitea token with `package:write` scope | `push`, `manual`, `tag` |
### Required CI Variables (Non-Secret)
| Variable | Example | Purpose |
|----------|---------|---------|
| `RELEASE_BASE_VERSION` | `0.0.1` | Base milestone version used to generate RC tags (`v0.0.1-rc.N`) |
### Setting Secrets via CLI
```bash
@@ -554,9 +622,9 @@ security-trivy:
mkdir -p ~/.docker
echo "{\"auths\":{\"REGISTRY\":{\"username\":\"$$GITEA_USER\",\"password\":\"$$GITEA_TOKEN\"}}}" > ~/.docker/config.json
trivy image --exit-code 1 --severity HIGH,CRITICAL --ignore-unfixed \
REGISTRY/ORG/IMAGE:$${CI_COMMIT_SHA:0:8}
REGISTRY/ORG/IMAGE:sha-$${CI_COMMIT_SHA:0:8}
when:
- branch: [main, develop]
- branch: [main]
event: [push, manual, tag]
depends_on:
- docker-build-SERVICE
@@ -574,7 +642,7 @@ Docker build MUST depend on ALL quality + security steps. Trivy runs AFTER build
## Monorepo Considerations
### pnpm + Turbo (Mosaic Stack pattern)
### pnpm + Turbo
```yaml
variables:
@@ -589,7 +657,7 @@ steps:
- pnpm build # Turbo handles dependency order and caching
```
### npm Workspaces (U-Connect pattern)
### npm Workspaces
```yaml
variables:
@@ -684,24 +752,28 @@ steps:
from_secret: gitea_username
GITEA_TOKEN:
from_secret: gitea_token
RELEASE_BASE_VERSION: ${RELEASE_BASE_VERSION}
CI_COMMIT_BRANCH: ${CI_COMMIT_BRANCH}
CI_COMMIT_TAG: ${CI_COMMIT_TAG}
CI_COMMIT_SHA: ${CI_COMMIT_SHA}
CI_PIPELINE_NUMBER: ${CI_PIPELINE_NUMBER}
commands:
- *kaniko_setup
- |
DESTINATIONS="--destination git.example.com/org/api:${CI_COMMIT_SHA:0:8}"
SHORT_SHA="${CI_COMMIT_SHA:0:8}"
BUILD_ID="${CI_PIPELINE_NUMBER:-$SHORT_SHA}"
BASE_VERSION="${RELEASE_BASE_VERSION:?RELEASE_BASE_VERSION is required}"
DESTINATIONS="--destination git.example.com/org/api:sha-$SHORT_SHA"
if [ "$CI_COMMIT_BRANCH" = "main" ]; then
DESTINATIONS="$DESTINATIONS --destination git.example.com/org/api:latest"
elif [ "$CI_COMMIT_BRANCH" = "develop" ]; then
DESTINATIONS="$DESTINATIONS --destination git.example.com/org/api:dev"
DESTINATIONS="$DESTINATIONS --destination git.example.com/org/api:v${BASE_VERSION}-rc.${BUILD_ID}"
DESTINATIONS="$DESTINATIONS --destination git.example.com/org/api:testing"
fi
if [ -n "$CI_COMMIT_TAG" ]; then
DESTINATIONS="$DESTINATIONS --destination git.example.com/org/api:$CI_COMMIT_TAG"
fi
/kaniko/executor --context . --dockerfile src/api/Dockerfile $DESTINATIONS
when:
- branch: [main, develop]
- branch: [main]
event: [push, manual, tag]
depends_on: [build]
@@ -712,24 +784,28 @@ steps:
from_secret: gitea_username
GITEA_TOKEN:
from_secret: gitea_token
RELEASE_BASE_VERSION: ${RELEASE_BASE_VERSION}
CI_COMMIT_BRANCH: ${CI_COMMIT_BRANCH}
CI_COMMIT_TAG: ${CI_COMMIT_TAG}
CI_COMMIT_SHA: ${CI_COMMIT_SHA}
CI_PIPELINE_NUMBER: ${CI_PIPELINE_NUMBER}
commands:
- *kaniko_setup
- |
DESTINATIONS="--destination git.example.com/org/web:${CI_COMMIT_SHA:0:8}"
SHORT_SHA="${CI_COMMIT_SHA:0:8}"
BUILD_ID="${CI_PIPELINE_NUMBER:-$SHORT_SHA}"
BASE_VERSION="${RELEASE_BASE_VERSION:?RELEASE_BASE_VERSION is required}"
DESTINATIONS="--destination git.example.com/org/web:sha-$SHORT_SHA"
if [ "$CI_COMMIT_BRANCH" = "main" ]; then
DESTINATIONS="$DESTINATIONS --destination git.example.com/org/web:latest"
elif [ "$CI_COMMIT_BRANCH" = "develop" ]; then
DESTINATIONS="$DESTINATIONS --destination git.example.com/org/web:dev"
DESTINATIONS="$DESTINATIONS --destination git.example.com/org/web:v${BASE_VERSION}-rc.${BUILD_ID}"
DESTINATIONS="$DESTINATIONS --destination git.example.com/org/web:testing"
fi
if [ -n "$CI_COMMIT_TAG" ]; then
DESTINATIONS="$DESTINATIONS --destination git.example.com/org/web:$CI_COMMIT_TAG"
fi
/kaniko/executor --context . --dockerfile src/web/Dockerfile $DESTINATIONS
when:
- branch: [main, develop]
- branch: [main]
event: [push, manual, tag]
depends_on: [build]
@@ -763,8 +839,8 @@ steps:
}
link_package "api"
link_package "web"
when:
- branch: [main, develop]
when:
- branch: [main]
event: [push, manual, tag]
depends_on:
- docker-build-api
@@ -780,49 +856,130 @@ steps:
5. **Add package linking step** after all Docker builds
6. **Update `docker-compose.yml`** to reference registry images instead of local builds:
```yaml
image: git.example.com/org/service:${IMAGE_TAG:-dev}
image: git.example.com/org/service@${IMAGE_DIGEST}
```
7. **Test on develop branch first** — push a small change and verify the pipeline
7. **Test on a short-lived non-main branch first** — open a PR and verify quality gates before merging to `main`
8. **Verify images appear** in Gitea Packages tab after successful pipeline
## Post-Merge CI Monitoring (Hard Rule)
For source-code delivery, completion is not allowed at "PR opened" stage.
Required sequence:
1. Merge PR to `main` (squash) via Mosaic wrapper.
2. Monitor CI to terminal status:
```bash
~/.config/mosaic/rails/git/pr-ci-wait.sh -n <PR_NUMBER>
```
3. Require green status before claiming completion.
4. If CI fails, create remediation task(s) and continue until green.
5. If monitoring command fails, report blocker with the exact failed wrapper command and stop.
Woodpecker note:
- In Gitea + Woodpecker environments, commit status contexts generally reflect Woodpecker pipeline results.
- Always include CI run/status evidence in completion report.
## Queue Guard Before Push/Merge (Hard Rule)
Before pushing a branch or merging a PR, guard against overlapping project pipelines:
```bash
~/.config/mosaic/rails/git/ci-queue-wait.sh --purpose push -B main
~/.config/mosaic/rails/git/ci-queue-wait.sh --purpose merge -B main
```
Behavior:
- If pipeline state is running/queued/pending, wait until queue clears.
- If timeout or API/auth failure occurs, treat as `blocked`, report exact failed wrapper command, and stop.
## Gitea as Unified Platform
Gitea provides **three services in one**, eliminating the need for separate Harbor and Verdaccio deployments:
Gitea provides **multiple services in one**, eliminating the need for separate registry platforms:
| Service | What Gitea Replaces | Registry URL |
|---------|---------------------|-------------|
| **Git hosting** | GitHub/GitLab | `https://GITEA_HOST/org/repo` |
| **Container registry** | Harbor, Docker Hub | `docker pull GITEA_HOST/org/image:tag` |
| **npm registry** | Verdaccio, Artifactory | `https://GITEA_HOST/api/packages/org/npm/` |
| **PyPI registry** | Private PyPI/Artifactory | `https://GITEA_HOST/api/packages/org/pypi` |
| **Maven registry** | Nexus, Artifactory | `https://GITEA_HOST/api/packages/org/maven` |
| **NuGet registry** | Azure Artifacts, Artifactory | `https://GITEA_HOST/api/packages/org/nuget/index.json` |
| **Cargo registry** | crates.io mirrors, Artifactory | `https://GITEA_HOST/api/packages/org/cargo` |
| **Composer registry** | Private Packagist, Artifactory | `https://GITEA_HOST/api/packages/org/composer` |
| **Conan registry** | Artifactory Conan | `https://GITEA_HOST/api/packages/org/conan` |
| **Conda registry** | Anaconda Server, Artifactory | `https://GITEA_HOST/api/packages/org/conda` |
| **Generic registry** | Generic binary stores | `https://GITEA_HOST/api/packages/org/generic` |
### Additional Package Types
Gitea also supports PyPI, Maven, NuGet, Cargo, Composer, Conan, Conda, Generic, and more. All use the same token authentication.
### Single Token, Three Services
### Single Token, Multiple Services
A Gitea token with `package:write` scope handles:
- `git push` / `git pull`
- `docker push` / `docker pull` (container registry)
- `npm publish` / `npm install` (npm registry)
- `twine upload` / `pip install` (PyPI registry)
- package operations for Maven/NuGet/Cargo/Composer/Conan/Conda/Generic registries
This means a single `gitea_token` secret in Woodpecker CI covers all CI/CD operations.
This means a single `gitea_token` secret in Woodpecker CI covers all CI/CD package operations.
## Python Packages on Gitea PyPI
For Python libraries and internal packages, use Gitea's built-in PyPI registry.
### Publish (Local or CI)
```bash
python -m pip install --upgrade build twine
python -m build
python -m twine upload \
--repository-url "https://GITEA_HOST/api/packages/ORG/pypi" \
--username "$GITEA_USERNAME" \
--password "$GITEA_TOKEN" \
dist/*
```
### Install (Consumer Projects)
```bash
pip install \
--extra-index-url "https://$GITEA_USERNAME:$GITEA_TOKEN@GITEA_HOST/api/packages/ORG/pypi/simple" \
your-package-name
```
### Woodpecker Step (Python Publish)
```yaml
publish-python-package:
image: python:3.12-slim
environment:
GITEA_USERNAME:
from_secret: gitea_username
GITEA_TOKEN:
from_secret: gitea_token
commands:
- python -m pip install --upgrade build twine
- python -m build
- python -m twine upload --repository-url https://GITEA_HOST/api/packages/ORG/pypi --username "$$GITEA_USERNAME" --password "$$GITEA_TOKEN" dist/*
when:
branch: [main]
event: [push]
```
### Architecture Simplification
**Before (3 services):**
**Before (4 services):**
```
Gitea (git) + Harbor (containers) + Verdaccio (npm)
↓ separate auth ↓ separate auth ↓ OAuth/Authentik
3 tokens 1 robot account 1 OIDC integration
3 backup targets complex RBAC group-based access
Gitea (git) + Harbor (containers) + Verdaccio (npm) + Private PyPI
↓ separate auth ↓ separate auth ↓ extra auth ↓ extra auth
multiple tokens robot/service users npm-specific token pip/twine token
fragmented access fragmented RBAC fragmented RBAC fragmented RBAC
```
**After (1 service):**
```
Gitea (git + containers + npm)
↓ single token
1 secret in Woodpecker
Gitea (git + containers + npm + pypi)
↓ unified secrets
1 credentials model in CI
1 backup target
unified RBAC via Gitea teams
```
@@ -900,5 +1057,5 @@ If a project currently uses Verdaccio (e.g., U-Connect at `npm.uscllc.net`), fol
- Link manually: Gitea UI > Packages > Select package > Link to repository
### Pipeline runs Docker builds on pull requests
- Verify `when` clause on Docker build steps restricts to `branch: [main, develop]`
- Verify `when` clause on Docker build steps restricts to `branch: [main]`
- Pull requests should only run quality gates, not build/push images