Compare commits
4 Commits
0af3e218a1
...
fix/t-a292
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
a665c0ac4c | ||
| 755df9079e | |||
| ac5650d9f9 | |||
| bd83f86740 |
@@ -1,8 +1,21 @@
|
||||
import { mkdirSync } from 'node:fs';
|
||||
import { homedir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import { Global, Inject, Module, type OnApplicationShutdown } from '@nestjs/common';
|
||||
import { createDb, createPgliteDb, type Db, type DbHandle } from '@mosaicstack/db';
|
||||
import {
|
||||
Global,
|
||||
Inject,
|
||||
Logger,
|
||||
Module,
|
||||
type OnApplicationShutdown,
|
||||
type OnModuleInit,
|
||||
} from '@nestjs/common';
|
||||
import {
|
||||
createDb,
|
||||
createPgliteDb,
|
||||
runPgliteMigrations,
|
||||
type Db,
|
||||
type DbHandle,
|
||||
} from '@mosaicstack/db';
|
||||
import { createStorageAdapter, type StorageAdapter } from '@mosaicstack/storage';
|
||||
import type { MosaicConfig } from '@mosaicstack/config';
|
||||
import { MOSAIC_CONFIG } from '../config/config.module.js';
|
||||
@@ -39,12 +52,37 @@ export const STORAGE_ADAPTER = 'STORAGE_ADAPTER';
|
||||
],
|
||||
exports: [DB, STORAGE_ADAPTER],
|
||||
})
|
||||
export class DatabaseModule implements OnApplicationShutdown {
|
||||
export class DatabaseModule implements OnApplicationShutdown, OnModuleInit {
|
||||
private readonly logger = new Logger(DatabaseModule.name);
|
||||
|
||||
constructor(
|
||||
@Inject(DB_HANDLE) private readonly handle: DbHandle,
|
||||
@Inject(STORAGE_ADAPTER) private readonly storageAdapter: StorageAdapter,
|
||||
@Inject(MOSAIC_CONFIG) private readonly config: MosaicConfig,
|
||||
) {}
|
||||
|
||||
// Migrations must complete before any module that injects DB starts serving
|
||||
// requests. NestJS awaits onModuleInit before app.listen(), and modules that
|
||||
// inject DB are initialized after this one — so all DB-dependent code sees a
|
||||
// populated schema before the first HTTP request lands.
|
||||
//
|
||||
// Local (PGlite) tier: we run gateway-DB migrations explicitly here. The
|
||||
// storage adapter writes to a separate PGlite directory and only manages its
|
||||
// own KV tables, so we still call its migrate() afterwards.
|
||||
//
|
||||
// Postgres tier: PostgresAdapter.migrate() already calls runMigrations() on
|
||||
// the same DATABASE_URL, so a single call covers both the gateway DB and
|
||||
// the storage tables. We deliberately do NOT call runMigrations() here to
|
||||
// avoid opening a second short-lived connection and doubling startup cost.
|
||||
async onModuleInit(): Promise<void> {
|
||||
if (this.config.tier === 'local') {
|
||||
this.logger.log('Applying PGlite schema migrations...');
|
||||
await runPgliteMigrations(this.handle);
|
||||
}
|
||||
this.logger.log(`Initializing storage adapter (${this.storageAdapter.name})...`);
|
||||
await this.storageAdapter.migrate();
|
||||
}
|
||||
|
||||
async onApplicationShutdown(): Promise<void> {
|
||||
await Promise.all([this.handle.close(), this.storageAdapter.close()]);
|
||||
}
|
||||
|
||||
@@ -30,6 +30,7 @@ These are MVP-level checks that don't belong to any single workstream. Updated b
|
||||
| MVP-T04 | not-started | Sync `.mosaic/orchestrator/mission.json` MVP slot with this manifest (milestone enumeration, etc.) | Coord state file; consider whether to repopulate via `mosaic coord` or accept hand-edit |
|
||||
| MVP-T05 | in-progress | Kick off W1 / FED-M1 — federated tier infrastructure | Session 16 (2026-04-19): FED-M1-01 in-progress on `feat/federation-m1-tier-config` |
|
||||
| MVP-T06 | not-started | Declare additional workstreams (web dashboard, TUI/CLI parity, remote control, etc.) as scope solidifies | Track each new workstream by adding a row to the Workstream Rollup |
|
||||
| T-A292E96F | in-progress | Fix Mosaic Gitea PR metadata/login wrapper regression for U-Connect merge preflight | Kanban `t_a292e96f`; branch `fix/t-a292e96f-gitea-pr-metadata`; scratchpad `docs/scratchpads/t-a292e96f-gitea-pr-metadata.md` |
|
||||
|
||||
## Pointer to Active Workstream
|
||||
|
||||
|
||||
53
docs/scratchpads/t-a292e96f-gitea-pr-metadata.md
Normal file
53
docs/scratchpads/t-a292e96f-gitea-pr-metadata.md
Normal file
@@ -0,0 +1,53 @@
|
||||
# t_a292e96f — Gitea PR metadata wrapper fix
|
||||
|
||||
## Objective
|
||||
|
||||
Repair Mosaic git wrappers so Gitea PR metadata and merge preflight work for U-Connect PRs on `git.uscllc.com` without selecting the unrelated `git.mosaicstack.dev` tea login.
|
||||
|
||||
## Findings
|
||||
|
||||
- Reproduced the failure from `/src/uconnect-worktrees/t_39ce717c-authentik-smoke-gate` with the current `pr-metadata.sh`:
|
||||
- PR #1905 returned JSON with `number=null`, `baseRefName=""`, `headRefName=""`.
|
||||
- PR #1908 returned JSON with `number=null`, `baseRefName=""`, `headRefName=""`.
|
||||
- Root cause: the wrapper treated HTTP/API error payloads as PR payloads and normalized missing fields to empty strings.
|
||||
- The credential loader can return a non-working `git.uscllc.com` API token in this environment, while host-specific `~/.git-credentials` basic auth succeeds. The wrapper now falls back by host before normalization.
|
||||
- `tea login list` has only `git.mosaicstack.dev` configured here; `pr-merge.sh` previously forced `--login mosaicstack`, which is invalid for `git.uscllc.com` and caused `Login name mosaicstack does not exist`.
|
||||
|
||||
## Changes
|
||||
|
||||
- `packages/mosaic/framework/tools/git/detect-platform.sh`
|
||||
- Added `get_gitea_basic_auth <host>` to retrieve host-specific HTTPS credentials from `~/.git-credentials` without printing secrets.
|
||||
- `packages/mosaic/framework/tools/git/pr-metadata.sh`
|
||||
- Uses strict bash mode.
|
||||
- Checks Gitea HTTP status and fails nonzero on API errors/non-JSON instead of emitting empty branch fields.
|
||||
- Falls back from token auth to host-specific basic auth.
|
||||
- Normalizes standard `head.ref`/`base.ref` and fallback branch fields.
|
||||
- Requires non-empty `headRefName` and `baseRefName`.
|
||||
- Preserves GitHub `gh pr view` behavior.
|
||||
- `packages/mosaic/framework/tools/git/pr-merge.sh`
|
||||
- Reads metadata once for base-branch policy preflight.
|
||||
- Selects a `tea` login only when its configured URL matches the repo host.
|
||||
- Falls back to authenticated Gitea merge API when no matching `tea` login exists, avoiding the wrong `mosaicstack` login for USC repos.
|
||||
- Keeps squash-only and main-only merge policy.
|
||||
- `packages/mosaic/framework/tools/git/test-pr-metadata-gitea.sh`
|
||||
- Added fixture-based regression harness for standard Gitea fields, fallback branch fields, `refs/pull/<n>/head` plus `head.label` normalization, and API error payloads.
|
||||
|
||||
## Documentation / changelog note
|
||||
|
||||
This repository currently has no root `CHANGELOG.md`; the scratchpad and `docs/TASKS.md` carry the task-level change record for this wrapper fix.
|
||||
|
||||
## Verification log
|
||||
|
||||
- Red regression check: copied the new `test-pr-metadata-gitea.sh` harness next to `origin/main` wrapper scripts and ran it with `MOSAIC_TEST_WORK_DIR=$PWD/.mosaic-test-work/pr-metadata-gitea-red`; it failed as expected with `headRefName=''` and `baseRefName=''` on the fixture API-error path.
|
||||
- `bash -n packages/mosaic/framework/tools/git/{detect-platform.sh,pr-metadata.sh,pr-merge.sh,test-pr-metadata-gitea.sh}`: passed.
|
||||
- `shellcheck -x -P . -e SC1090 packages/mosaic/framework/tools/git/{detect-platform.sh,pr-metadata.sh,pr-merge.sh,test-pr-metadata-gitea.sh}`: passed.
|
||||
- `MOSAIC_TEST_WORK_DIR=$PWD/.mosaic-test-work/pr-metadata-gitea packages/mosaic/framework/tools/git/test-pr-metadata-gitea.sh`: passed; verifies standard Gitea fields, fallback branch fields, `refs/pull/<n>/head` label normalization, and nonzero API-error handling.
|
||||
- Installed wrapper parity: `/home/hermes/.config/mosaic/tools/git/{detect-platform.sh,pr-metadata.sh,pr-merge.sh}` byte-match the PR source copies after validation, so active U-Connect wrapper invocations use the same fix while source PR review runs.
|
||||
- Live sanitized U-Connect metadata from `/src/uconnect` with `MOSAIC_CREDENTIALS_FILE=/src/jarvis-brain/credentials.json`:
|
||||
- PR #1905: `number=1905`, `baseRefName=main`, `headRefName=edith/t_39ce717c-authentik-smoke-gate`, `state=open`, `host=git.uscllc.com`.
|
||||
- PR #1908: `number=1908`, `baseRefName=main`, `headRefName=fix/t_23fa9e1d-portal-health-backend`, `state=closed`, `host=git.uscllc.com`.
|
||||
- Merge preflight dry runs from installed wrappers:
|
||||
- PR #1905: `Dry run: would merge PR #1905 on git.uscllc.com with authenticated Gitea API fallback (base=main, method=squash).`
|
||||
- PR #1908: `Dry run: would merge PR #1908 on git.uscllc.com with authenticated Gitea API fallback (base=main, method=squash).`
|
||||
- PR: `https://git.mosaicstack.dev/mosaicstack/stack/pulls/518`, branch `fix/t-a292e96f-gitea-pr-metadata`.
|
||||
- CI: Recent PR/push pipelines failed before clone/test execution due Woodpecker/Kubernetes PVC API timeout: `dial tcp 10.43.0.1:443: i/o timeout`. No repository test step executed in CI; local targeted verification above remains clean.
|
||||
@@ -42,6 +42,7 @@
|
||||
"access": "public"
|
||||
},
|
||||
"files": [
|
||||
"dist"
|
||||
"dist",
|
||||
"drizzle"
|
||||
]
|
||||
}
|
||||
|
||||
@@ -1,10 +1,12 @@
|
||||
import { PGlite } from '@electric-sql/pglite';
|
||||
import { vector } from '@electric-sql/pglite/vector';
|
||||
import { drizzle } from 'drizzle-orm/pglite';
|
||||
import * as schema from './schema.js';
|
||||
import type { DbHandle } from './client.js';
|
||||
|
||||
export function createPgliteDb(dataDir: string): DbHandle {
|
||||
const client = new PGlite(dataDir);
|
||||
// pgvector extension is required by migration 0001 (insights.embedding column).
|
||||
const client = new PGlite(dataDir, { extensions: { vector } });
|
||||
const db = drizzle(client, { schema });
|
||||
return {
|
||||
db: db as unknown as DbHandle['db'],
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
export { createDb, type Db, type DbHandle } from './client.js';
|
||||
export { createPgliteDb } from './client-pglite.js';
|
||||
export { runMigrations } from './migrate.js';
|
||||
export { runMigrations, runPgliteMigrations } from './migrate.js';
|
||||
export * from './schema.js';
|
||||
export * from './federation.js';
|
||||
export {
|
||||
|
||||
70
packages/db/src/migrate.test.ts
Normal file
70
packages/db/src/migrate.test.ts
Normal file
@@ -0,0 +1,70 @@
|
||||
import { mkdtempSync, rmSync } from 'node:fs';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import { sql } from 'drizzle-orm';
|
||||
import { afterEach, beforeEach, describe, expect, it } from 'vitest';
|
||||
import { createPgliteDb } from './client-pglite.js';
|
||||
import { runPgliteMigrations } from './migrate.js';
|
||||
import type { DbHandle } from './client.js';
|
||||
|
||||
interface PgliteExec {
|
||||
exec(query: string): Promise<unknown>;
|
||||
}
|
||||
|
||||
describe('runPgliteMigrations', () => {
|
||||
let dataDir: string;
|
||||
let handle: DbHandle;
|
||||
|
||||
beforeEach(() => {
|
||||
dataDir = mkdtempSync(join(tmpdir(), 'mosaic-db-migrate-test-'));
|
||||
handle = createPgliteDb(dataDir);
|
||||
});
|
||||
|
||||
afterEach(async () => {
|
||||
await handle.close();
|
||||
rmSync(dataDir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it('creates the BetterAuth tables required by the gateway', async () => {
|
||||
await runPgliteMigrations(handle);
|
||||
|
||||
const result = (await handle.db.execute(sql`
|
||||
SELECT table_name FROM information_schema.tables
|
||||
WHERE table_schema = 'public'
|
||||
ORDER BY table_name
|
||||
`)) as unknown as { rows: Array<{ table_name: string }> };
|
||||
|
||||
const tables = result.rows.map((r) => r.table_name);
|
||||
|
||||
// Auth tables — required for sign-in / bootstrap to function.
|
||||
expect(tables).toContain('users');
|
||||
expect(tables).toContain('sessions');
|
||||
expect(tables).toContain('accounts');
|
||||
expect(tables).toContain('verifications');
|
||||
|
||||
// Schema sanity check — admin token table consumed by mosaic gateway config.
|
||||
expect(tables).toContain('admin_tokens');
|
||||
});
|
||||
|
||||
it('is idempotent — running twice does not error', async () => {
|
||||
await runPgliteMigrations(handle);
|
||||
await expect(runPgliteMigrations(handle)).resolves.toBeUndefined();
|
||||
});
|
||||
|
||||
it('surfaces statement-level error context on failure and leaves no ledger row', async () => {
|
||||
// Pre-create a `users` table that conflicts with migration 0000's CREATE TABLE,
|
||||
// forcing it to fail without IF NOT EXISTS.
|
||||
const client = (handle.db as unknown as { $client: PgliteExec }).$client;
|
||||
await client.exec('CREATE TABLE users (sentinel text)');
|
||||
|
||||
await expect(runPgliteMigrations(handle)).rejects.toThrow(
|
||||
/migration hash=[a-f0-9]+ statement #\d+ failed/,
|
||||
);
|
||||
|
||||
// Ledger should be empty — partial application must not pretend to be complete.
|
||||
const ledger = (await handle.db.execute(
|
||||
sql`SELECT count(*)::int AS count FROM drizzle.__drizzle_migrations`,
|
||||
)) as unknown as { rows: Array<{ count: number }> };
|
||||
expect(ledger.rows[0]?.count).toBe(0);
|
||||
});
|
||||
});
|
||||
@@ -1,18 +1,109 @@
|
||||
import { dirname, resolve } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { drizzle } from 'drizzle-orm/postgres-js';
|
||||
import { migrate } from 'drizzle-orm/postgres-js/migrator';
|
||||
import { sql } from 'drizzle-orm';
|
||||
import { drizzle as drizzlePostgres } from 'drizzle-orm/postgres-js';
|
||||
import { migrate as migratePostgres } from 'drizzle-orm/postgres-js/migrator';
|
||||
import { readMigrationFiles } from 'drizzle-orm/migrator';
|
||||
import postgres from 'postgres';
|
||||
import { DEFAULT_DATABASE_URL } from './defaults.js';
|
||||
import type { DbHandle } from './client.js';
|
||||
|
||||
interface PgliteExecutor {
|
||||
exec(query: string): Promise<unknown>;
|
||||
}
|
||||
|
||||
interface ExecuteRows<T> {
|
||||
rows: T[];
|
||||
}
|
||||
|
||||
function migrationsFolder(): string {
|
||||
const here = dirname(fileURLToPath(import.meta.url));
|
||||
return resolve(here, '../drizzle');
|
||||
}
|
||||
|
||||
export async function runMigrations(url?: string): Promise<void> {
|
||||
const connectionString = url ?? process.env['DATABASE_URL'] ?? DEFAULT_DATABASE_URL;
|
||||
const sql = postgres(connectionString, { max: 1 });
|
||||
const db = drizzle(sql);
|
||||
const __dirname = dirname(fileURLToPath(import.meta.url));
|
||||
const sqlClient = postgres(connectionString, { max: 1 });
|
||||
const db = drizzlePostgres(sqlClient);
|
||||
try {
|
||||
await migrate(db, { migrationsFolder: resolve(__dirname, '../drizzle') });
|
||||
// TODO: postgres-tier first-install also fails because (a) Drizzle wraps every
|
||||
// migration in one transaction (breaks 0009's ALTER TYPE ADD VALUE → SET DEFAULT
|
||||
// sequence) and (b) drizzle/meta/_journal.json has 0009 ordered before 0008,
|
||||
// which the postgres-js migrator skips by `created_at < folderMillis`. The
|
||||
// PGlite path below sidesteps both. A follow-up should either share the
|
||||
// per-statement loop (see runPgliteMigrations) or fix the journal ordering.
|
||||
await migratePostgres(db, { migrationsFolder: migrationsFolder() });
|
||||
} finally {
|
||||
await sql.end();
|
||||
await sqlClient.end();
|
||||
}
|
||||
}
|
||||
|
||||
// Apply Drizzle migrations against an embedded PGlite database.
|
||||
//
|
||||
// We don't reuse drizzle's pglite migrator because it wraps ALL migrations in
|
||||
// one outer transaction, which breaks Postgres' `check_safe_enum_use` rule —
|
||||
// e.g. migration 0009 does `ALTER TYPE ADD VALUE 'pending'` then references
|
||||
// `'pending'` as a default in the same tx. PGlite's `exec()` runs each
|
||||
// statement under the Simple Query protocol, autocommitting between them.
|
||||
//
|
||||
// We still write to the standard `drizzle.__drizzle_migrations` ledger so the
|
||||
// result is interoperable with `runMigrations()` on a postgres-backed deploy
|
||||
// (modulo the journal-ordering bug noted above).
|
||||
//
|
||||
// We skip-by-hash rather than skip-by-folderMillis (which is what Drizzle's
|
||||
// postgres-js migrator does). That's deliberate — out-of-order timestamps in
|
||||
// `_journal.json` won't silently drop migrations.
|
||||
//
|
||||
// Failure model: each statement autocommits, and the ledger row is written
|
||||
// only after all statements in a migration succeed. A crash mid-migration
|
||||
// leaves the prefix applied with no ledger entry, so the next boot will
|
||||
// replay those statements and fail loudly on "already exists". Recovery:
|
||||
// drop the partially-applied objects, or insert the migration's hash into
|
||||
// `drizzle.__drizzle_migrations` manually. The error log identifies which
|
||||
// statement of which migration was the culprit.
|
||||
export async function runPgliteMigrations(handle: DbHandle): Promise<void> {
|
||||
const client = (handle.db as unknown as { $client?: PgliteExecutor }).$client;
|
||||
if (!client || typeof client.exec !== 'function') {
|
||||
throw new Error('runPgliteMigrations: handle.db is not backed by a PGlite client');
|
||||
}
|
||||
|
||||
await client.exec('CREATE SCHEMA IF NOT EXISTS drizzle');
|
||||
await client.exec(`
|
||||
CREATE TABLE IF NOT EXISTS drizzle.__drizzle_migrations (
|
||||
id SERIAL PRIMARY KEY,
|
||||
hash text NOT NULL,
|
||||
created_at bigint
|
||||
)
|
||||
`);
|
||||
|
||||
const appliedRows = (await handle.db.execute(
|
||||
sql`SELECT hash FROM drizzle.__drizzle_migrations`,
|
||||
)) as unknown as ExecuteRows<{ hash: string }>;
|
||||
const applied = new Set(appliedRows.rows.map((r) => r.hash));
|
||||
|
||||
const migrations = readMigrationFiles({ migrationsFolder: migrationsFolder() });
|
||||
for (const migration of migrations) {
|
||||
if (applied.has(migration.hash)) continue;
|
||||
|
||||
// Run each statement-breakpoint chunk in its own exec() call so PGlite
|
||||
// commits between statements — this is what lets `ALTER TYPE ADD VALUE`
|
||||
// become visible before a subsequent statement references the new value.
|
||||
for (const [stmtIdx, stmt] of migration.sql.entries()) {
|
||||
const trimmed = stmt.trim();
|
||||
if (!trimmed) continue;
|
||||
try {
|
||||
await client.exec(trimmed);
|
||||
} catch (err) {
|
||||
const cause = err instanceof Error ? err.message : String(err);
|
||||
throw new Error(
|
||||
`runPgliteMigrations: migration hash=${migration.hash} statement #${stmtIdx} failed: ${cause}\n` +
|
||||
`Statement: ${trimmed.slice(0, 200)}${trimmed.length > 200 ? '…' : ''}`,
|
||||
{ cause: err },
|
||||
);
|
||||
}
|
||||
}
|
||||
await handle.db.execute(
|
||||
sql`INSERT INTO drizzle.__drizzle_migrations (hash, created_at) VALUES (${migration.hash}, ${migration.folderMillis})`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -103,16 +103,28 @@ get_gitea_token() {
|
||||
if [[ -f "$cred_loader" ]]; then
|
||||
local token
|
||||
token=$(
|
||||
# shellcheck source=/dev/null
|
||||
source "$cred_loader"
|
||||
# Host-specific wrapper resolution must not inherit caller/global GITEA_*.
|
||||
# load_credentials intentionally preserves existing env vars for interactive use,
|
||||
# but metadata/merge wrappers need credentials matching the remote host.
|
||||
unset GITEA_TOKEN GITEA_URL
|
||||
case "$host" in
|
||||
git.mosaicstack.dev) load_credentials gitea-mosaicstack 2>/dev/null ;;
|
||||
git.uscllc.com) load_credentials gitea-usc 2>/dev/null ;;
|
||||
*)
|
||||
local matched=false
|
||||
for svc in gitea-mosaicstack gitea-usc; do
|
||||
load_credentials "$svc" 2>/dev/null || continue
|
||||
[[ "${GITEA_URL:-}" == *"$host"* ]] && break
|
||||
unset GITEA_TOKEN GITEA_URL
|
||||
load_credentials "$svc" 2>/dev/null || continue
|
||||
if [[ "${GITEA_URL:-}" == "https://$host" || "${GITEA_URL:-}" == "http://$host" || "${GITEA_URL:-}" == *"//$host" ]]; then
|
||||
matched=true
|
||||
break
|
||||
fi
|
||||
done
|
||||
if [[ "$matched" != true ]]; then
|
||||
unset GITEA_TOKEN GITEA_URL
|
||||
fi
|
||||
;;
|
||||
esac
|
||||
echo "${GITEA_TOKEN:-}"
|
||||
@@ -123,11 +135,13 @@ get_gitea_token() {
|
||||
fi
|
||||
fi
|
||||
|
||||
# 2. GITEA_TOKEN env var (may be set by caller)
|
||||
# 2. GITEA_TOKEN env var (only when GITEA_URL, if present, matches the remote host)
|
||||
if [[ -n "${GITEA_TOKEN:-}" ]]; then
|
||||
if [[ -z "${GITEA_URL:-}" || "${GITEA_URL:-}" == "https://$host" || "${GITEA_URL:-}" == "http://$host" || "${GITEA_URL:-}" == *"//$host" ]]; then
|
||||
echo "$GITEA_TOKEN"
|
||||
return 0
|
||||
fi
|
||||
fi
|
||||
|
||||
# 3. ~/.git-credentials file
|
||||
local creds="$HOME/.git-credentials"
|
||||
@@ -143,6 +157,37 @@ get_gitea_token() {
|
||||
return 1
|
||||
}
|
||||
|
||||
# Resolve HTTPS basic auth credentials for a Gitea host from ~/.git-credentials.
|
||||
# Prints "username:password" for direct curl -u consumption. Callers must not log it.
|
||||
get_gitea_basic_auth() {
|
||||
local host="$1"
|
||||
local creds="$HOME/.git-credentials"
|
||||
if [[ ! -f "$creds" ]]; then
|
||||
return 1
|
||||
fi
|
||||
|
||||
python3 - "$host" "$creds" <<'PY'
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from urllib.parse import unquote, urlparse
|
||||
|
||||
host = sys.argv[1]
|
||||
creds = Path(sys.argv[2])
|
||||
|
||||
for line in creds.read_text(encoding="utf-8").splitlines():
|
||||
parsed = urlparse(line.strip())
|
||||
if parsed.hostname != host:
|
||||
continue
|
||||
username = unquote(parsed.username or "")
|
||||
password = unquote(parsed.password or "")
|
||||
if username and password:
|
||||
print(f"{username}:{password}")
|
||||
raise SystemExit(0)
|
||||
|
||||
raise SystemExit(1)
|
||||
PY
|
||||
}
|
||||
|
||||
# If script is run directly (not sourced), output the platform
|
||||
if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
|
||||
detect_platform
|
||||
|
||||
@@ -2,9 +2,10 @@
|
||||
# pr-merge.sh - Merge pull requests on Gitea or GitHub
|
||||
# Usage: pr-merge.sh -n PR_NUMBER [-m squash] [-d] [--skip-queue-guard]
|
||||
|
||||
set -e
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# shellcheck source=packages/mosaic/framework/tools/git/detect-platform.sh
|
||||
source "$SCRIPT_DIR/detect-platform.sh"
|
||||
|
||||
# Default values
|
||||
@@ -12,6 +13,7 @@ PR_NUMBER=""
|
||||
MERGE_METHOD="squash"
|
||||
DELETE_BRANCH=false
|
||||
SKIP_QUEUE_GUARD=false
|
||||
DRY_RUN=false
|
||||
|
||||
usage() {
|
||||
cat <<EOF
|
||||
@@ -24,6 +26,7 @@ Options:
|
||||
-m, --method METHOD Merge method: squash only (default: squash)
|
||||
-d, --delete-branch Delete the head branch after merge
|
||||
--skip-queue-guard Skip CI queue guard wait before merge
|
||||
--dry-run Run metadata/login preflight without merging
|
||||
-h, --help Show this help message
|
||||
|
||||
Examples:
|
||||
@@ -54,6 +57,11 @@ while [[ $# -gt 0 ]]; do
|
||||
SKIP_QUEUE_GUARD=true
|
||||
shift
|
||||
;;
|
||||
--dry-run)
|
||||
DRY_RUN=true
|
||||
SKIP_QUEUE_GUARD=true
|
||||
shift
|
||||
;;
|
||||
-h|--help)
|
||||
usage
|
||||
;;
|
||||
@@ -74,7 +82,8 @@ if [[ "$MERGE_METHOD" != "squash" ]]; then
|
||||
exit 1
|
||||
fi
|
||||
|
||||
BASE_BRANCH="$("$SCRIPT_DIR/pr-metadata.sh" -n "$PR_NUMBER" | python3 -c 'import json, sys; print((json.load(sys.stdin).get("baseRefName") or "").strip())')"
|
||||
PR_METADATA="$("$SCRIPT_DIR/pr-metadata.sh" -n "$PR_NUMBER")"
|
||||
BASE_BRANCH="$(printf '%s' "$PR_METADATA" | python3 -c 'import json, sys; print((json.load(sys.stdin).get("baseRefName") or "").strip())')"
|
||||
if [[ "$BASE_BRANCH" != "main" ]]; then
|
||||
echo "Error: Mosaic policy allows merges only for PRs targeting 'main' (found '$BASE_BRANCH')." >&2
|
||||
exit 1
|
||||
@@ -92,21 +101,122 @@ PLATFORM=$(detect_platform)
|
||||
OWNER=$(get_repo_owner)
|
||||
REPO=$(get_repo_name)
|
||||
|
||||
find_tea_login_for_host() {
|
||||
local host="$1"
|
||||
local logins_json
|
||||
command -v tea >/dev/null 2>&1 || return 1
|
||||
logins_json=$(tea login list --output json 2>/dev/null) || return 1
|
||||
TEA_LOGINS_JSON="$logins_json" python3 - "$host" <<'PY'
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
|
||||
host = sys.argv[1]
|
||||
try:
|
||||
logins = json.loads(os.environ.get("TEA_LOGINS_JSON", "[]"))
|
||||
except Exception:
|
||||
raise SystemExit(1)
|
||||
|
||||
for login in logins if isinstance(logins, list) else []:
|
||||
url = str(login.get("url") or login.get("URL") or "")
|
||||
name = str(login.get("name") or login.get("Name") or "")
|
||||
if url.rstrip("/").endswith(host) and name:
|
||||
print(name)
|
||||
raise SystemExit(0)
|
||||
|
||||
raise SystemExit(1)
|
||||
PY
|
||||
}
|
||||
|
||||
merge_gitea_with_api() {
|
||||
local host="$1" api_url token basic_auth body_file raw_code payload
|
||||
body_file=$(mktemp)
|
||||
payload='{"Do":"squash"}'
|
||||
|
||||
token=$(get_gitea_token "$host" || true)
|
||||
if [[ -n "$token" ]]; then
|
||||
raw_code=$(curl -sS -w '%{http_code}' -o "$body_file" \
|
||||
-X POST \
|
||||
-H "Authorization: token $token" \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d "$payload" \
|
||||
"$api_url" || true)
|
||||
if [[ "$raw_code" =~ ^2 ]]; then
|
||||
rm -f "$body_file"
|
||||
return 0
|
||||
fi
|
||||
fi
|
||||
|
||||
basic_auth=$(get_gitea_basic_auth "$host" || true)
|
||||
if [[ -n "$basic_auth" ]]; then
|
||||
raw_code=$(curl -sS -w '%{http_code}' -o "$body_file" \
|
||||
-X POST \
|
||||
-u "$basic_auth" \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d "$payload" \
|
||||
"$api_url" || true)
|
||||
if [[ "$raw_code" =~ ^2 ]]; then
|
||||
rm -f "$body_file"
|
||||
return 0
|
||||
fi
|
||||
fi
|
||||
|
||||
python3 - "${raw_code:-000}" "$body_file" <<'PY' >&2
|
||||
import json
|
||||
import sys
|
||||
code, path = sys.argv[1], sys.argv[2]
|
||||
try:
|
||||
data = json.load(open(path, encoding="utf-8"))
|
||||
message = data.get("message") or data.get("error") or "unknown API error"
|
||||
except Exception:
|
||||
message = open(path, encoding="utf-8", errors="replace").read()[:200] or "empty response"
|
||||
print(f"Error: Gitea API merge failed with HTTP {code}: {message}")
|
||||
PY
|
||||
rm -f "$body_file"
|
||||
return 1
|
||||
}
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
if [[ "$PLATFORM" == "gitea" ]]; then
|
||||
HOST=$(get_remote_host) || {
|
||||
echo "Error: Cannot determine host from origin remote URL" >&2
|
||||
exit 1
|
||||
}
|
||||
TEA_LOGIN="${GITEA_LOGIN:-$(find_tea_login_for_host "$HOST" || true)}"
|
||||
if [[ -n "$TEA_LOGIN" ]]; then
|
||||
echo "Dry run: would merge PR #$PR_NUMBER on $HOST with tea login '$TEA_LOGIN' (base=$BASE_BRANCH, method=squash)."
|
||||
else
|
||||
echo "Dry run: would merge PR #$PR_NUMBER on $HOST with authenticated Gitea API fallback (base=$BASE_BRANCH, method=squash)."
|
||||
fi
|
||||
else
|
||||
echo "Dry run: would merge PR #$PR_NUMBER on $PLATFORM (base=$BASE_BRANCH, method=squash)."
|
||||
fi
|
||||
exit 0
|
||||
fi
|
||||
|
||||
case "$PLATFORM" in
|
||||
github)
|
||||
CMD="gh pr merge $PR_NUMBER --squash"
|
||||
[[ "$DELETE_BRANCH" == true ]] && CMD="$CMD --delete-branch"
|
||||
eval "$CMD"
|
||||
GH_ARGS=(pr merge "$PR_NUMBER" --squash)
|
||||
[[ "$DELETE_BRANCH" == true ]] && GH_ARGS+=(--delete-branch)
|
||||
gh "${GH_ARGS[@]}"
|
||||
;;
|
||||
gitea)
|
||||
CMD="tea pr merge $PR_NUMBER --style squash --repo $OWNER/$REPO --login ${GITEA_LOGIN:-mosaicstack}"
|
||||
HOST=$(get_remote_host) || {
|
||||
echo "Error: Cannot determine host from origin remote URL" >&2
|
||||
exit 1
|
||||
}
|
||||
TEA_LOGIN="${GITEA_LOGIN:-$(find_tea_login_for_host "$HOST" || true)}"
|
||||
if [[ -n "$TEA_LOGIN" ]]; then
|
||||
tea pr merge "$PR_NUMBER" --style squash --repo "$OWNER/$REPO" --login "$TEA_LOGIN"
|
||||
else
|
||||
echo "No tea login configured for $HOST; using authenticated Gitea API merge fallback." >&2
|
||||
merge_gitea_with_api "$HOST" "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/pulls/${PR_NUMBER}/merge"
|
||||
fi
|
||||
|
||||
# Delete branch after merge if requested
|
||||
if [[ "$DELETE_BRANCH" == true ]]; then
|
||||
echo "Note: Branch deletion after merge may need to be done separately with tea" >&2
|
||||
fi
|
||||
|
||||
eval "$CMD"
|
||||
;;
|
||||
*)
|
||||
echo "Error: Could not detect git platform" >&2
|
||||
|
||||
@@ -2,9 +2,10 @@
|
||||
# pr-metadata.sh - Get PR metadata as JSON on GitHub or Gitea
|
||||
# Usage: pr-metadata.sh -n <pr_number> [-o <output_file>]
|
||||
|
||||
set -e
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# shellcheck source=packages/mosaic/framework/tools/git/detect-platform.sh
|
||||
source "$SCRIPT_DIR/detect-platform.sh"
|
||||
|
||||
# Parse arguments
|
||||
@@ -31,7 +32,7 @@ while [[ $# -gt 0 ]]; do
|
||||
exit 0
|
||||
;;
|
||||
*)
|
||||
echo "Unknown option: $1"
|
||||
echo "Unknown option: $1" >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
@@ -42,56 +43,168 @@ if [[ -z "$PR_NUMBER" ]]; then
|
||||
exit 1
|
||||
fi
|
||||
|
||||
write_metadata() {
|
||||
local metadata="$1"
|
||||
if [[ -n "$OUTPUT_FILE" ]]; then
|
||||
printf '%s\n' "$metadata" > "$OUTPUT_FILE"
|
||||
else
|
||||
printf '%s\n' "$metadata"
|
||||
fi
|
||||
}
|
||||
|
||||
curl_gitea_pull() {
|
||||
local api_url="$1"
|
||||
local token basic_auth raw_code body_file http_code
|
||||
body_file=$(mktemp)
|
||||
|
||||
token=$(get_gitea_token "$HOST" || true)
|
||||
if [[ -n "$token" ]]; then
|
||||
raw_code=$(curl -sS -w '%{http_code}' -o "$body_file" -H "Authorization: token $token" "$api_url" || true)
|
||||
if [[ "$raw_code" =~ ^2 ]]; then
|
||||
cat "$body_file"
|
||||
rm -f "$body_file"
|
||||
return 0
|
||||
fi
|
||||
http_code="$raw_code"
|
||||
fi
|
||||
|
||||
basic_auth=$(get_gitea_basic_auth "$HOST" || true)
|
||||
if [[ -n "$basic_auth" ]]; then
|
||||
raw_code=$(curl -sS -w '%{http_code}' -o "$body_file" -u "$basic_auth" "$api_url" || true)
|
||||
if [[ "$raw_code" =~ ^2 ]]; then
|
||||
cat "$body_file"
|
||||
rm -f "$body_file"
|
||||
return 0
|
||||
fi
|
||||
http_code="$raw_code"
|
||||
fi
|
||||
|
||||
if [[ -z "${http_code:-}" ]]; then
|
||||
raw_code=$(curl -sS -w '%{http_code}' -o "$body_file" "$api_url" || true)
|
||||
http_code="$raw_code"
|
||||
fi
|
||||
|
||||
python3 - "$http_code" "$body_file" <<'PY' >&2
|
||||
import json
|
||||
import sys
|
||||
|
||||
code, path = sys.argv[1], sys.argv[2]
|
||||
try:
|
||||
data = json.load(open(path, encoding="utf-8"))
|
||||
message = data.get("message") or data.get("error") or "unknown API error"
|
||||
except Exception:
|
||||
message = open(path, encoding="utf-8", errors="replace").read()[:200] or "empty response"
|
||||
print(f"Error: Gitea pull request API request failed with HTTP {code}: {message}")
|
||||
PY
|
||||
rm -f "$body_file"
|
||||
return 1
|
||||
}
|
||||
|
||||
detect_platform > /dev/null
|
||||
|
||||
if [[ "$PLATFORM" == "github" ]]; then
|
||||
METADATA=$(gh pr view "$PR_NUMBER" --json number,title,body,state,author,headRefName,baseRefName,files,labels,assignees,milestone,createdAt,updatedAt,url,isDraft)
|
||||
|
||||
if [[ -n "$OUTPUT_FILE" ]]; then
|
||||
echo "$METADATA" > "$OUTPUT_FILE"
|
||||
else
|
||||
echo "$METADATA"
|
||||
fi
|
||||
write_metadata "$METADATA"
|
||||
elif [[ "$PLATFORM" == "gitea" ]]; then
|
||||
OWNER=$(get_repo_owner)
|
||||
REPO=$(get_repo_name)
|
||||
REMOTE_URL=$(git remote get-url origin 2>/dev/null)
|
||||
|
||||
# Extract host from remote URL
|
||||
if [[ "$REMOTE_URL" == https://* ]]; then
|
||||
HOST=$(echo "$REMOTE_URL" | sed -E 's|https://([^/]+)/.*|\1|')
|
||||
elif [[ "$REMOTE_URL" == git@* ]]; then
|
||||
HOST=$(echo "$REMOTE_URL" | sed -E 's|git@([^:]+):.*|\1|')
|
||||
else
|
||||
echo "Error: Cannot determine host from remote URL" >&2
|
||||
HOST=$(get_remote_host) || {
|
||||
echo "Error: Cannot determine host from origin remote URL" >&2
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
API_URL="https://${HOST}/api/v1/repos/${OWNER}/${REPO}/pulls/${PR_NUMBER}"
|
||||
|
||||
GITEA_API_TOKEN=$(get_gitea_token "$HOST" || true)
|
||||
|
||||
if [[ -n "$GITEA_API_TOKEN" ]]; then
|
||||
RAW=$(curl -sS -H "Authorization: token $GITEA_API_TOKEN" "$API_URL")
|
||||
if [[ -n "${MOSAIC_GITEA_PR_METADATA_RAW_FILE:-}" ]]; then
|
||||
RAW=$(cat "$MOSAIC_GITEA_PR_METADATA_RAW_FILE")
|
||||
else
|
||||
RAW=$(curl -sS "$API_URL")
|
||||
RAW=$(curl_gitea_pull "$API_URL")
|
||||
fi
|
||||
|
||||
# Normalize Gitea response to match our expected schema
|
||||
METADATA=$(echo "$RAW" | python3 -c "
|
||||
import json, sys
|
||||
data = json.load(sys.stdin)
|
||||
# Normalize Gitea response to match GitHub's expected metadata schema.
|
||||
METADATA=$(printf '%s' "$RAW" | python3 -c "
|
||||
import json
|
||||
import sys
|
||||
|
||||
def first_non_empty(*values):
|
||||
for value in values:
|
||||
if value is None:
|
||||
continue
|
||||
if isinstance(value, str):
|
||||
value = value.strip()
|
||||
if value:
|
||||
return value
|
||||
return ''
|
||||
|
||||
def nested(data, *keys):
|
||||
current = data
|
||||
for key in keys:
|
||||
if not isinstance(current, dict):
|
||||
return None
|
||||
current = current.get(key)
|
||||
return current
|
||||
|
||||
try:
|
||||
data = json.load(sys.stdin)
|
||||
except json.JSONDecodeError as exc:
|
||||
print(f'Error: Gitea API returned non-JSON response: {exc}', file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
if not isinstance(data, dict):
|
||||
print('Error: Gitea API returned an unexpected non-object response', file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
if data.get('message') and not data.get('number'):
|
||||
print(f\"Error: Gitea API error: {data.get('message')}\", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
head_ref = first_non_empty(
|
||||
nested(data, 'head', 'ref'),
|
||||
nested(data, 'head', 'name'),
|
||||
nested(data, 'head', 'branch'),
|
||||
data.get('head_branch'),
|
||||
data.get('head_ref'),
|
||||
nested(data, 'head', 'label'),
|
||||
data.get('head_label'),
|
||||
)
|
||||
if isinstance(head_ref, str) and head_ref.startswith('refs/pull/'):
|
||||
head_ref = first_non_empty(
|
||||
nested(data, 'head', 'label'),
|
||||
data.get('head_label'),
|
||||
nested(data, 'head', 'name'),
|
||||
nested(data, 'head', 'branch'),
|
||||
data.get('head_branch'),
|
||||
data.get('head_ref'),
|
||||
head_ref,
|
||||
)
|
||||
base_ref = first_non_empty(
|
||||
nested(data, 'base', 'ref'),
|
||||
nested(data, 'base', 'name'),
|
||||
nested(data, 'base', 'branch'),
|
||||
data.get('base_branch'),
|
||||
data.get('base_ref'),
|
||||
data.get('base_label'),
|
||||
)
|
||||
|
||||
if not head_ref or not base_ref:
|
||||
available = ', '.join(sorted(data.keys()))
|
||||
print(
|
||||
'Error: Unable to resolve non-empty Gitea PR head/base refs '
|
||||
f'(headRefName={head_ref!r}, baseRefName={base_ref!r}; keys={available})',
|
||||
file=sys.stderr,
|
||||
)
|
||||
sys.exit(1)
|
||||
|
||||
normalized = {
|
||||
'number': data.get('number'),
|
||||
'title': data.get('title'),
|
||||
'body': data.get('body', ''),
|
||||
'state': data.get('state'),
|
||||
'author': data.get('user', {}).get('login', ''),
|
||||
'headRefName': data.get('head', {}).get('ref', ''),
|
||||
'baseRefName': data.get('base', {}).get('ref', ''),
|
||||
'labels': [l.get('name', '') for l in data.get('labels', [])],
|
||||
'assignees': [a.get('login', '') for a in data.get('assignees', [])],
|
||||
'milestone': data.get('milestone', {}).get('title', '') if data.get('milestone') else '',
|
||||
'author': nested(data, 'user', 'login') or '',
|
||||
'headRefName': head_ref,
|
||||
'baseRefName': base_ref,
|
||||
'labels': [l.get('name', '') for l in data.get('labels', []) if isinstance(l, dict)],
|
||||
'assignees': [a.get('login', '') for a in data.get('assignees', []) if isinstance(a, dict)],
|
||||
'milestone': nested(data, 'milestone', 'title') or '',
|
||||
'createdAt': data.get('created_at', ''),
|
||||
'updatedAt': data.get('updated_at', ''),
|
||||
'url': data.get('html_url', ''),
|
||||
@@ -102,11 +215,7 @@ normalized = {
|
||||
json.dump(normalized, sys.stdout, indent=2)
|
||||
")
|
||||
|
||||
if [[ -n "$OUTPUT_FILE" ]]; then
|
||||
echo "$METADATA" > "$OUTPUT_FILE"
|
||||
else
|
||||
echo "$METADATA"
|
||||
fi
|
||||
write_metadata "$METADATA"
|
||||
else
|
||||
echo "Error: Unknown platform" >&2
|
||||
exit 1
|
||||
|
||||
87
packages/mosaic/framework/tools/git/test-pr-metadata-gitea.sh
Executable file
87
packages/mosaic/framework/tools/git/test-pr-metadata-gitea.sh
Executable file
@@ -0,0 +1,87 @@
|
||||
#!/usr/bin/env bash
|
||||
# Regression harness for Gitea PR metadata normalization.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
WORK_DIR="${MOSAIC_TEST_WORK_DIR:-$PWD/.mosaic-test-work/pr-metadata-gitea}"
|
||||
REPO_DIR="$WORK_DIR/repo"
|
||||
FIXTURE_DIR="$WORK_DIR/fixtures"
|
||||
|
||||
rm -rf "$WORK_DIR"
|
||||
mkdir -p "$REPO_DIR" "$FIXTURE_DIR"
|
||||
|
||||
git -C "$REPO_DIR" init -q
|
||||
git -C "$REPO_DIR" remote add origin https://git.uscllc.com/USC/uconnect.git
|
||||
|
||||
cat > "$FIXTURE_DIR/gitea-standard.json" <<'JSON'
|
||||
{
|
||||
"number": 1905,
|
||||
"title": "Smoke gate fix",
|
||||
"state": "open",
|
||||
"user": {"login": "edith"},
|
||||
"head": {"ref": "edith/t_39ce717c-authentik-smoke-gate"},
|
||||
"base": {"ref": "main"},
|
||||
"labels": [{"name": "ci"}],
|
||||
"assignees": [{"login": "edith"}],
|
||||
"html_url": "https://git.uscllc.com/USC/uconnect/pulls/1905"
|
||||
}
|
||||
JSON
|
||||
|
||||
cat > "$FIXTURE_DIR/gitea-fallback.json" <<'JSON'
|
||||
{
|
||||
"number": 1908,
|
||||
"title": "Fallback branch fields",
|
||||
"state": "open",
|
||||
"user": {"login": "edith"},
|
||||
"head_branch": "fix/fallback-head",
|
||||
"base_branch": "main",
|
||||
"html_url": "https://git.uscllc.com/USC/uconnect/pulls/1908"
|
||||
}
|
||||
JSON
|
||||
|
||||
cat > "$FIXTURE_DIR/gitea-refs-pull-label.json" <<'JSON'
|
||||
{
|
||||
"number": 1908,
|
||||
"title": "Closed merged PR with synthetic pull ref",
|
||||
"state": "closed",
|
||||
"user": {"login": "edith"},
|
||||
"head": {"ref": "refs/pull/1908/head", "label": "fix/t_23fa9e1d-portal-health-backend"},
|
||||
"base": {"ref": "main"},
|
||||
"html_url": "https://git.uscllc.com/USC/uconnect/pulls/1908"
|
||||
}
|
||||
JSON
|
||||
|
||||
cat > "$FIXTURE_DIR/gitea-error.json" <<'JSON'
|
||||
{"message": "user does not exist [uid: 0, name: ]", "url": "https://git.uscllc.com/api/swagger"}
|
||||
JSON
|
||||
|
||||
run_case() {
|
||||
local fixture="$1" expected_number="$2" expected_head="$3"
|
||||
local output
|
||||
output=$(cd "$REPO_DIR" && MOSAIC_GITEA_PR_METADATA_RAW_FILE="$fixture" "$SCRIPT_DIR/pr-metadata.sh" -n "$expected_number")
|
||||
PR_METADATA_OUTPUT="$output" python3 - "$expected_number" "$expected_head" <<'PY'
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
|
||||
data = json.loads(os.environ["PR_METADATA_OUTPUT"])
|
||||
expected_number = int(sys.argv[1])
|
||||
expected_head = sys.argv[2]
|
||||
assert data["number"] == expected_number, data
|
||||
assert data["baseRefName"] == "main", data
|
||||
assert data["headRefName"] == expected_head, data
|
||||
PY
|
||||
}
|
||||
|
||||
run_case "$FIXTURE_DIR/gitea-standard.json" 1905 edith/t_39ce717c-authentik-smoke-gate
|
||||
run_case "$FIXTURE_DIR/gitea-fallback.json" 1908 fix/fallback-head
|
||||
run_case "$FIXTURE_DIR/gitea-refs-pull-label.json" 1908 fix/t_23fa9e1d-portal-health-backend
|
||||
|
||||
if cd "$REPO_DIR" && MOSAIC_GITEA_PR_METADATA_RAW_FILE="$FIXTURE_DIR/gitea-error.json" "$SCRIPT_DIR/pr-metadata.sh" -n 1909 >/dev/null 2>"$WORK_DIR/error.log"; then
|
||||
echo "Expected API error fixture to fail" >&2
|
||||
exit 1
|
||||
fi
|
||||
grep -q "Gitea API error" "$WORK_DIR/error.log"
|
||||
|
||||
echo "Gitea PR metadata regression harness passed"
|
||||
@@ -16,8 +16,15 @@ import fs from 'node:fs/promises';
|
||||
import os from 'node:os';
|
||||
import path from 'node:path';
|
||||
|
||||
import { users, teams, teamMembers, conversations, messages } from '@mosaicstack/db';
|
||||
import { createPgliteDbWithVector, runPgliteMigrations } from './test-utils/pglite-with-vector.js';
|
||||
import {
|
||||
users,
|
||||
teams,
|
||||
teamMembers,
|
||||
conversations,
|
||||
messages,
|
||||
createPgliteDb,
|
||||
runPgliteMigrations,
|
||||
} from '@mosaicstack/db';
|
||||
|
||||
import postgres from 'postgres';
|
||||
import { afterAll, describe, expect, it } from 'vitest';
|
||||
@@ -102,11 +109,8 @@ describe.skipIf(!run)('migrate-tier — PGlite → federated PG', () => {
|
||||
/* ---- 1. Create a temp PGlite db ---------------------------------- */
|
||||
|
||||
pgliteDataDir = await fs.mkdtemp(path.join(os.tmpdir(), 'fed-m1-08-'));
|
||||
const handle = createPgliteDbWithVector(pgliteDataDir);
|
||||
|
||||
// Run Drizzle migrations against PGlite.
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
await runPgliteMigrations(handle.db as any);
|
||||
const handle = createPgliteDb(pgliteDataDir);
|
||||
await runPgliteMigrations(handle);
|
||||
|
||||
/* ---- 2. Seed representative data --------------------------------- */
|
||||
|
||||
|
||||
@@ -1,52 +0,0 @@
|
||||
/**
|
||||
* Test-only helpers for creating a PGlite database with the pgvector extension
|
||||
* and running Drizzle migrations against it.
|
||||
*
|
||||
* These are intentionally NOT exported from @mosaicstack/db to avoid pulling
|
||||
* the WASM vector bundle into the public API surface.
|
||||
*/
|
||||
|
||||
import { createRequire } from 'node:module';
|
||||
import { dirname, resolve } from 'node:path';
|
||||
|
||||
import { PGlite } from '@electric-sql/pglite';
|
||||
import { vector } from '@electric-sql/pglite/vector';
|
||||
import { drizzle } from 'drizzle-orm/pglite';
|
||||
import { migrate as migratePglite } from 'drizzle-orm/pglite/migrator';
|
||||
import type { PgliteDatabase } from 'drizzle-orm/pglite';
|
||||
import * as schema from '@mosaicstack/db';
|
||||
import type { DbHandle } from '@mosaicstack/db';
|
||||
|
||||
/**
|
||||
* Create a PGlite DB handle with the pgvector extension loaded.
|
||||
* Required for running Drizzle migrations that include `CREATE EXTENSION vector`.
|
||||
*/
|
||||
export function createPgliteDbWithVector(dataDir: string): DbHandle {
|
||||
const client = new PGlite(dataDir, { extensions: { vector } });
|
||||
const db = drizzle(client, { schema });
|
||||
return {
|
||||
db: db as unknown as DbHandle['db'],
|
||||
close: async () => {
|
||||
await client.close();
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Run Drizzle migrations against an already-open PGlite database handle.
|
||||
* Resolves the migrations folder from @mosaicstack/db's installed location.
|
||||
*
|
||||
* @param db A PgliteDatabase instance (from drizzle-orm/pglite).
|
||||
*/
|
||||
export async function runPgliteMigrations(
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
db: PgliteDatabase<any>,
|
||||
): Promise<void> {
|
||||
// Resolve @mosaicstack/db package root to locate its drizzle migrations folder.
|
||||
const _require = createRequire(import.meta.url);
|
||||
const dbPkgMain = _require.resolve('@mosaicstack/db');
|
||||
// dbPkgMain → …/packages/db/dist/index.js → dirname = dist/
|
||||
// go up one level from dist/ to find the sibling drizzle/ folder
|
||||
const migrationsFolder = resolve(dirname(dbPkgMain), '../drizzle');
|
||||
await migratePglite(db, { migrationsFolder });
|
||||
}
|
||||
125
scratchpads/fix-db-bootstrap-migrations.md
Normal file
125
scratchpads/fix-db-bootstrap-migrations.md
Normal file
@@ -0,0 +1,125 @@
|
||||
# fix(db): bootstrap migrations on local-tier gateway startup
|
||||
|
||||
## Problem
|
||||
|
||||
Fresh `mosaic gateway install` (npm-installed) leaves the gateway DB schema empty:
|
||||
|
||||
```
|
||||
relation "users" does not exist
|
||||
```
|
||||
|
||||
Sign-in 500s, `auth users create` says "Not signed in", `admin/bootstrap setup`
|
||||
also fails — every entry point queries `users` before doing anything else.
|
||||
|
||||
## Scope
|
||||
|
||||
This PR fixes the **local (PGlite) tier** end-to-end. The postgres-tier path
|
||||
has additional pre-existing bugs (see "Known issues, out of scope" below) and
|
||||
needs a separate change with real Postgres validation.
|
||||
|
||||
## Root causes addressed (5 stacked bugs on the local-tier path)
|
||||
|
||||
1. **`packages/db/package.json` `files: ["dist"]`** — the `drizzle/` SQL
|
||||
migrations folder is excluded from the published tarball. Even if a
|
||||
migrate runner existed, it would have nothing to apply.
|
||||
|
||||
2. **`packages/db/src/migrate.ts`** only supports `postgres-js`. Local-tier
|
||||
gateways use embedded PGlite, which can't be reached over a postgres wire
|
||||
protocol — so `runMigrations()` is unusable for the local tier.
|
||||
|
||||
3. **`apps/gateway/src/database/database.module.ts`** never invokes
|
||||
migrations at startup. The module creates the DB handle and storage
|
||||
adapter, but no consumer calls `.migrate()` on either. `mosaic storage
|
||||
migrate` CLI even claims "pglite runs schema setup automatically on first
|
||||
connection via `adapter.migrate()`" — but `adapter.migrate()` is only
|
||||
called by tests, never at runtime.
|
||||
|
||||
4. **`createPgliteDb` does not load the pgvector extension.** Migration 0001
|
||||
declares `CREATE EXTENSION IF NOT EXISTS vector;` for the
|
||||
`insights.embedding` column. Bare PGlite has no pgvector — the migration
|
||||
fails on extension control file lookup.
|
||||
|
||||
5. **Drizzle's PG migrator wraps every migration in one outer transaction.**
|
||||
Migration 0009 does `ALTER TYPE grant_status ADD VALUE 'pending'` and then
|
||||
`ALTER TABLE federation_grants ALTER COLUMN status SET DEFAULT 'pending'`.
|
||||
Postgres' `check_safe_enum_use` rejects the second statement because the
|
||||
new enum value isn't committed yet. Splitting the migration into two
|
||||
files doesn't help — drizzle batches all migrations into one outer tx.
|
||||
|
||||
## Fix
|
||||
|
||||
- `packages/db/package.json` — ship `drizzle/` in `files`.
|
||||
- `packages/db/src/client-pglite.ts` — load `@electric-sql/pglite/vector`.
|
||||
- `packages/db/src/migrate.ts` — add `runPgliteMigrations(handle)`. Walks the
|
||||
Drizzle journal and runs each statement-breakpoint chunk through PGlite's
|
||||
`client.exec()` (Simple Query protocol → autocommit per statement). Writes
|
||||
to the standard `drizzle.__drizzle_migrations` ledger so the result is
|
||||
interoperable with `runMigrations()` on a postgres-backed deployment.
|
||||
Per-statement try/catch surfaces which statement of which migration failed
|
||||
and the ledger row is only written on full success.
|
||||
- `packages/db/src/index.ts` — re-export.
|
||||
- `apps/gateway/src/database/database.module.ts` — implement `OnModuleInit`:
|
||||
- Local tier → `runPgliteMigrations(handle)`, then `storageAdapter.migrate()`
|
||||
(the local storage adapter has its own kv tables in a separate PGlite dir).
|
||||
- Postgres tier → `storageAdapter.migrate()` only, since
|
||||
`PostgresAdapter.migrate()` already calls `runMigrations(url)` against
|
||||
the same DATABASE_URL — we deliberately don't double-call.
|
||||
|
||||
NestJS awaits `onModuleInit` before `app.listen()`, so DB-dependent modules
|
||||
see a populated schema before any HTTP traffic is accepted.
|
||||
|
||||
- `packages/storage/src/test-utils/pglite-with-vector.ts` — **deleted**.
|
||||
The "intentionally not exported" rationale is moot now that migration 0001
|
||||
forces pgvector load anyway. `migrate-tier.integration.test.ts` switched
|
||||
to `createPgliteDb` + `runPgliteMigrations` from `@mosaicstack/db`.
|
||||
|
||||
## Tests
|
||||
|
||||
`packages/db/src/migrate.test.ts`:
|
||||
|
||||
- Verifies `runPgliteMigrations` creates the BetterAuth tables (the original
|
||||
failure mode).
|
||||
- Idempotence (transitively re-runs migration 0009).
|
||||
- Partial-failure: pre-creates a conflicting `users` table, asserts the
|
||||
thrown error includes statement context (`hash=… statement #N failed`)
|
||||
and that no ledger row was written.
|
||||
|
||||
## QA evidence
|
||||
|
||||
End-to-end on a fresh PGlite install:
|
||||
|
||||
- `[DatabaseModule] Applying PGlite schema migrations...` then
|
||||
`Initializing storage adapter (pglite)...` in startup log.
|
||||
- `GET /api/bootstrap/status` → `{"needsSetup":true}` HTTP 200 (was 500
|
||||
with `relation "users" does not exist`).
|
||||
- `POST /api/bootstrap/setup` with empty body → HTTP 400 with Zod
|
||||
validation error (was 500), confirming the request reached the
|
||||
validator past the table-existence check.
|
||||
|
||||
## Known issues, out of scope (file separately)
|
||||
|
||||
- **Postgres-tier first install is still broken.** `runMigrations()` uses
|
||||
Drizzle's `migratePostgres`, which has the same outer-transaction problem
|
||||
as PGlite's migrator. A fresh standalone-tier install would also fail at
|
||||
migration 0009. Inline TODO in `migrate.ts:31-35` flags this. Fixing it
|
||||
needs either (a) a shared per-statement loop reused for both drivers, or
|
||||
(b) splitting migration 0009.
|
||||
- **`drizzle/meta/_journal.json` has 0009 ordered before 0008** (`when`
|
||||
values `1745280000000` < `1776822435828`). `migratePostgres` skips by
|
||||
`created_at < folderMillis`, so on a postgres deployment that already
|
||||
applied 0008, 0009 would be skipped forever. Our hash-based skip in the
|
||||
PGlite path sidesteps this.
|
||||
- **No advisory lock around the migration loop.** Two gateway processes
|
||||
pointed at the same DATABASE_URL would race. PGlite is single-process by
|
||||
file lock so the local tier is fine; postgres-tier deployments should add
|
||||
`pg_advisory_lock(<deterministic-id>)` around the loop in a follow-up.
|
||||
- **`mosaic storage migrate` CLI message is misleading** — it claims
|
||||
"automatic on first connection via adapter.migrate()" but the adapter
|
||||
doesn't self-migrate. With this PR the gateway invokes it explicitly, but
|
||||
the CLI message could still be tightened.
|
||||
- **Crash mid-migration leaves a partial-state PGlite DB without a ledger
|
||||
row.** Detected loudly on next boot (the replay errors on "already
|
||||
exists"), but recovery is manual (drop the partially-applied objects or
|
||||
insert the migration hash into `drizzle.__drizzle_migrations`). A robust
|
||||
fix would add a "started_at" column to a sidecar table to detect
|
||||
half-applied state and refuse to start with actionable guidance.
|
||||
Reference in New Issue
Block a user