stack/docs/COOLIFY-DEPLOYMENT.md

Mosaic Stack — Coolify Deployment

Overview

Coolify deployment on VM 10.1.1.44 (Proxmox). Replaces the Docker Swarm deployment on w-docker0 (10.1.1.45).

Architecture

Internet → Cloudflare → Public IP (174.137.97.162)
  → Main Traefik (10.1.1.43) — TCP TLS passthrough for *.woltje.com
    → Coolify Traefik (10.1.1.44) — terminates TLS via Cloudflare DNS-01 wildcard certs
      → Service containers

Services (Core Stack)

Service	Image	Internal Port	External Domain
postgres	git.mosaicstack.dev/mosaic/stack-postgres	5432	—
valkey	valkey/valkey:8-alpine	6379	—
api	git.mosaicstack.dev/mosaic/stack-api	3001	api.mosaic.woltje.com
web	git.mosaicstack.dev/mosaic/stack-web	3000	mosaic.woltje.com
coordinator	git.mosaicstack.dev/mosaic/stack-coordinator	8000	—
orchestrator	git.mosaicstack.dev/mosaic/stack-orchestrator	3001 (internal)	—

Matrix (synapse, element-web) and speech services (speaches, kokoro-tts) are NOT included in the core stack. Deploy separately if needed.

Compose File

docker-compose.coolify.yml in the repo root. This is the Coolify-compatible version of the deployment compose.

Key differences from the Swarm compose (docker-compose.swarm.portainer.yml):

• No deploy: blocks (Swarm-only)
• No Traefik labels (Coolify manages routing)
• Bridge network instead of overlay
• restart: unless-stopped instead of Swarm restart policies
• SERVICE_FQDN_* magic environment variables for Coolify domain assignment
• List-style environment syntax (required for Coolify magic vars)

Coolify IDs

Resource	UUID
Project	rs04g008kgkkw4s0wgsk40w4
Environment	gko8csc804g8og0oosc8ccs8
Service	ug0ssok4g44wocok8kws8gg8
Server	as8kcogk08skskkcsok888g4

Application UUIDs

App	UUID
postgres	jcw0ogskkw040os48ggkgkc8
valkey	skssgwcggc0c8owoogcso8og
api	mc40cgwwo8okwwoko84408k4k
web	c48gcwgc40ok44scscowc8cc
coordinator	s8gwog4c44w08c8sgkcg04k8
orchestrator	uo4wkg88co0ckc4c4k44sowc

Coolify API

Base URL: http://10.1.1.44:8000/api/v1 Auth: Bearer token from credentials.json → coolify.app_token

Patterns & Gotchas

• Compose must be base64-encoded when sending via docker_compose_raw field
• SERVICE_FQDN_* magic vars: Coolify reads these from the compose to auto-assign domains. Format: SERVICE_FQDN_{NAME}_{PORT} (e.g., SERVICE_FQDN_API_3001). Must use list-style env syntax (- SERVICE_FQDN_API_3001), NOT dict-style.
• FQDN updates on sub-applications: Coolify API doesn't support updating FQDNs on compose service sub-apps via REST. Workaround: update directly in Coolify's PostgreSQL DB (coolify-db container, service_applications table).
• Environment variable management: Use PATCH /api/v1/services/{uuid}/envs with { "key": "VAR_NAME", "value": "val", "is_preview": false }
• Service start: POST /api/v1/services/{uuid}/start
• Coolify uses PostgreSQL (not SQLite) for its internal database — container coolify-db

DB Access (for workarounds)

ssh localadmin@10.1.1.44
docker exec -it coolify-db psql -U coolify -d coolify

-- Check service app FQDNs
SELECT name, fqdn FROM service_applications WHERE service_id = (
  SELECT id FROM services WHERE uuid = 'ug0ssok4g44wocok8kws8gg8'
);

Environment Variables

All env vars are set via Coolify API and stored in /data/coolify/services/{uuid}/.env on the node.

Critical vars that were missing initially:

• BETTER_AUTH_URL — Required in production. API won't start without it. Set to https://api.mosaic.woltje.com.

Current State (2026-02-22)

Working

• All 6 containers running and healthy
• API health endpoint responds at https://api.mosaic.woltje.com/health
• Database migrations completed
• Inter-service networking (api→postgres, api→valkey) confirmed via health checks

Issues

1. DNS: mosaic.woltje.com points to wrong server

   • Resolves to 10.1.1.45 (old Swarm node) instead of through Cloudflare (174.137.97.162)
   • api.mosaic.woltje.com resolves correctly through Cloudflare
   • Fix: Update Cloudflare DNS A record for mosaic.woltje.com

2. Coordinator: OTLP exporter noise

   • Trying to export traces to localhost:4318 which doesn't exist
   • Container is healthy, errors are non-critical
   • Fix: Set MOSAIC_TELEMETRY_ENABLED=false in Coolify env vars, or deploy an OTLP collector

3. Coolify managed lifecycle

   • CoolifyTask was failing when starting the service via API/UI
   • Containers were started manually via docker compose up -d from the service directory
   • Coolify recognizes the containers (correct naming convention) but may not properly manage restarts/redeploys
   • Needs investigation: check Coolify task logs, verify compose processing

4. Full connectivity verification needed

   • web→api communication untested (blocked by DNS issue)
   • Orchestrator→valkey and orchestrator→api connectivity unverified
   • Coordinator webhook endpoint untested

SSH Access

ssh localadmin@10.1.1.44
# Note: localadmin cannot sudo without TTY/password
# Use docker to access files:
docker run --rm -v /data/coolify/services:/srv alpine cat /srv/{uuid}/docker-compose.yml
# Use docker exec for Coolify DB:
docker exec -it coolify-db psql -U coolify -d coolify