diff --git a/docs/guides/25-gitea-cicd-docker-deploy.md b/docs/guides/25-gitea-cicd-docker-deploy.md new file mode 100644 index 0000000..ca007a3 --- /dev/null +++ b/docs/guides/25-gitea-cicd-docker-deploy.md @@ -0,0 +1,417 @@ +# Guide 25: Gitea CI/CD with Docker Compose + +How to wire a multi-service app for continuous integration and deployment on **Gitea Actions**, using VigilCare Clinical as a concrete example. The same layout works for any stack: keep local infra in one Compose file, ship only app images in production, put secrets on the host (not in git), and let workflows build, migrate, and deploy on version tags. + +Related docs in this repo: + +- [`.gitea/workflows/ci.yml`](../../.gitea/workflows/ci.yml) / [`.gitea/workflows/cd.yml`](../../.gitea/workflows/cd.yml) +- [`docker-compose.yml`](../../docker-compose.yml) (local + CI dependencies) +- [`docker-compose.prod.yml`](../../docker-compose.prod.yml) (production app stack) +- [`.env.example`](../../.env.example) +- [`docs/ops/cd-deploy-ssh-setup.md`](../ops/cd-deploy-ssh-setup.md) +- [`docs/instructions-for-env.md`](../instructions-for-env.md) + +--- + +## Mental model + +| Layer | What it is | Who owns it | +|-------|------------|-------------| +| **Source** | App code, Dockerfiles, Compose files, workflow YAML | Git repo | +| **CI** | On every push/PR: start deps, build, test | Gitea Actions + `act_runner` | +| **CD** | On `v*` tags: build/push images, migrate DB, SSH deploy | Same runner + container registry | +| **Secrets on host** | Production `.env` with DB URLs, JWT keys, etc. | Deploy VM only (never committed, never SCP’d by CD) | +| **Runtime** | Pulled images + Compose prod overlay | Deploy VM (`/opt//`) | + +``` +Developer push/PR ──► CI (compose deps + tests) +Developer git tag v1.2.3 ──► CD + ├─ build & push images → Gitea registry + ├─ apply EF migrations (DDL user) + └─ SSH → pull images, up -d, smoke test +``` + +Gitea Actions is largely compatible with GitHub Actions syntax (`on:`, `jobs:`, `uses: actions/checkout@v4`, etc.). Jobs run on a self-hosted **act_runner** that needs Docker, curl, ssh, scp, and bash, labeled `ubuntu-latest` (or whatever label you set in the workflow). + +--- + +## 1. Split Compose: local/CI vs production + +Do **not** reuse the same Compose file for laptop and production. + +### Local / CI — `docker-compose.yml` + +Defines **infrastructure** (Postgres, Redis, Kafka, RabbitMQ, Elasticsearch, MinIO, …) and optional app profiles. Developers and CI start only what tests need: + +```bash +docker compose up -d postgres redis rabbitmq kafka elasticsearch minio +docker compose --profile ward-gateway up -d ward-gateway-db ward-gateway-redis ward-gateway-rabbitmq +``` + +Useful patterns (as in this repo): + +- **Published ports** so host processes (or CI job containers via `host.docker.internal`) can reach brokers. +- **Profiles** (`ward-gateway`, `full`) so optional services stay off by default. +- **CI-friendly Kafka advertising** — override advertised host for runners: + +```yaml +KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka:29092,EXTERNAL://${KAFKA_EXTERNAL_HOST:-localhost}:9092 +``` + +CI sets `KAFKA_EXTERNAL_HOST: host.docker.internal` so the test process in a container reconnects to the published port on the Docker host, not to itself. + +### Production — `docker-compose.prod.yml` + +Runs **only the deployable apps** (here: `api`, `gateway`, `dashboard`). Databases and brokers are assumed to already exist; Compose wires them through environment variables from `.env`. + +```yaml +services: + api: + image: ${REGISTRY}/clinical-api:${IMAGE_TAG} + environment: + ConnectionStrings__DefaultConnection: "${PG_CONNECTION}" + # … Jwt, Kafka, Redis, etc. from .env + volumes: + - dp_keys:/app/data-protection-keys # durable secrets / keyrings + networks: + - vigilcare_prod + - shared-services # external network owned by infra compose +``` + +Principles: + +1. **Image coordinates** via `REGISTRY` + `IMAGE_TAG` — CD updates only `IMAGE_TAG`. +2. **External networks** for shared Postgres/Redis stacks already running on the host. +3. **No build:** on the VM — `docker compose pull` then `up -d`. +4. **Named volumes** for anything that must survive recreate (e.g. Data Protection keys). + +CD copies **only** this file to the server each release; it does not copy `.env`. + +--- + +## 2. Dockerfiles that CI and CD can trust + +### Multi-stage builds + +Keep a **SDK/build** stage and a thin **runtime** stage. Example (API): + +```dockerfile +FROM mcr.microsoft.com/dotnet/sdk:8.0 AS src +# restore with project files only → cache NuGet layer +# then COPY source, publish + +FROM mcr.microsoft.com/dotnet/aspnet:8.0 AS runtime +COPY --from=build /app/publish . +USER app +HEALTHCHECK CMD curl -fsS http://localhost:8080/health/live || exit 1 +ENTRYPOINT ["dotnet", "VigilCareClinicalAPI.dll"] +``` + +Frontend (Vite) must bake public API URLs at **build** time: + +```dockerfile +ARG VITE_API_URL +ENV VITE_API_URL=$VITE_API_URL +RUN npm run build +``` + +CD passes `--build-arg VITE_API_URL="${{ vars.PROD_API_URL }}"`. + +### Build context + +Match Compose and CD: + +| Image | Dockerfile | Context | Why | +|-------|------------|---------|-----| +| clinical-api | `VigilCareClinicalAPI/Dockerfile` | **repo root** | Sibling `ProjectReference`s | +| ward-gateway | `VigilCare.WardGateway/Dockerfile` | **repo root** | Same | +| dashboard | `vigilcare-dashboard/Dockerfile` | `vigilcare-dashboard/` | `package.json`, `nginx.conf` | + +Wrong context is the most common “works on my machine, fails in CD” failure. + +### Scrub secrets from published config + +Dev `appsettings.json` often contains placeholder keys. Strip them in the image so production **must** supply env vars: + +```dockerfile +RUN sed -i \ + -e 's/"SigningKey": "[^"]*"/"SigningKey": ""/' \ + -e 's/"Gateway": "dev-[^"]*"/"Gateway": ""/' \ + /app/publish/appsettings.json +``` + +### Optional: migration target in the same Dockerfile + +```dockerfile +FROM src AS migrate +RUN dotnet ef migrations bundle ... --output /out/migrate-api +``` + +CD builds `--target migrate`, copies the binary out, and runs it with a **DDL** connection string that never enters the API container. Prefer `docker build` + `docker cp` over `docker run -v` on act_runner — bind mounts resolve on the Docker host, not the job workspace. + +### `.dockerignore` + +Exclude `bin/`, `obj/`, `node_modules/`, tests, `.env`, docs noise. Keep anything the image must ship (e.g. scenario JSON under `VigilCare.Simulator/Scenarios/`). + +--- + +## 3. `.env.example` vs production `.env` + +| File | In git? | Purpose | +|------|---------|---------| +| `.env.example` | Yes | Document every key; safe placeholders | +| `.env` (laptop) | No (`.gitignore`) | Local experimentation only | +| `/opt/vigilcare/.env` on VM | No | **Source of truth** for production | + +`.gitignore` pattern used here: + +``` +.env +.env.* +!.env.example +``` + +Group keys clearly in `.env.example`: + +1. `REGISTRY` / `IMAGE_TAG` +2. Host ports (`API_PORT`, …) +3. External service connection strings +4. App secrets (`JWT_SIGNING_KEY`, API keys) — generate with `openssl rand -base64 48` +5. Bootstrap / seed users + +**Privilege split:** runtime `PG_CONNECTION` (DML app user) vs `PG_CONNECTION_DDL` (migrator only). The DDL string is a Gitea secret for the migrate job, not an API container env var. + +### One-time place `.env` on the VM + +CD never uploads `.env`. Before the first tag deploy: + +```bash +ssh deploy@YOUR_HOST "mkdir -p /opt/vigilcare" +scp .env deploy@YOUR_HOST:/opt/vigilcare/.env +ssh deploy@YOUR_HOST "chmod 600 /opt/vigilcare/.env" +``` + +Later secret rotations: edit `/opt/vigilcare/.env` on the server (or replace your secret-management process). See [`docs/instructions-for-env.md`](../instructions-for-env.md). + +--- + +## 4. Gitea Actions — CI workflow + +Path: `.gitea/workflows/ci.yml` + +### Triggers + +```yaml +on: + push: + branches: [master] + pull_request: + branches: [master] +``` + +### Backend job pattern + +1. Checkout +2. `docker compose up -d` for dependencies +3. Wait loops (`pg_isready`, Kafka broker API, Redis `PING`, RabbitMQ diagnostics as the `rabbitmq` user — avoid root creating a bad `.erlang.cookie`) +4. Create test databases +5. `dotnet restore` / `build` / `test` with connection env vars pointing at `host.docker.internal` and published ports +6. Upload test artifacts; always tear down with `docker compose ... down -v` + +### Frontend job pattern + +```yaml +frontend: + runs-on: ubuntu-latest + container: + image: node:22-alpine + steps: + - uses: actions/checkout@v4 + - run: npm ci && npm run test && npm run build + working-directory: vigilcare-dashboard +``` + +### Runner requirement for Compose-based tests + +The job (or its Docker sibling) must reach published ports. On Docker Desktop / many Linux runners, that means `host.docker.internal` and runner `extra_hosts: host-gateway`. Wire that into your act_runner config if tests hang on “connection refused”. + +--- + +## 5. Gitea Actions — CD workflow + +Path: `.gitea/workflows/cd.yml` + +### Triggers + +```yaml +on: + push: + tags: ["v*"] + workflow_dispatch: + inputs: + image_tag: + description: "Image tag to deploy" + required: false +``` + +Release flow: merge to `master` → `git tag v1.2.3 && git push origin v1.2.3`. + +### Jobs + +``` +build-and-push ──► migrate ──► deploy (smoke + rollback on failure) +``` + +**build-and-push** + +1. Resolve tag (`GITHUB_REF_NAME` or `workflow_dispatch` input) and optional `vars.REGISTRY` +2. `docker login` to the Gitea package registry with `REGISTRY_USERNAME` / `REGISTRY_TOKEN` +3. Build each image; tag both `:v1.2.3` and `:latest`; push + +**migrate** + +1. Build `--target migrate` +2. Extract `migrate-api` +3. `./migrate-api --connection "${{ secrets.PG_CONNECTION_DDL }}"` + +Migrations must be **backwards-compatible** with the still-running previous image (expand-then-contract). Rollback restores the old image tag only — it does not reverse schema. + +**deploy** + +1. Configure SSH from `DEPLOY_SSH_KEY` (see [`cd-deploy-ssh-setup.md`](../ops/cd-deploy-ssh-setup.md)) +2. `scp docker-compose.prod.yml` → `/opt/vigilcare/` +3. On the host: save previous `IMAGE_TAG`, set new tag in `.env`, `pull`, `up -d`, prune dangling images +4. Smoke: curl `/health/ready`, gateway live, dashboard `/` +5. On failure: restore `.env.previous` and `up -d` again + +Do not `source .env` in bash — Compose env files are not shell (semicolons in connection strings, spaces, CRLF). Read individual keys with `sed` if needed. + +--- + +## 6. Gitea secrets and variables + +Repo → **Settings** → **Actions** + +### Secrets (sensitive) + +| Secret | Used by | +|--------|---------| +| `REGISTRY_USERNAME` | `docker login` | +| `REGISTRY_TOKEN` | `docker login` (access token / PAT with package write) | +| `PG_CONNECTION_DDL` | migrate job only | +| `DEPLOY_HOST` | SSH / SCP | +| `DEPLOY_USER` | SSH / SCP | +| `DEPLOY_SSH_KEY` | Private key PEM / OpenSSH private key body | + +### Variables (non-secret config) + +| Variable | Used by | +|----------|---------| +| `PROD_API_URL` | Dashboard image build-arg (public API origin) | +| `REGISTRY` | Optional override of default `gitea.example.com/org` | + +Enable **Packages** (container registry) for the org/user that owns `REGISTRY`. + +--- + +## 7. One-time infrastructure checklist + +Use this when cloning the pattern onto a new project or a fresh Gitea instance. + +### Gitea + runner + +- [ ] Gitea with Actions enabled +- [ ] act_runner registered, label matches `runs-on:` (e.g. `ubuntu-latest`) +- [ ] Runner can run Docker (socket or DinD) and has `docker compose`, `ssh`, `scp`, `curl`, `bash` +- [ ] Container registry reachable from runner and from the deploy host + +### Repo layout + +- [ ] `.gitea/workflows/ci.yml` and `cd.yml` +- [ ] Dockerfile(s) with runtime HEALTHCHECK +- [ ] `docker-compose.yml` for local/CI deps +- [ ] `docker-compose.prod.yml` for app-only deploy +- [ ] `.env.example` + `.gitignore` excluding `.env` +- [ ] `.dockerignore` with correct exceptions + +### Deploy host + +- [ ] User with Docker rights and home for SSH keys +- [ ] Directory e.g. `/opt//` with filled `.env` (`chmod 600`) +- [ ] External networks / infra Compose already up if prod overlay declares `external: true` +- [ ] Public DNS / reverse proxy pointing at published ports +- [ ] Deploy SSH key installed ([setup guide](../ops/cd-deploy-ssh-setup.md)) + +### First release + +- [ ] Secrets and variables set in Gitea +- [ ] CI green on `master` +- [ ] Tag `v0.1.0` (or `workflow_dispatch` with `image_tag`) +- [ ] Confirm images in registry, migrate succeeded, smoke passed + +--- + +## 8. Adapting this to another project + +Strip VigilCare-specific names; keep the skeleton: + +1. **CI:** start your test deps → wait healthy → run unit/integration tests with host-reachable URLs. +2. **CD build:** one `docker build`/`push` per deployable service; fix contexts. +3. **CD migrate:** optional job if you have schema (EF bundle, Flyway, Prisma migrate, etc.) using a privileged secret. +4. **CD deploy:** SCP prod Compose → patch `IMAGE_TAG` → pull/up → health curl → rollback tag on failure. +5. **Host `.env`:** all runtime config; CD touches only the image tag line. + +Minimal prod Compose for a single API: + +```yaml +services: + api: + image: ${REGISTRY}/my-api:${IMAGE_TAG} + ports: + - "${API_PORT:-8080}:8080" + environment: + ConnectionStrings__Default: "${PG_CONNECTION}" + restart: unless-stopped +``` + +Minimal CD deploy fragment: + +```bash +sed -i "s|^IMAGE_TAG=.*|IMAGE_TAG=${IMAGE_TAG}|" .env +docker compose -f docker-compose.prod.yml --env-file .env pull +docker compose -f docker-compose.prod.yml --env-file .env up -d --remove-orphans +``` + +--- + +## 9. Common pitfalls + +| Symptom | Likely cause | +|---------|----------------| +| Kafka / broker tests hang in CI | Advertised listener still `localhost` inside a job container — set `KAFKA_EXTERNAL_HOST=host.docker.internal` | +| RabbitMQ dies after health probe | Probing as root created a root-owned `.erlang.cookie` — probe as `rabbitmq` | +| `docker build` missing project references | Context not repo root | +| Dashboard calls wrong API in prod | Forgot `PROD_API_URL` / `VITE_*` build-arg | +| First CD fails at `cd /opt/...` | `.env` / directory never created on VM | +| Rollback “worked” but app errors | Schema already migrated; ensure expand-then-contract migrations | +| `source .env` breaks deploy scripts | Connection strings aren’t valid bash — don’t source Compose env files | + +--- + +## 10. Day-to-day commands + +```bash +# Local deps +docker compose up -d + +# Production-shaped run (on the VM, after images exist) +docker compose -f docker-compose.prod.yml --env-file .env up -d + +# Cut a release (after CI is green) +git tag v1.4.0 +git push origin v1.4.0 + +# Manual redeploy of an existing tag (Gitea UI → Actions → CD → Run workflow) +``` + +That is the full loop this repository uses: Compose for deps and for prod apps, env files only on the host, Gitea workflows for test and tagged releases. diff --git a/docs/technology-guides-index.md b/docs/technology-guides-index.md index acd53ef..a42f1c0 100644 --- a/docs/technology-guides-index.md +++ b/docs/technology-guides-index.md @@ -15,6 +15,9 @@ Custom metric families (counters, histograms, gauges), third-party collectors, d ### 3. Structured Logging with Serilog + Seq Enrichers (correlation ID, machine name, thread ID), multiple sinks (console, Seq), structured property filtering, and centralized log search for distributed services. +### Gitea CI/CD with Docker Compose +Local vs production Compose split, multi-stage Dockerfiles, `.env` on the deploy host (not in CD), Gitea Actions CI (deps + tests) and CD (registry push, migrations, SSH deploy, smoke test, tag rollback). Full walkthrough: [`docs/guides/25-gitea-cicd-docker-deploy.md`](guides/25-gitea-cicd-docker-deploy.md). + --- ## Data Layer