Trent/vigilcare-clinical
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
9e88ff61137c30f68935ef04e3e1220de83542b6
vigilcare-clinical/docs/guides/01-docker-compose-orchestration.md
T
voltsrage 666d683d67 feature: Explainable Alerts
2026-06-25 00:25:31 +08:00

20 KiB
Raw Blame History

Guide 1: Docker Compose for Multi-Service Orchestration

What is Docker and Docker Compose?

Docker lets you run software inside isolated packages called containers. A container bundles an application with everything it needs — the operating system libraries, configuration files, and runtime — so it works the same on every machine. Instead of installing PostgreSQL on your laptop directly (and dealing with version conflicts, path issues, and OS differences), you run a PostgreSQL container that always behaves identically.

A Docker image is the blueprint; a container is a running instance of that image. You pull images from registries (like Docker Hub), and Docker runs them as containers.

Docker Compose is a tool that lets you define and run multiple containers together using a single YAML file (docker-compose.yml). You describe each service (which image, which ports, which settings) and Compose starts them all with one command: docker compose up.

Why Docker Compose?

VigilCareClinical runs 10+ infrastructure services (PostgreSQL, Redis, Kafka, Elasticsearch, RabbitMQ, MinIO, Prometheus, Grafana, Seq) alongside the application API. Docker Compose defines the entire stack in a single docker-compose.yml file, allowing a developer to bring up the full environment with one command. Without it, each service would need manual installation, port configuration, and startup ordering — a process that could take hours and differ across machines.

The Compose File Structure

The file lives at the project root: docker-compose.yml. It defines three top-level sections:

  • Services — each container you want to run (which image, which ports to expose, which environment variables to set, where to store data)
  • Volumes — named storage locations that persist data even when containers are stopped or deleted
  • Networks — a virtual network so containers can talk to each other
volumes:
  pg_data:
  seq_data:
  kafka_data:
  es_data:
  minio_data:
  prometheus_data:
  grafana_data:
  ward_pg_data:

networks:
  vigilcare_net:
    driver: bridge

What is a bridge network? A bridge network is a private network that Docker creates on your machine. Only containers attached to the same bridge can communicate with each other. Think of it like plugging all your services into the same network switch.

All services join vigilcare_net, which means inside any container, you can reach PostgreSQL at postgres:5432, Redis at redis:6379, etc. — Docker's built-in DNS resolves service names to container IPs, so you never need to know the actual IP address.

Service Definitions

Core Infrastructure

PostgreSQL 16 — Primary Database

postgres:
  image: postgres:16
  environment:
    POSTGRES_DB: vigilcare
    POSTGRES_USER: postgres
    POSTGRES_PASSWORD: password
  ports:
    - "5436:5432"
  volumes:
    - pg_data:/var/lib/postgresql/data
  networks:
    - vigilcare_net

Port mapping: The "5436:5432" syntax means "when I connect to port 5436 on my laptop (the host), forward it to port 5432 inside the container." Port 5432 is PostgreSQL's standard port. We use 5436 on the host side to avoid collisions if you already have PostgreSQL installed locally on 5432.

Named volume: pg_data is a storage location managed by Docker that persists database files across container restarts. Think of it as a hard drive for the container — without this, every time you stop the container with docker compose down, all your data would be deleted.

Redis 7 — Cache and State Store

redis:
  image: redis:7-alpine
  ports:
    - "6382:6379"
  networks:
    - vigilcare_net

Uses the alpine variant — Alpine Linux is a tiny Linux distribution, so the image is much smaller (~30MB vs ~130MB for the full image). Smaller images download faster and use less disk space. Port 6382 avoids collision with a local Redis on 6379.

Apache Kafka 3.7 — Event Streaming

kafka:
  image: apache/kafka:3.7.0
  ports:
    - "9092:9092"
  environment:
    KAFKA_NODE_ID: 1
    KAFKA_PROCESS_ROLES: broker,controller
    KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093
    KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
    KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT
    KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
    KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
    KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
    KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
    KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
    KAFKA_AUTO_CREATE_TOPICS_ENABLE: "false"
    CLUSTER_ID: "MkU3OEVBNTcwNTJENDM2Qk"
  volumes:
    - kafka_data:/var/lib/kafka/data
  networks:
    - vigilcare_net

Key design decisions:

  • KRaft mode (KAFKA_PROCESS_ROLES: broker,controller): Older versions of Kafka required a separate service called Zookeeper to coordinate the cluster. Kafka 3.7 can manage itself using a built-in protocol called KRaft (Kafka Raft). This means one less container to run and manage.
  • KAFKA_AUTO_CREATE_TOPICS_ENABLE: "false": Kafka organizes messages into topics (like channels or mailboxes). By default, Kafka auto-creates a topic the first time someone sends a message to it, but it creates it with only 1 partition (limiting parallelism). We disable this and instead create topics explicitly at application startup with the correct settings (6 partitions).
  • CLUSTER_ID: A fixed cluster ID prevents the broker from reinitializing its storage on restart, which would lose all topic data. Think of it like a serial number for the database files.
  • Replication factor = 1: In production, each message would be copied to 3+ Kafka nodes for redundancy. For local development with a single node, replication of 1 is fine.

Elasticsearch 8.13 — Search and Analytics

elasticsearch:
  image: docker.elastic.co/elasticsearch/elasticsearch:8.13.0
  environment:
    - discovery.type=single-node
    - xpack.security.enabled=false
    - ES_JAVA_OPTS=-Xms512m -Xmx512m
    - cluster.name=vigilcare-dev
  ports:
    - "9200:9200"
  volumes:
    - es_data:/usr/share/elasticsearch/data
  healthcheck:
    test: ["CMD", "curl", "-f", "http://localhost:9200/_cluster/health"]
    interval: 10s
    timeout: 5s
    retries: 10
  networks:
    - vigilcare_net
  • xpack.security.enabled=false: Disables TLS encryption and authentication for development. This makes it simpler to connect (no certificates or passwords needed). Production would enable this.
  • ES_JAVA_OPTS=-Xms512m -Xmx512m: Elasticsearch runs on the Java Virtual Machine (JVM). By default it can consume a lot of memory. This caps the JVM heap (working memory) at 512MB to prevent Elasticsearch from consuming all your system memory during development.
  • Health check: A health check is a command Docker runs periodically to verify the service is actually working, not just running. Here it hits the Elasticsearch health endpoint every 10 seconds. Other services that depend on Elasticsearch can use condition: service_healthy to wait until this check passes before starting.

RabbitMQ 3.13 — Notification Queuing

rabbitmq:
  image: rabbitmq:3.13-management-alpine
  container_name: vigilcare_rabbitmq
  ports:
    - "5674:5672"    # AMQP protocol
    - "15674:15672"  # Management UI
  environment:
    RABBITMQ_DEFAULT_USER: guest
    RABBITMQ_DEFAULT_PASS: guest
  healthcheck:
    test: ["CMD", "rabbitmq-diagnostics", "ping"]
    interval: 10s
    timeout: 5s
    retries: 5
  networks:
    - vigilcare_net

RabbitMQ is a message broker — it accepts, stores, and forwards messages between parts of your application (explained in detail in Guide 7). The management-alpine image variant includes a web-based management UI at http://localhost:15674 where you can see queues, messages, and connections. AMQP (Advanced Message Queuing Protocol) on port 5674 is the wire protocol applications use to talk to RabbitMQ. The health check uses rabbitmq-diagnostics ping rather than a simple TCP port check — this validates that the broker process is actually ready to accept connections, not just that the port is open.

MinIO — S3-Compatible Object Storage

minio:
  image: minio/minio:RELEASE.2024-07-04T14-25-45Z
  container_name: vigilcare_minio
  command: server /data --console-address ":9001"
  ports:
    - "9005:9000"    # S3 API
    - "9006:9001"    # Web console
  environment:
    MINIO_ROOT_USER: minioadmin
    MINIO_ROOT_PASSWORD: minioadmin
  volumes:
    - minio_data:/data
  networks:
    - vigilcare_net

MinIO is an open-source object storage server that speaks the same API as Amazon S3. This means you can develop locally against MinIO and deploy to AWS S3 later without changing your code. The command override tells MinIO to expose its web console on a separate port. Two ports: 9005 for the S3 API (your application code reads and writes files here), 9006 for the web console where you can browse stored files in your browser.

Seq — Log Aggregation

seq:
  image: datalust/seq:latest
  environment:
    ACCEPT_EULA: "Y"
    SEQ_FIRSTRUN_ADMINPASSWORD: "admin"
  ports:
    - "5345:80"
  volumes:
    - seq_data:/data
  networks:
    - vigilcare_net

Seq provides structured log search and filtering. The API runs on its default port 80 inside the container, mapped to 5345 externally. Serilog ships logs to http://localhost:5345.

Observability Services

Prometheus — Metrics Collection

prometheus:
  image: prom/prometheus:v2.52.0
  container_name: vigilcare_prometheus
  command:
    - "--config.file=/etc/prometheus/prometheus.yml"
    - "--storage.tsdb.retention.time=7d"
  ports:
    - "9101:9090"
  volumes:
    - ./infra/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
    - prometheus_data:/prometheus
  extra_hosts:
    - "host.docker.internal:host-gateway"
  networks:
    - vigilcare_net
  • extra_hosts: Containers live in their own private network. host.docker.internal:host-gateway adds a DNS entry so the Prometheus container can reach your laptop (the "host machine") where the .NET API runs. On macOS and Windows this mapping exists by default, but Linux needs it explicitly. The prometheus.yml config then uses host.docker.internal:5270 as the target to scrape metrics from.
  • --storage.tsdb.retention.time=7d: Keeps 7 days of metrics data (TSDB stands for Time-Series DataBase — Prometheus's internal storage format). This prevents disk usage from growing indefinitely during development.
  • Bind mount vs named volume: The :ro suffix means "read-only." A bind mount (./infra/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml) maps a specific file on your laptop directly into the container. Unlike a named volume (which Docker manages), you edit the file on your laptop and the container sees the changes immediately. The :ro flag means the container can read the file but cannot write to it.

Grafana — Dashboard Visualization

grafana:
  image: grafana/grafana:10.4.3
  container_name: vigilcare_grafana
  ports:
    - "3101:3000"
  environment:
    GF_SECURITY_ADMIN_USER: admin
    GF_SECURITY_ADMIN_PASSWORD: admin
    GF_USERS_ALLOW_SIGN_UP: "false"
  volumes:
    - ./infra/grafana/provisioning:/etc/grafana/provisioning:ro
    - ./infra/grafana/dashboards:/var/lib/grafana/dashboards:ro
    - grafana_data:/var/lib/grafana
  depends_on:
    - prometheus
  networks:
    - vigilcare_net
  • depends_on: prometheus: Grafana starts after Prometheus. Note that depends_on without a condition only controls start order, not readiness — Docker starts Grafana after starting (not after Prometheus is fully ready). Grafana won't crash if Prometheus takes a few seconds to start up, but the datasource won't return data until Prometheus is actually accepting connections.
  • Provisioning volumes: Two read-only bind mounts auto-configure Grafana on first start:
    • provisioning/ — datasource configuration pointing Grafana to http://prometheus:9090
    • dashboards/ — pre-built dashboard JSON files loaded automatically
  • grafana_data volume: Persists Grafana's internal database (saved queries, user preferences) across restarts.

Profiles: Optional Service Groups

The ward gateway services use Docker Compose profiles to avoid starting them during normal development:

ward-gateway-db:
  image: postgres:16
  profiles: ["ward-gateway", "full"]
  # ...

ward-gateway-redis:
  image: redis:7-alpine
  profiles: ["ward-gateway", "full"]
  # ...

ward-gateway-rabbitmq:
  image: rabbitmq:3.13-management-alpine
  profiles: ["ward-gateway", "full"]
  # ...

ward-gateway-api:
  profiles: ["ward-gateway", "full"]
  build:
    context: .
    dockerfile: VigilCare.WardGateway/Dockerfile
  # ...

Services without a profiles key start by default. Services with profiles only start when you explicitly request that profile:

# Start only the core stack (no gateway services)
docker compose up -d

# Start core + ward gateway
docker compose --profile ward-gateway up -d

# Start everything
docker compose --profile full up -d

This keeps the default development experience lightweight — 9 services instead of 13.

The Ward Gateway Service

The gateway is the only application service that runs inside Docker (the main API runs on the host via dotnet run). It uses a multi-stage Dockerfile:

FROM mcr.microsoft.com/dotnet/sdk:8.0 AS build
WORKDIR /src
COPY VigilCareClinical.sln ./
COPY VigilCare.ClinicalContracts/ VigilCare.ClinicalContracts/
COPY VigilCare.WardGateway/ VigilCare.WardGateway/
RUN dotnet restore VigilCare.WardGateway/VigilCare.WardGateway.csproj
RUN dotnet publish VigilCare.WardGateway/VigilCare.WardGateway.csproj \
    -c Release -o /app/publish --no-restore

FROM mcr.microsoft.com/dotnet/aspnet:8.0 AS runtime
WORKDIR /app
COPY --from=build /app/publish .
ENV ASPNETCORE_URLS=http://+:8080
EXPOSE 8080
ENTRYPOINT ["dotnet", "VigilCare.WardGateway.dll"]

What is a multi-stage build? A Dockerfile can have multiple FROM lines, each starting a new "stage." The first stage (AS build) uses the full .NET SDK image (~900MB) to compile the code. The second stage (AS runtime) starts fresh from the smaller ASP.NET runtime image (~220MB) and copies only the compiled output. The final image doesn't contain the SDK, source code, or NuGet package cache — it's lean and production-ready. This is like using a full workshop to build furniture, then shipping only the finished table to the customer.

Build context: Set to . (project root) because the gateway depends on VigilCare.ClinicalContracts — a shared library. The Dockerfile needs to COPY both projects into the build stage.

Gateway Environment Variables

The gateway container receives all its configuration through environment variables, overriding appsettings.json:

environment:
  ASPNETCORE_ENVIRONMENT: Development
  ConnectionStrings__GatewayDb: "Host=ward-gateway-db;Port=5432;..."
  Redis__ConnectionString: "ward-gateway-redis:6379"
  RabbitMq__Host: ward-gateway-rabbitmq
  CentralApi__BaseUrl: "http://host.docker.internal:5270"
  Gateway__GatewayId: "22222222-2222-2222-2222-222222222222"
  GATEWAY_SYNC_JWT: "${GATEWAY_SYNC_JWT:-}"

Key patterns:

  • Double underscore (__) maps to nested JSON keys. ASP.NET Core reads environment variables and translates __ into the : separator used in JSON. So ConnectionStrings__GatewayDb overrides {"ConnectionStrings": {"GatewayDb": "..."}} in appsettings.json.
  • host.docker.internal in CentralApi__BaseUrl: The gateway runs inside Docker, but the main API runs on your laptop. This special hostname lets the container reach back out to your laptop.
  • ${GATEWAY_SYNC_JWT:-}: This is shell variable substitution. Docker Compose reads the value of GATEWAY_SYNC_JWT from a .env file in the project root or from your host environment. The :- means "use an empty string as the default if the variable isn't set."

Gateway Dependency Conditions

depends_on:
  ward-gateway-db:
    condition: service_healthy
  ward-gateway-redis:
    condition: service_started
  ward-gateway-rabbitmq:
    condition: service_healthy
  • service_healthy waits for the health check to pass. Used for PostgreSQL and RabbitMQ which need time to initialize.
  • service_started only waits for the container process to start. Used for Redis which is ready almost immediately.

Health Checks

Three services have explicit health checks:

Service Check Command Interval Retries
Elasticsearch curl -f http://localhost:9200/_cluster/health 10s 10
RabbitMQ rabbitmq-diagnostics ping 10s 5
ward-gateway-db pg_isready -U postgres 10s 5

Health checks serve two purposes:

  1. Dependency ordering: depends_on with condition: service_healthy blocks until the check passes.
  2. Visibility: docker compose ps shows health status per container.

Services without health checks (Kafka, Redis, MinIO) start quickly enough that ordering isn't critical, or the application handles connection retries itself (e.g., ThresholdCacheLoader retries Redis 3 times with exponential backoff).

The extra_hosts Pattern

extra_hosts:
  - "host.docker.internal:host-gateway"

This appears on two services: Prometheus and ward-gateway-api. On Linux, Docker doesn't provide host.docker.internal by default (it does on macOS and Windows). The host-gateway magic value resolves to the host machine's IP on the Docker bridge network.

This is needed because the .NET API runs on the host (via dotnet run), not inside Docker. Prometheus needs to scrape host.docker.internal:5270/metrics, and the ward gateway needs to call host.docker.internal:5270 as the central API.

Port Mapping Summary

Service Host Port Container Port Purpose
PostgreSQL 5436 5432 Primary database
Redis 6382 6379 Cache/state
Kafka 9092 9092 Event streaming
Elasticsearch 9200 9200 Search
RabbitMQ 5674 5672 AMQP
RabbitMQ Management 15674 15672 Web UI
MinIO S3 9005 9000 Object storage API
MinIO Console 9006 9001 Web UI
Prometheus 9101 9090 Metrics
Grafana 3101 3000 Dashboards
Seq 5345 80 Log aggregation
Ward Gateway DB 5437 5432 Gateway database
Ward Gateway Redis 6383 6379 Gateway cache
Ward Gateway RabbitMQ 5675 5672 Gateway queues
Ward Gateway API 5081 8080 Gateway HTTP

All host ports are intentionally non-standard to avoid collisions with locally installed services.

Common Commands

# Start the core stack (background)
docker compose up -d

# Start with ward gateway
docker compose --profile ward-gateway up -d

# View running containers and health status
docker compose ps

# View logs for a specific service
docker compose logs -f kafka

# Restart a single service
docker compose restart elasticsearch

# Stop everything but keep volumes (data preserved)
docker compose down

# Stop everything and delete volumes (clean slate)
docker compose down -v

# Rebuild the ward gateway image after code changes
docker compose --profile ward-gateway build ward-gateway-api
docker compose --profile ward-gateway up -d ward-gateway-api

How the API Connects

The API runs on the host machine (dotnet run), not in Docker. It connects to containerized services using the host ports defined in the compose file. These are configured in appsettings.json:

{
  "ConnectionStrings": {
    "DefaultConnection": "Host=localhost;Port=5436;Database=vigilcare;..."
  },
  "Redis": { "ConnectionString": "localhost:6382" },
  "Kafka": { "BootstrapServers": "localhost:9092" },
  "Elasticsearch": { "Uri": "http://localhost:9200" },
  "RabbitMq": { "Host": "localhost", "Port": 5674 },
  "Minio": { "Endpoint": "localhost:9005" },
  "Seq": { "ServerUrl": "http://localhost:5345" }
}

This split — infrastructure in Docker, application on the host — gives the best development experience: fast iteration on application code (no rebuild/restart cycle) with production-like infrastructure.

Reference in New Issue View Git Blame Copy Permalink