# Guide 1: Docker Compose for Multi-Service Orchestration

## What is Docker and Docker Compose?

**Docker** lets you run software inside isolated packages called **containers**. A container bundles an application with everything it needs — the operating system libraries, configuration files, and runtime — so it works the same on every machine. Instead of installing PostgreSQL on your laptop directly (and dealing with version conflicts, path issues, and OS differences), you run a PostgreSQL container that always behaves identically.

A Docker **image** is the blueprint; a **container** is a running instance of that image. You pull images from registries (like Docker Hub), and Docker runs them as containers.

**Docker Compose** is a tool that lets you define and run **multiple containers together** using a single YAML file (`docker-compose.yml`). You describe each service (which image, which ports, which settings) and Compose starts them all with one command: `docker compose up`.

---

## Why Docker Compose?

VigilCareClinical runs 10+ infrastructure services (PostgreSQL, Redis, Kafka, Elasticsearch, RabbitMQ, MinIO, Prometheus, Grafana, Seq) alongside the application API. Docker Compose defines the entire stack in a single `docker-compose.yml` file, allowing a developer to bring up the full environment with one command. Without it, each service would need manual installation, port configuration, and startup ordering — a process that could take hours and differ across machines.

---

## The Compose File Structure

The file lives at the project root: `docker-compose.yml`. It defines three top-level sections:
- **Services** — each container you want to run (which image, which ports to expose, which environment variables to set, where to store data)
- **Volumes** — named storage locations that persist data even when containers are stopped or deleted
- **Networks** — a virtual network so containers can talk to each other

```yaml
volumes:
  pg_data:
  seq_data:
  kafka_data:
  es_data:
  minio_data:
  prometheus_data:
  grafana_data:
  ward_pg_data:

networks:
  vigilcare_net:
    driver: bridge
```

**What is a bridge network?** A bridge network is a private network that Docker creates on your machine. Only containers attached to the same bridge can communicate with each other. Think of it like plugging all your services into the same network switch.

All services join `vigilcare_net`, which means inside any container, you can reach PostgreSQL at `postgres:5432`, Redis at `redis:6379`, etc. — Docker's built-in DNS resolves service names to container IPs, so you never need to know the actual IP address.

---

## Service Definitions

### Core Infrastructure

#### PostgreSQL 16 — Primary Database
```yaml
postgres:
  image: postgres:16
  environment:
    POSTGRES_DB: vigilcare
    POSTGRES_USER: postgres
    POSTGRES_PASSWORD: password
  ports:
    - "5436:5432"
  volumes:
    - pg_data:/var/lib/postgresql/data
  networks:
    - vigilcare_net
```

**Port mapping**: The `"5436:5432"` syntax means "when I connect to port 5436 on my laptop (the host), forward it to port 5432 inside the container." Port 5432 is PostgreSQL's standard port. We use 5436 on the host side to avoid collisions if you already have PostgreSQL installed locally on 5432.

**Named volume**: `pg_data` is a storage location managed by Docker that persists database files across container restarts. Think of it as a hard drive for the container — without this, every time you stop the container with `docker compose down`, all your data would be deleted.

#### Redis 7 — Cache and State Store
```yaml
redis:
  image: redis:7-alpine
  ports:
    - "6382:6379"
  networks:
    - vigilcare_net
```

Uses the `alpine` variant — Alpine Linux is a tiny Linux distribution, so the image is much smaller (~30MB vs ~130MB for the full image). Smaller images download faster and use less disk space. Port 6382 avoids collision with a local Redis on 6379.

#### Apache Kafka 3.7 — Event Streaming
```yaml
kafka:
  image: apache/kafka:3.7.0
  ports:
    - "9092:9092"
  environment:
    KAFKA_NODE_ID: 1
    KAFKA_PROCESS_ROLES: broker,controller
    KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093
    KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
    KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT
    KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
    KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
    KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
    KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
    KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
    KAFKA_AUTO_CREATE_TOPICS_ENABLE: "false"
    CLUSTER_ID: "MkU3OEVBNTcwNTJENDM2Qk"
  volumes:
    - kafka_data:/var/lib/kafka/data
  networks:
    - vigilcare_net
```

Key design decisions:

- **KRaft mode** (`KAFKA_PROCESS_ROLES: broker,controller`): Older versions of Kafka required a separate service called Zookeeper to coordinate the cluster. Kafka 3.7 can manage itself using a built-in protocol called KRaft (Kafka Raft). This means one less container to run and manage.
- **`KAFKA_AUTO_CREATE_TOPICS_ENABLE: "false"`**: Kafka organizes messages into **topics** (like channels or mailboxes). By default, Kafka auto-creates a topic the first time someone sends a message to it, but it creates it with only 1 partition (limiting parallelism). We disable this and instead create topics explicitly at application startup with the correct settings (6 partitions).
- **`CLUSTER_ID`**: A fixed cluster ID prevents the broker from reinitializing its storage on restart, which would lose all topic data. Think of it like a serial number for the database files.
- **Replication factor = 1**: In production, each message would be copied to 3+ Kafka nodes for redundancy. For local development with a single node, replication of 1 is fine.

#### Elasticsearch 8.13 — Search and Analytics
```yaml
elasticsearch:
  image: docker.elastic.co/elasticsearch/elasticsearch:8.13.0
  environment:
    - discovery.type=single-node
    - xpack.security.enabled=false
    - ES_JAVA_OPTS=-Xms512m -Xmx512m
    - cluster.name=vigilcare-dev
  ports:
    - "9200:9200"
  volumes:
    - es_data:/usr/share/elasticsearch/data
  healthcheck:
    test: ["CMD", "curl", "-f", "http://localhost:9200/_cluster/health"]
    interval: 10s
    timeout: 5s
    retries: 10
  networks:
    - vigilcare_net
```

- **`xpack.security.enabled=false`**: Disables TLS encryption and authentication for development. This makes it simpler to connect (no certificates or passwords needed). Production would enable this.
- **`ES_JAVA_OPTS=-Xms512m -Xmx512m`**: Elasticsearch runs on the Java Virtual Machine (JVM). By default it can consume a lot of memory. This caps the JVM heap (working memory) at 512MB to prevent Elasticsearch from consuming all your system memory during development.
- **Health check**: A health check is a command Docker runs periodically to verify the service is actually working, not just running. Here it hits the Elasticsearch health endpoint every 10 seconds. Other services that depend on Elasticsearch can use `condition: service_healthy` to wait until this check passes before starting.

#### RabbitMQ 3.13 — Notification Queuing
```yaml
rabbitmq:
  image: rabbitmq:3.13-management-alpine
  container_name: vigilcare_rabbitmq
  ports:
    - "5674:5672"    # AMQP protocol
    - "15674:15672"  # Management UI
  environment:
    RABBITMQ_DEFAULT_USER: guest
    RABBITMQ_DEFAULT_PASS: guest
  healthcheck:
    test: ["CMD", "rabbitmq-diagnostics", "ping"]
    interval: 10s
    timeout: 5s
    retries: 5
  networks:
    - vigilcare_net
```

RabbitMQ is a message broker — it accepts, stores, and forwards messages between parts of your application (explained in detail in Guide 7). The `management-alpine` image variant includes a web-based management UI at `http://localhost:15674` where you can see queues, messages, and connections. AMQP (Advanced Message Queuing Protocol) on port 5674 is the wire protocol applications use to talk to RabbitMQ. The health check uses `rabbitmq-diagnostics ping` rather than a simple TCP port check — this validates that the broker process is actually ready to accept connections, not just that the port is open.

#### MinIO — S3-Compatible Object Storage
```yaml
minio:
  image: minio/minio:RELEASE.2024-07-04T14-25-45Z
  container_name: vigilcare_minio
  command: server /data --console-address ":9001"
  ports:
    - "9005:9000"    # S3 API
    - "9006:9001"    # Web console
  environment:
    MINIO_ROOT_USER: minioadmin
    MINIO_ROOT_PASSWORD: minioadmin
  volumes:
    - minio_data:/data
  networks:
    - vigilcare_net
```

MinIO is an open-source object storage server that speaks the same API as Amazon S3. This means you can develop locally against MinIO and deploy to AWS S3 later without changing your code. The `command` override tells MinIO to expose its web console on a separate port. Two ports: 9005 for the S3 API (your application code reads and writes files here), 9006 for the web console where you can browse stored files in your browser.

#### Seq — Log Aggregation
```yaml
seq:
  image: datalust/seq:latest
  environment:
    ACCEPT_EULA: "Y"
    SEQ_FIRSTRUN_ADMINPASSWORD: "admin"
  ports:
    - "5345:80"
  volumes:
    - seq_data:/data
  networks:
    - vigilcare_net
```

Seq provides structured log search and filtering. The API runs on its default port 80 inside the container, mapped to 5345 externally. Serilog ships logs to `http://localhost:5345`.

### Observability Services

#### Prometheus — Metrics Collection
```yaml
prometheus:
  image: prom/prometheus:v2.52.0
  container_name: vigilcare_prometheus
  command:
    - "--config.file=/etc/prometheus/prometheus.yml"
    - "--storage.tsdb.retention.time=7d"
  ports:
    - "9101:9090"
  volumes:
    - ./infra/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
    - prometheus_data:/prometheus
  extra_hosts:
    - "host.docker.internal:host-gateway"
  networks:
    - vigilcare_net
```

- **`extra_hosts`**: Containers live in their own private network. `host.docker.internal:host-gateway` adds a DNS entry so the Prometheus container can reach your laptop (the "host machine") where the .NET API runs. On macOS and Windows this mapping exists by default, but Linux needs it explicitly. The prometheus.yml config then uses `host.docker.internal:5270` as the target to scrape metrics from.
- **`--storage.tsdb.retention.time=7d`**: Keeps 7 days of metrics data (TSDB stands for Time-Series DataBase — Prometheus's internal storage format). This prevents disk usage from growing indefinitely during development.
- **Bind mount vs named volume**: The `:ro` suffix means "read-only." A **bind mount** (`./infra/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml`) maps a specific file on your laptop directly into the container. Unlike a named volume (which Docker manages), you edit the file on your laptop and the container sees the changes immediately. The `:ro` flag means the container can read the file but cannot write to it.

#### Grafana — Dashboard Visualization
```yaml
grafana:
  image: grafana/grafana:10.4.3
  container_name: vigilcare_grafana
  ports:
    - "3101:3000"
  environment:
    GF_SECURITY_ADMIN_USER: admin
    GF_SECURITY_ADMIN_PASSWORD: admin
    GF_USERS_ALLOW_SIGN_UP: "false"
  volumes:
    - ./infra/grafana/provisioning:/etc/grafana/provisioning:ro
    - ./infra/grafana/dashboards:/var/lib/grafana/dashboards:ro
    - grafana_data:/var/lib/grafana
  depends_on:
    - prometheus
  networks:
    - vigilcare_net
```

- **`depends_on: prometheus`**: Grafana starts after Prometheus. Note that `depends_on` without a `condition` only controls **start order**, not readiness — Docker starts Grafana after starting (not after Prometheus is fully ready). Grafana won't crash if Prometheus takes a few seconds to start up, but the datasource won't return data until Prometheus is actually accepting connections.
- **Provisioning volumes**: Two read-only bind mounts auto-configure Grafana on first start:
  - `provisioning/` — datasource configuration pointing Grafana to `http://prometheus:9090`
  - `dashboards/` — pre-built dashboard JSON files loaded automatically
- **`grafana_data` volume**: Persists Grafana's internal database (saved queries, user preferences) across restarts.

---

## Profiles: Optional Service Groups

The ward gateway services use Docker Compose **profiles** to avoid starting them during normal development:

```yaml
ward-gateway-db:
  image: postgres:16
  profiles: ["ward-gateway", "full"]
  # ...

ward-gateway-redis:
  image: redis:7-alpine
  profiles: ["ward-gateway", "full"]
  # ...

ward-gateway-rabbitmq:
  image: rabbitmq:3.13-management-alpine
  profiles: ["ward-gateway", "full"]
  # ...

ward-gateway-api:
  profiles: ["ward-gateway", "full"]
  build:
    context: .
    dockerfile: VigilCare.WardGateway/Dockerfile
  # ...
```

Services without a `profiles` key start by default. Services with profiles only start when you explicitly request that profile:

```bash
# Start only the core stack (no gateway services)
docker compose up -d

# Start core + ward gateway
docker compose --profile ward-gateway up -d

# Start everything
docker compose --profile full up -d
```

This keeps the default development experience lightweight — 9 services instead of 13.

---

## The Ward Gateway Service

The gateway is the only application service that runs inside Docker (the main API runs on the host via `dotnet run`). It uses a **multi-stage Dockerfile**:

```dockerfile
FROM mcr.microsoft.com/dotnet/sdk:8.0 AS build
WORKDIR /src
COPY VigilCareClinical.sln ./
COPY VigilCare.ClinicalContracts/ VigilCare.ClinicalContracts/
COPY VigilCare.WardGateway/ VigilCare.WardGateway/
RUN dotnet restore VigilCare.WardGateway/VigilCare.WardGateway.csproj
RUN dotnet publish VigilCare.WardGateway/VigilCare.WardGateway.csproj \
    -c Release -o /app/publish --no-restore

FROM mcr.microsoft.com/dotnet/aspnet:8.0 AS runtime
WORKDIR /app
COPY --from=build /app/publish .
ENV ASPNETCORE_URLS=http://+:8080
EXPOSE 8080
ENTRYPOINT ["dotnet", "VigilCare.WardGateway.dll"]
```

**What is a multi-stage build?** A Dockerfile can have multiple `FROM` lines, each starting a new "stage." The first stage (`AS build`) uses the full .NET SDK image (~900MB) to compile the code. The second stage (`AS runtime`) starts fresh from the smaller ASP.NET runtime image (~220MB) and copies only the compiled output. The final image doesn't contain the SDK, source code, or NuGet package cache — it's lean and production-ready. This is like using a full workshop to build furniture, then shipping only the finished table to the customer.

**Build context**: Set to `.` (project root) because the gateway depends on `VigilCare.ClinicalContracts` — a shared library. The Dockerfile needs to COPY both projects into the build stage.

### Gateway Environment Variables

The gateway container receives all its configuration through environment variables, overriding `appsettings.json`:

```yaml
environment:
  ASPNETCORE_ENVIRONMENT: Development
  ConnectionStrings__GatewayDb: "Host=ward-gateway-db;Port=5432;..."
  Redis__ConnectionString: "ward-gateway-redis:6379"
  RabbitMq__Host: ward-gateway-rabbitmq
  CentralApi__BaseUrl: "http://host.docker.internal:5270"
  Gateway__GatewayId: "22222222-2222-2222-2222-222222222222"
  GATEWAY_SYNC_JWT: "${GATEWAY_SYNC_JWT:-}"
```

Key patterns:
- **Double underscore** (`__`) maps to nested JSON keys. ASP.NET Core reads environment variables and translates `__` into the `:` separator used in JSON. So `ConnectionStrings__GatewayDb` overrides `{"ConnectionStrings": {"GatewayDb": "..."}}` in `appsettings.json`.
- **`host.docker.internal`** in `CentralApi__BaseUrl`: The gateway runs inside Docker, but the main API runs on your laptop. This special hostname lets the container reach back out to your laptop.
- **`${GATEWAY_SYNC_JWT:-}`**: This is shell variable substitution. Docker Compose reads the value of `GATEWAY_SYNC_JWT` from a `.env` file in the project root or from your host environment. The `:-` means "use an empty string as the default if the variable isn't set."

### Gateway Dependency Conditions

```yaml
depends_on:
  ward-gateway-db:
    condition: service_healthy
  ward-gateway-redis:
    condition: service_started
  ward-gateway-rabbitmq:
    condition: service_healthy
```

- **`service_healthy`** waits for the health check to pass. Used for PostgreSQL and RabbitMQ which need time to initialize.
- **`service_started`** only waits for the container process to start. Used for Redis which is ready almost immediately.

---

## Health Checks

Three services have explicit health checks:

| Service | Check Command | Interval | Retries |
|---------|--------------|----------|---------|
| Elasticsearch | `curl -f http://localhost:9200/_cluster/health` | 10s | 10 |
| RabbitMQ | `rabbitmq-diagnostics ping` | 10s | 5 |
| ward-gateway-db | `pg_isready -U postgres` | 10s | 5 |

Health checks serve two purposes:
1. **Dependency ordering**: `depends_on` with `condition: service_healthy` blocks until the check passes.
2. **Visibility**: `docker compose ps` shows health status per container.

Services without health checks (Kafka, Redis, MinIO) start quickly enough that ordering isn't critical, or the application handles connection retries itself (e.g., `ThresholdCacheLoader` retries Redis 3 times with exponential backoff).

---

## The `extra_hosts` Pattern

```yaml
extra_hosts:
  - "host.docker.internal:host-gateway"
```

This appears on two services: **Prometheus** and **ward-gateway-api**. On Linux, Docker doesn't provide `host.docker.internal` by default (it does on macOS and Windows). The `host-gateway` magic value resolves to the host machine's IP on the Docker bridge network.

This is needed because the .NET API runs on the host (via `dotnet run`), not inside Docker. Prometheus needs to scrape `host.docker.internal:5270/metrics`, and the ward gateway needs to call `host.docker.internal:5270` as the central API.

---

## Port Mapping Summary

| Service | Host Port | Container Port | Purpose |
|---------|-----------|---------------|---------|
| PostgreSQL | 5436 | 5432 | Primary database |
| Redis | 6382 | 6379 | Cache/state |
| Kafka | 9092 | 9092 | Event streaming |
| Elasticsearch | 9200 | 9200 | Search |
| RabbitMQ | 5674 | 5672 | AMQP |
| RabbitMQ Management | 15674 | 15672 | Web UI |
| MinIO S3 | 9005 | 9000 | Object storage API |
| MinIO Console | 9006 | 9001 | Web UI |
| Prometheus | 9101 | 9090 | Metrics |
| Grafana | 3101 | 3000 | Dashboards |
| Seq | 5345 | 80 | Log aggregation |
| Ward Gateway DB | 5437 | 5432 | Gateway database |
| Ward Gateway Redis | 6383 | 6379 | Gateway cache |
| Ward Gateway RabbitMQ | 5675 | 5672 | Gateway queues |
| Ward Gateway API | 5081 | 8080 | Gateway HTTP |

All host ports are intentionally non-standard to avoid collisions with locally installed services.

---

## Common Commands

```bash
# Start the core stack (background)
docker compose up -d

# Start with ward gateway
docker compose --profile ward-gateway up -d

# View running containers and health status
docker compose ps

# View logs for a specific service
docker compose logs -f kafka

# Restart a single service
docker compose restart elasticsearch

# Stop everything but keep volumes (data preserved)
docker compose down

# Stop everything and delete volumes (clean slate)
docker compose down -v

# Rebuild the ward gateway image after code changes
docker compose --profile ward-gateway build ward-gateway-api
docker compose --profile ward-gateway up -d ward-gateway-api
```

---

## How the API Connects

The API runs on the host machine (`dotnet run`), not in Docker. It connects to containerized services using the **host ports** defined in the compose file. These are configured in `appsettings.json`:

```json
{
  "ConnectionStrings": {
    "DefaultConnection": "Host=localhost;Port=5436;Database=vigilcare;..."
  },
  "Redis": { "ConnectionString": "localhost:6382" },
  "Kafka": { "BootstrapServers": "localhost:9092" },
  "Elasticsearch": { "Uri": "http://localhost:9200" },
  "RabbitMq": { "Host": "localhost", "Port": 5674 },
  "Minio": { "Endpoint": "localhost:9005" },
  "Seq": { "ServerUrl": "http://localhost:5345" }
}
```

This split — infrastructure in Docker, application on the host — gives the best development experience: fast iteration on application code (no rebuild/restart cycle) with production-like infrastructure.
