vigilcare-clinical/docs/docker-compose-usage-and-troubleshooting.md

Docker & Docker Compose Guide (VigilCare)

This guide is a practical reference for running this project with Docker, plus troubleshooting for common issues seen in this repo.

It is written for junior developers, so each section explains not just what to do, but why.

---

1) Quick start

From repo root:

docker compose up -d

Check status:

docker compose ps

Stop everything (keep data):

docker compose stop

Start again:

docker compose start

---

2) Core concepts (simple mental model)

Container

• A running process with its own filesystem and network namespace.
• Example: vigilcare_prometheus is one container.

Service (in docker-compose.yml)

• A recipe for how to run a container.
• Example: the prometheus: section defines image, ports, volumes, networks.

Image

• Template used to create a container.
• Example: prom/prometheus:v2.52.0.

Volume

• Persistent storage managed by Docker.
• Survives container restarts/recreates.
• Example: prometheus_data, grafana_data, pg_data.

Network

• Virtual network connecting containers.
• Containers can reach each other by service name (DNS).
• Example: Grafana reaches Prometheus at http://prometheus:9090 inside Docker.

---

3) Host ports vs container ports

In Compose, this format is used:

ports:
  - "HOST:CONTAINER"

Example from this project:

• Prometheus: "9101:9090"
  • Open in browser with http://localhost:9101
  • Inside Docker, service still listens on 9090
• Grafana: "3101:3000"
  • Open in browser with http://localhost:3101

If a UI is not loading, first verify host port mappings in docker-compose.yml.

---

4) Project networking (vigilcare_net)

This project uses a user-defined bridge network:

networks:
  vigilcare_net:
    driver: bridge

All services should join it:

networks:
  - vigilcare_net

Why this matters:

• Service-to-service DNS works (prometheus, grafana, postgres, etc.).
• Keeps local environment predictable.

Important Linux note: host.docker.internal

Prometheus scrapes the API via host address in this project:

• http://host.docker.internal:5270/metrics

On Linux, host.docker.internal may not resolve by default.
Fix by adding this to the prometheus service:

extra_hosts:
  - "host.docker.internal:host-gateway"

Then recreate Prometheus:

docker compose up -d --force-recreate prometheus

Symptom when missing:

• Prometheus target vigilcare_api is down
• Error: lookup host.docker.internal ... no such host

Important: API must listen on all interfaces (not only localhost)

After extra_hosts is fixed, Prometheus may still show:

dial tcp 172.17.0.1:5270: connect: connection refused

Why: dotnet run with http://localhost:5270 binds only to 127.0.0.1.
localhost inside a container means the container itself, not your host machine.
Prometheus inside Docker reaches the host via the gateway IP (172.17.0.1 via host.docker.internal), not host loopback.

Check binding:

ss -tlnp | rg ':5270'

If you see 127.0.0.1:5270, Prometheus cannot scrape from Docker.

Fix (local dev): bind on all interfaces in VigilCareClinicalAPI/Properties/launchSettings.json:

"applicationUrl": "http://0.0.0.0:5270"

Or start the API with:

ASPNETCORE_URLS=http://0.0.0.0:5270 dotnet run --project VigilCareClinicalAPI

Then restart the API and confirm:

ss -tlnp | rg ':5270'    # should show 0.0.0.0:5270
curl -sS http://localhost:5270/metrics | head

Security note: 0.0.0.0 is fine for local development. In production, bind explicitly and use proper network controls.

---

5) Volumes and persistence

This repo uses named volumes for persistent data:

• pg_data
• seq_data
• kafka_data
• es_data
• minio_data
• prometheus_data
• grafana_data

Why your data still exists after restart

• docker compose up -d --force-recreate recreates containers, but volumes remain.
• This is expected and usually desired.

Full reset (destructive)

If you need a totally clean environment:

docker compose down -v

Warning:

• -v removes named volumes (database/log/index data lost).

---

6) Common commands and when to use them

Apply config change to one service

Use when you changed only one section (e.g., Prometheus extra_hosts):

docker compose up -d --force-recreate prometheus

Restart service without recreate

Use when config did not change and you just want a restart:

docker compose restart prometheus

Rebuild image service

Use when Dockerfile/app code in image changed:

docker compose up -d --build <service>

View service logs

docker compose logs -f prometheus
docker compose logs -f grafana

---

7) Troubleshooting playbook

A) “Service is up but endpoint won’t open”

1. Check container state:

   docker compose ps
   

2. Verify port mapping in docker-compose.yml.
3. Check logs:

   docker compose logs --tail=100 <service>
   

B) “Prometheus healthy, but target is DOWN”

1. Open Prometheus targets page:
   • http://localhost:9101/targets
2. Read the exact lastError.
3. If error mentions host.docker.internal on Linux:
   • add extra_hosts fix (section 4)
   • recreate Prometheus.
4. If error is connection refused to 172.17.0.1:5270:
   • API is likely bound to 127.0.0.1 only
   • use http://0.0.0.0:5270 and restart API (section 4).

C) “Docker compose command cannot connect to daemon”

Example:

• failed to connect to the docker API at unix:///var/run/docker.sock

Fix:

• Start Docker Desktop / Docker daemon.
• Re-run docker compose ps.

D) “Permission denied writing files under bind-mounted folder”

This can happen when directories/files were created as root.

Symptoms:

• Cannot create/edit files in folders like Grafana dashboard path.

Fix options:

1. Correct ownership on host:

   sudo chown -R $USER:$USER <folder>
   

2. Recreate problematic directory as your user.

E) “Script fails preflight even though services seem running”

Check these directly:

curl -sS -o /dev/null -w "%{http_code}\n" http://localhost:5270/api/v1/alert-thresholds
curl -sS -o /dev/null -w "%{http_code}\n" http://localhost:9101/-/healthy
curl -sS -o /dev/null -w "%{http_code}\n" http://localhost:5345

Expected: 200, 200, 200.

---

8) Current project-specific paths and notes

• Compose file: docker-compose.yml
• Prometheus config: infra/prometheus/prometheus.yml
• Grafana provisioning:
  • infra/grafana/provisioning/datasources/prometheus.yml
  • infra/grafana/provisioning/datasources/dashboards/config.yml
• Grafana dashboards expected path:
  • infra/grafana/dashboards/

Note: if you accidentally create a typo folder like dashbpards, Grafana provisioning will not load dashboards from it.

---

9) Safe workflow for config changes (recommended)

1. Edit docker-compose.yml.

2. Recreate only changed services:

   docker compose up -d --force-recreate <service>
   

3. Verify logs and health endpoints.

4. Run project verification scripts (examples):

   ./scripts/run-phase8-verification.sh
   ./scripts/run-phase9-verification.sh
   

   Phase 9 also benefits from the MinIO client (mc) and DuckDB CLI for object and Parquet checks. Install without sudo — see docs/plans/phase-9-plan.md.

This avoids unnecessary full resets and speeds up local development.