vigilcare-clinical/docs/ops/production-observability.md

Production observability wiring (Phase 36 Step 9)

Local-dev Prometheus/Grafana/Seq remain in docker-compose.yml + infra/.

Production uses existing external instances — do not start those services

from docker-compose.prod.yml.

Prometheus

1. Keep infra/prometheus/prometheus.yml as the local-dev scrape of host.docker.internal:5270. Do not add production jobs there.
2. On the external Prometheus host, merge infra/prometheus/production/scrape-vigilcare.yml into scrape_configs.
3. Load infra/prometheus/production/alert-rules.yml via rule_files.
4. Optionally enable the commented blackbox probe for /health/ready (same scrape file).

/metrics access

The Clinical API uses a global FallbackPolicy that requires an authenticated user. /metrics is explicitly .AllowAnonymous() so Prometheus can scrape without a JWT. Restrict at the network layer — firewall so only the Prometheus host (and operators) can reach API_PORT /metrics. Metric labels include department and gateway identifiers.

Grafana

Prefer file provisioning over UI import (UI imports are lost on Grafana redeploy):

1. Copy datasources: infra/grafana/production/datasources.yml — keep uid: prometheus so existing dashboards keep working.
2. Copy dashboards from infra/grafana/dashboards/ to the path referenced by infra/grafana/production/dashboards-provider.yml.
3. Confirm panels render against the external Prometheus (job labels vigilcare_api_prod / environment=production).

Minimum production alerts (also in Prometheus rules): API down, outbox backlog, Kafka consumer lag, ready probe failing, gateway offline, unacked CRITICAL alerts.

Seq

1. Point production at Seq via env (compose already sets these):

   • Seq__ServerUrl / Serilog__WriteTo__1__Args__serverUrl → SEQ_URL
   • Serilog__WriteTo__1__Args__apiKey → SEQ_API_KEY (ingest-only key)

2. appsettings.Production.json sets Microsoft.EntityFrameworkCore.Database.Command to Warning so SQL with patient identifiers is not shipped to Seq.

3. Serilog Properties:Application is set to VigilCareClinicalAPI / VigilCare.WardGateway for filtering.

4. In Seq, create a signal (or shared dashboard) approximately:

   Application = 'VigilCareClinicalAPI' and @Level in ['Error', 'Fatal']
   

   Optionally a second signal for the ward gateway with Application = 'VigilCare.WardGateway'.