# Production observability wiring (Phase 36 Step 9)
#
# Local-dev Prometheus/Grafana/Seq remain in docker-compose.yml + infra/.
# Production uses *existing external* instances — do not start those services
# from docker-compose.prod.yml.

## Prometheus

1. Keep [infra/prometheus/prometheus.yml](../infra/prometheus/prometheus.yml) as the
   local-dev scrape of `host.docker.internal:5270`. Do not add production jobs there.
2. On the external Prometheus host, merge
   [infra/prometheus/production/scrape-vigilcare.yml](../infra/prometheus/production/scrape-vigilcare.yml)
   into `scrape_configs`.
3. Load
   [infra/prometheus/production/alert-rules.yml](../infra/prometheus/production/alert-rules.yml)
   via `rule_files`.
4. Optionally enable the commented blackbox probe for `/health/ready` (same scrape file).

### `/metrics` access

The Clinical API uses a global `FallbackPolicy` that requires an authenticated user.
`/metrics` is explicitly `.AllowAnonymous()` so Prometheus can scrape without a JWT.
**Restrict at the network layer** — firewall so only the Prometheus host (and
operators) can reach `API_PORT` /metrics. Metric labels include department and
gateway identifiers.

## Grafana

Prefer file provisioning over UI import (UI imports are lost on Grafana redeploy):

1. Copy datasources:
   [infra/grafana/production/datasources.yml](../infra/grafana/production/datasources.yml)
   — keep `uid: prometheus` so existing dashboards keep working.
2. Copy dashboards from [infra/grafana/dashboards/](../infra/grafana/dashboards/) to the
   path referenced by
   [infra/grafana/production/dashboards-provider.yml](../infra/grafana/production/dashboards-provider.yml).
3. Confirm panels render against the external Prometheus (job labels
   `vigilcare_api_prod` / `environment=production`).

Minimum production alerts (also in Prometheus rules): API down, outbox backlog,
Kafka consumer lag, ready probe failing, gateway offline, unacked CRITICAL alerts.

## Seq

1. Point production at Seq via env (compose already sets these):
   - `Seq__ServerUrl` / `Serilog__WriteTo__1__Args__serverUrl` → `SEQ_URL`
   - `Serilog__WriteTo__1__Args__apiKey` → `SEQ_API_KEY` (ingest-only key)
2. `appsettings.Production.json` sets
   `Microsoft.EntityFrameworkCore.Database.Command` to **Warning** so SQL with
   patient identifiers is not shipped to Seq.
3. Serilog `Properties:Application` is set to `VigilCareClinicalAPI` /
   `VigilCare.WardGateway` for filtering.
4. In Seq, create a signal (or shared dashboard) approximately:

   ```
   Application = 'VigilCareClinicalAPI' and @Level in ['Error', 'Fatal']
   ```

   Optionally a second signal for the ward gateway with
   `Application = 'VigilCare.WardGateway'`.