61 lines
2.7 KiB
Markdown
61 lines
2.7 KiB
Markdown
# Production observability wiring (Phase 36 Step 9)
|
|
#
|
|
# Local-dev Prometheus/Grafana/Seq remain in docker-compose.yml + infra/.
|
|
# Production uses *existing external* instances — do not start those services
|
|
# from docker-compose.prod.yml.
|
|
|
|
## Prometheus
|
|
|
|
1. Keep [infra/prometheus/prometheus.yml](../infra/prometheus/prometheus.yml) as the
|
|
local-dev scrape of `host.docker.internal:5270`. Do not add production jobs there.
|
|
2. On the external Prometheus host, merge
|
|
[infra/prometheus/production/scrape-vigilcare.yml](../infra/prometheus/production/scrape-vigilcare.yml)
|
|
into `scrape_configs`.
|
|
3. Load
|
|
[infra/prometheus/production/alert-rules.yml](../infra/prometheus/production/alert-rules.yml)
|
|
via `rule_files`.
|
|
4. Optionally enable the commented blackbox probe for `/health/ready` (same scrape file).
|
|
|
|
### `/metrics` access
|
|
|
|
The Clinical API uses a global `FallbackPolicy` that requires an authenticated user.
|
|
`/metrics` is explicitly `.AllowAnonymous()` so Prometheus can scrape without a JWT.
|
|
**Restrict at the network layer** — firewall so only the Prometheus host (and
|
|
operators) can reach `API_PORT` /metrics. Metric labels include department and
|
|
gateway identifiers.
|
|
|
|
## Grafana
|
|
|
|
Prefer file provisioning over UI import (UI imports are lost on Grafana redeploy):
|
|
|
|
1. Copy datasources:
|
|
[infra/grafana/production/datasources.yml](../infra/grafana/production/datasources.yml)
|
|
— keep `uid: prometheus` so existing dashboards keep working.
|
|
2. Copy dashboards from [infra/grafana/dashboards/](../infra/grafana/dashboards/) to the
|
|
path referenced by
|
|
[infra/grafana/production/dashboards-provider.yml](../infra/grafana/production/dashboards-provider.yml).
|
|
3. Confirm panels render against the external Prometheus (job labels
|
|
`vigilcare_api_prod` / `environment=production`).
|
|
|
|
Minimum production alerts (also in Prometheus rules): API down, outbox backlog,
|
|
Kafka consumer lag, ready probe failing, gateway offline, unacked CRITICAL alerts.
|
|
|
|
## Seq
|
|
|
|
1. Point production at Seq via env (compose already sets these):
|
|
- `Seq__ServerUrl` / `Serilog__WriteTo__1__Args__serverUrl` → `SEQ_URL`
|
|
- `Serilog__WriteTo__1__Args__apiKey` → `SEQ_API_KEY` (ingest-only key)
|
|
2. `appsettings.Production.json` sets
|
|
`Microsoft.EntityFrameworkCore.Database.Command` to **Warning** so SQL with
|
|
patient identifiers is not shipped to Seq.
|
|
3. Serilog `Properties:Application` is set to `VigilCareClinicalAPI` /
|
|
`VigilCare.WardGateway` for filtering.
|
|
4. In Seq, create a signal (or shared dashboard) approximately:
|
|
|
|
```
|
|
Application = 'VigilCareClinicalAPI' and @Level in ['Error', 'Fatal']
|
|
```
|
|
|
|
Optionally a second signal for the ward gateway with
|
|
`Application = 'VigilCare.WardGateway'`.
|