feature: Observability: Prometheus Metrics and Grafana
This commit is contained in:
voltsrage
@@ -54,11 +54,11 @@ An `OutboxEvent` is written in the same transaction as any observation or alert,
|
||||
- **Kafka Pipeline** — three topics (`observation.recorded`, `alert.generated`, `encounter.status.changed`) with six partitions each; KRaft mode, no Zookeeper; `KAFKA_AUTO_CREATE_TOPICS_ENABLE=false` — topics are provisioned explicitly by `KafkaTopicProvisioner`
|
||||
- **Elasticsearch CQRS Projection** — `EsIndexerService` consumer group upserts `patient_encounters` documents, appends to the `observations` index, and updates `openAlertCount` on alert events; patient/encounter search; per-encounter observation trend (hourly avg/min/max); alert volume summary by department and severity; population query (numeric range aggregation across all patients)
|
||||
- **Sepsis Early Warning Engine** — `SepsisEngineService` Kafka consumer evaluates SIRS criteria (temperature, heart rate, respiratory rate, WBC) per encounter using Redis keys with a 30-minute TTL sliding window; on ≥2 active criteria, inserts a `SEPSIS_WARNING / CRITICAL` alert idempotently (`INSERT WHERE NOT EXISTS`)
|
||||
- **RabbitMQ Notification Workers** — `NotificationPublisherService` reads `alert.generated` from Kafka and publishes paging jobs to `alerts.paging.queue`; `PagingWorkerService` sends the page and waits for acknowledgment; NACK on timeout routes to `alerts.paging.dlq` with `x-message-ttl = 300000ms`; `EscalationWorkerService` pages the on-call backup and sets alert status to `escalated`; `DischargeSummaryWorkerService` reads `encounter.status.changed`, generates a discharge summary, and stores it in MinIO under `/discharge-summaries/{encounterId}/summary.pdf`
|
||||
- **RabbitMQ Notification Workers** — `NotificationPublisherService` reads `alert.generated` from Kafka and publishes paging jobs to `alerts.paging.queue`; `PagingWorkerService` sends the page and waits for acknowledgment; if no ack arrives before timeout it NACKs to `alerts.paging.dlq` with `x-message-ttl = 300000ms`; if the host is stopping, in-flight paging messages are NACKed with `requeue=true` so they are retried after restart and do not false-escalate; `EscalationWorkerService` pages the on-call backup and sets alert status to `escalated`; `DischargeSummaryWorkerService` reads `encounter.status.changed`, generates a discharge summary, and stores it in MinIO under `/discharge-summaries/{encounterId}/summary.pdf`
|
||||
- **Data Lake Writer** — Kafka consumer writing partitioned Parquet files to MinIO (`/observations/`, `/alerts/`, `/encounters/` by date); flush policy: 1,000 events or 5 minutes, whichever comes first; columnar format for 10-year regulatory retention
|
||||
- **Reconciliation Jobs** — three scheduled checks: (1) unacknowledged CRITICAL alerts older than 30 minutes, (2) pending orders without results after 4 hours, (3) active inpatients with no observation in 2 hours; each finding creates a `reconciliation_alerts` row and publishes to RabbitMQ
|
||||
- **Standard Envelope** — all responses use a consistent `{ success, statusCode, data, error }` wrapper; validation errors use the same shape; `ApiBehaviorOptions` overridden so model validation also produces the standard envelope
|
||||
- **Observability** — Serilog structured logging enriched with `correlationId`, `encounterId`, `patientId` on alert paths; Seq sink; per-request correlation IDs in request logs and response headers
|
||||
- **Observability** — Serilog structured logging enriched with `correlationId`, `encounterId`, `patientId` on alert paths; Seq sink (`http://localhost:5345`); Prometheus (`http://localhost:9101`) scrapes `GET /metrics`; Grafana dashboards (`http://localhost:3101`, admin/admin) for clinical metrics including `alerts_unacknowledged_gauge`; per-request correlation IDs in request logs and response headers
|
||||
- **Swagger UI** — OpenAPI spec via Swashbuckle (Development only)
|
||||
|
||||
---
|
||||
@@ -84,9 +84,10 @@ IHostedServices (background):
|
||||
EsIndexerService → Kafka → Elasticsearch (consumer group: es-indexer)
|
||||
SepsisEngineService → Kafka → Redis SIRS state → PostgreSQL alert (consumer group: sepsis-engine)
|
||||
NotificationPublisherService → Kafka → RabbitMQ paging.queue (consumer group: notification-publisher)
|
||||
PagingWorkerService → RabbitMQ paging.queue → log page → NACK on timeout
|
||||
PagingWorkerService → RabbitMQ paging.queue → log page → NACK on timeout (or requeue on shutdown)
|
||||
EscalationWorkerService → RabbitMQ escalation.queue → update alert status
|
||||
DischargeSummaryWorkerService → RabbitMQ discharge.queue → MinIO Parquet
|
||||
DischargeSummaryWorkerService → RabbitMQ discharge.queue → MinIO PDF
|
||||
ReconciliationScheduler → three scheduled safety checks → reconciliation_alerts + RabbitMQ
|
||||
```
|
||||
|
||||
**Why Kafka and RabbitMQ coexist:** Kafka is an append-only log — the same observation event reaches the Elasticsearch indexer, the sepsis engine, and the data lake independently without coordination. Each consumer holds its own offset and can replay from the beginning. RabbitMQ handles the action side: one message, one worker, one page. A duplicate page at 3am is a patient safety concern, not a minor inconvenience — RabbitMQ's acknowledgment-then-delete model is correct here. The DLQ TTL-based escalation has no equivalent in Kafka.
|
||||
@@ -105,6 +106,7 @@ IHostedServices (background):
|
||||
| Search / analytics | Elasticsearch 8.13 (CQRS read projection) |
|
||||
| Data lake | MinIO (Parquet, S3-compatible) |
|
||||
| Logging | Serilog + Seq sink |
|
||||
| Metrics / dashboards | Prometheus 2.52 + Grafana 10.4 |
|
||||
| Docs | Swagger / OpenAPI (Swashbuckle) |
|
||||
| Testing | xUnit + Testcontainers |
|
||||
|
||||
@@ -115,7 +117,7 @@ IHostedServices (background):
|
||||
```
|
||||
VigilCareClinicalAPI/
|
||||
├── Program.cs # Service registration, middleware, seed on startup
|
||||
├── appsettings.json # Connection strings, Kafka, Elasticsearch, RabbitMQ, MinIO, Serilog
|
||||
├── appsettings.json # Connection strings, Kafka, Elasticsearch, RabbitMQ, MinIO, Serilog, ReconciliationJobs
|
||||
├── Controllers/
|
||||
│ ├── PatientsController.cs # Patient CRUD, search by name/MRN
|
||||
│ ├── EncountersController.cs # Encounter open, status PATCH, timeline
|
||||
@@ -140,7 +142,7 @@ VigilCareClinicalAPI/
|
||||
│ ├── AlertStatus.cs # Open, Acknowledged, Resolved, Escalated
|
||||
│ ├── AlertType.cs # ThresholdBreach, SepsisWarning, …
|
||||
│ ├── ObservationSource.cs # Device, Manual, Lab
|
||||
│ └── OrderType.cs / ReconciliationCheckType.cs
|
||||
│ └── OrderType.cs / ReconciliationCheckType.cs / Department.cs / OrderStatus.cs
|
||||
├── Services/
|
||||
│ ├── PatientService.cs
|
||||
│ ├── EncounterService.cs # Status state machine + ConflictException on invalid transitions
|
||||
@@ -153,15 +155,22 @@ VigilCareClinicalAPI/
|
||||
├── BackgroundServices/
|
||||
│ ├── ThresholdCacheLoader.cs # Pre-loads all thresholds into Redis on startup
|
||||
│ ├── KafkaTopicProvisioner.cs # Creates topics with NumPartitions from config
|
||||
│ ├── ElasticIndexProvisioner.cs # Creates patient_encounters, observations, clinical_alerts indices
|
||||
│ ├── OutboxRelayService.cs # Polls outbox every 500ms; publishes to Kafka; marks processed
|
||||
│ ├── EsIndexerService.cs # consumer group: es-indexer; upserts Elasticsearch documents
|
||||
│ ├── ElasticsSearch/
|
||||
│ │ ├── ElasticIndexProvisioner.cs # Creates patient_encounters, observations, clinical_alerts indices
|
||||
│ │ └── EsIndexerService.cs # consumer group: es-indexer; upserts Elasticsearch documents
|
||||
│ ├── SepsisEngineService.cs # consumer group: sepsis-engine; SIRS eval via Redis TTL keys
|
||||
│ └── Notifications/
|
||||
│ ├── NotificationPublisherService.cs # consumer group: notification-publisher; alert.generated → RabbitMQ paging.queue
|
||||
│ ├── PagingWorkerService.cs # RabbitMQ consumer; logs page; NACK on ack timeout → DLQ
|
||||
│ ├── EscalationWorkerService.cs # RabbitMQ escalation.queue; logs escalation; sets alert.status = escalated
|
||||
│ └── DischargeSummaryWorkerService.cs # RabbitMQ discharge.queue; generates summary; uploads to MinIO
|
||||
│ ├── Notifications/
|
||||
│ │ ├── NotificationPublisherService.cs # consumer group: notification-publisher; alert.generated → RabbitMQ paging.queue
|
||||
│ │ ├── PagingWorkerService.cs # RabbitMQ consumer; logs page; NACK on ack timeout → DLQ, requeue on graceful shutdown
|
||||
│ │ ├── EscalationWorkerService.cs # RabbitMQ escalation.queue; logs escalation; sets alert.status = escalated
|
||||
│ │ └── DischargeSummaryWorkerService.cs # RabbitMQ discharge.queue; generates summary PDF; uploads to MinIO
|
||||
│ └── Reconciliation/
|
||||
│ ├── ReconciliationScheduler.cs # Runs three safety checks on a configurable interval
|
||||
│ ├── UnacknowledgedAlertsCheck.cs # CRITICAL alerts unacknowledged > 30 min
|
||||
│ ├── PendingOrdersCheck.cs # Pending orders without results > 4 hours
|
||||
│ ├── DisconnectedMonitorsCheck.cs # Active inpatients with no observation > 2 hours
|
||||
│ └── ReconciliationPublisher.cs # Publishes findings to notifications.reconciliation.queue
|
||||
├── Sepsis/
|
||||
│ ├── SirsDetector.cs # Redis SIRS state management (SET/DEL/MGET)
|
||||
│ └── SirsEvaluator.cs # Per-code criterion evaluation
|
||||
@@ -190,13 +199,21 @@ VigilCareClinicalAPI/
|
||||
│ └── ExceptionHandlerMiddleware.cs
|
||||
└── Migrations/
|
||||
|
||||
infra/
|
||||
├── prometheus/
|
||||
│ └── prometheus.yml # Scrape config for vigilcare_api /metrics
|
||||
└── grafana/
|
||||
├── provisioning/ # Datasource + dashboard provider config
|
||||
└── dashboards/ # vigilcare.json clinical dashboard
|
||||
|
||||
tests/
|
||||
└── VigilCareClinicalAPI.Tests/
|
||||
├── ObservationIngestTests.cs # Ingest happy path, critical alert creation, discharged encounter rejection, idempotency
|
||||
├── AlertLifecycleTests.cs # Acknowledge, resolve, escalation guard
|
||||
├── SirsDetectorTests.cs # Redis SIRS state SET/DEL/MGET logic
|
||||
├── SirsEvaluatorTests.cs # Per-code criterion evaluation
|
||||
└── NotificationPipelineTests.cs # RabbitMQ topology, DLQ routing
|
||||
├── NotificationPipelineTests.cs # RabbitMQ topology, DLQ routing
|
||||
└── ReconciliationTests.cs # Three reconciliation checks, deduplication, RabbitMQ publish
|
||||
```
|
||||
|
||||
---
|
||||
@@ -239,7 +256,7 @@ PostgreSQL is always the write side and the source of truth. Elasticsearch is a
|
||||
|
||||
### DLQ as a Clinical Escalation Protocol
|
||||
|
||||
The five-minute escalation is not a retry — it is a clinical workflow. When an alert is created, the paging worker sends a page to the attending physician. If no acknowledgment arrives within five minutes, the message NACKs to `alerts.paging.dlq` with `x-message-ttl = 300000ms`. After TTL expires, the DLQ re-routes to `alerts.escalation.queue` and the on-call backup is paged. This pattern has no equivalent in Kafka — Kafka has no concept of per-message TTL or conditional re-routing based on consumer acknowledgment.
|
||||
The five-minute escalation is not a retry — it is a clinical workflow. When an alert is created, the paging worker sends a page to the attending physician. If no acknowledgment arrives within five minutes, the message NACKs to `alerts.paging.dlq` with `x-message-ttl = 300000ms`. After TTL expires, the DLQ re-routes to `alerts.escalation.queue` and the on-call backup is paged. During graceful shutdown, cancellation of the in-flight wait loop is treated as non-failure and the message is NACKed with `requeue=true`, preventing false escalation during deploy/restart windows. This pattern has no equivalent in Kafka — Kafka has no concept of per-message TTL or conditional re-routing based on consumer acknowledgment.
|
||||
|
||||
### Why Not a Time-Series Database for Observations?
|
||||
|
||||
@@ -260,6 +277,8 @@ A medium hospital with 200 concurrent inpatients at five observations per patien
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
All services join the `vigilcare_net` bridge network so containers can reach each other by service name (e.g. Grafana → `http://prometheus:9090`). Connection strings in `appsettings.json` use **host** ports when running `dotnet run` on your machine.
|
||||
|
||||
| Service | Host Port | Notes |
|
||||
|---|---|---|
|
||||
| PostgreSQL 16 | 5436 | Database: `vigilcare`, user: `postgres`, password: `password` |
|
||||
@@ -269,9 +288,30 @@ docker compose up -d
|
||||
| Elasticsearch 8.13 | 9200 | Security disabled for development |
|
||||
| RabbitMQ 3.13 | 5674 (AMQP), 15674 (UI) | login: `guest` / `guest` |
|
||||
| MinIO | 9005 (S3 API), 9006 (console) | login: `minioadmin` / `minioadmin` |
|
||||
| Prometheus 2.52 | 9101 | UI at `http://localhost:9101` — scrapes `GET /metrics` on the API |
|
||||
| Grafana 10.4 | 3101 | UI at `http://localhost:3101` — login: `admin` / `admin` |
|
||||
|
||||
**Seq first-run:** `SEQ_FIRSTRUN_ADMINPASSWORD=admin` is set in `docker-compose.yml`. This password is only applied on the very first container start (when the `/data` volume is empty). After initialization, the password is stored in the volume and this env var is ignored.
|
||||
|
||||
### Docker notes for Linux
|
||||
|
||||
Prometheus scrapes the API using `host.docker.internal:5270`. On Linux, two things are required:
|
||||
|
||||
1. In `docker-compose.yml` under `prometheus`:
|
||||
```yaml
|
||||
extra_hosts:
|
||||
- "host.docker.internal:host-gateway"
|
||||
```
|
||||
2. Run the API bound to all interfaces (not only loopback), so containers can reach it:
|
||||
- Use `http://0.0.0.0:5270` (or `ASPNETCORE_URLS=http://0.0.0.0:5270`)
|
||||
|
||||
Without this, Prometheus may show target errors like:
|
||||
- `lookup host.docker.internal ... no such host` (DNS mapping missing), or
|
||||
- `dial tcp 172.17.0.1:5270: connect: connection refused` (API bound only to `127.0.0.1`)
|
||||
|
||||
For full Docker troubleshooting and recovery steps, see:
|
||||
- `docs/docker-compose-usage-and-troubleshooting.md`
|
||||
|
||||
### Install and Run
|
||||
|
||||
```bash
|
||||
@@ -286,6 +326,7 @@ On startup the application:
|
||||
3. Pre-loads all thresholds into Redis
|
||||
4. Provisions Kafka topics and Elasticsearch indices
|
||||
5. Declares the RabbitMQ exchange and queue topology
|
||||
6. Starts the reconciliation scheduler (three safety checks on a configurable interval)
|
||||
|
||||
Swagger UI is available at `http://localhost:<port>/swagger` in Development.
|
||||
|
||||
@@ -681,6 +722,7 @@ Exchange: `clinical.notifications.exchange` (direct)
|
||||
| `alerts.paging.dlq` | Dead-letter queue; `x-message-ttl = 300000ms` | → `alerts.escalation.queue` on TTL expiry |
|
||||
| `alerts.escalation.queue` | On-call backup paging | — |
|
||||
| `notifications.discharge.queue` | Discharge summary PDF generation + MinIO upload | — |
|
||||
| `notifications.reconciliation.queue` | Reconciliation safety findings from scheduled checks | — |
|
||||
| `notifications.appointment.queue` | Appointment reminder SMS | — |
|
||||
|
||||
---
|
||||
@@ -773,3 +815,6 @@ Observation history uses cursor pagination on `(recorded_at DESC, id DESC)`. Off
|
||||
| 4 | Elasticsearch CQRS projection (`EsIndexerService`); patient search; observation trend; alert summary; population aggregation; replay procedure | Done |
|
||||
| 5 | Sepsis detection engine (`SepsisEngineService`); Redis SIRS state with 30-min TTL; idempotent alert creation; integration tests | Done |
|
||||
| 6 | RabbitMQ exchange and queue topology; `NotificationPublisherService`; `PagingWorkerService`; DLQ escalation (`EscalationWorkerService`); discharge summary (`DischargeSummaryWorkerService` → MinIO); integration tests | Done |
|
||||
| 7 | Reconciliation scheduler — unacknowledged critical alerts, stale pending orders, disconnected monitors; `reconciliation_alerts` table; RabbitMQ publish; integration tests | Done |
|
||||
| 8 | Prometheus metrics (`GET /metrics`); Grafana dashboards; eight application metric families | In progress |
|
||||
| 9 | Data lake writer — Kafka consumer group `data-lake-writer`; Parquet flush to MinIO | Planned |
|
||||
|
||||
Reference in New Issue
Block a user