Trent/vigilcare-clinical
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feature: NEWS2 Composite Scoring Engine

Browse Source
This commit is contained in:
voltsrage
2026-06-18 17:39:31 +08:00
parent c3fbc20ddc
commit e6f7989298
31 changed files with 3118 additions and 45 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+127 -31
View File
@@ -1,19 +1,19 @@
# VigilCare Clinical API
A production-quality clinical backend built with ASP.NET Core 8, PostgreSQL, Apache Kafka, RabbitMQ, Elasticsearch, Redis, and MinIO. The domain models the observe-alert-acknowledge lifecycle at the center of any clinical monitoring system: patient encounters, continuous vital sign and lab result ingest, real-time sepsis detection, and clinician notification with automatic escalation.
A production-quality clinical backend built with ASP.NET Core 8, PostgreSQL, Apache Kafka, RabbitMQ, Elasticsearch, Redis, and MinIO. The domain models the observe-alert-acknowledge lifecycle at the center of any clinical monitoring system: patient encounters, continuous vital sign and lab result ingest, real-time sepsis and NEWS2 scoring, and clinician notification with automatic escalation.
**Implementation status:** All ten planned phases are complete — from schema and CRUD through Kafka, Elasticsearch CQRS, sepsis detection, RabbitMQ paging with DLQ escalation, reconciliation jobs, Prometheus/Grafana observability, the MinIO Parquet data lake, and clinical data model expansion (patient demographics, encounter enrichment, 12 observation codes). See [Implemented Phases](#implemented-phases) for the full breakdown.
**Implementation status:** All twelve planned phases are complete — from schema and CRUD through Kafka, Elasticsearch CQRS, sepsis detection, RabbitMQ paging with DLQ escalation, reconciliation jobs, Prometheus/Grafana observability, the MinIO Parquet data lake, clinical data model expansion (patient demographics, encounter enrichment, 12 observation codes), warning alerts and orders, and the NEWS2 composite scoring engine. See [Implemented Phases](#implemented-phases) for the full breakdown.
## Domain Model — How It Maps to a Real Clinical System
In a hospital, a patient presents for care and an encounter is opened. Bedside monitors and lab systems post observations continuously against that encounter. A rules engine evaluates each observation against configured thresholds and flags abnormal values as clinical alerts. Clinicians acknowledge and resolve alerts. If a critical alert goes unacknowledged for five minutes, the system escalates to the on-call backup. All events flow through Kafka so the Elasticsearch dashboard, sepsis engine, and data lake writer consume the same stream independently.
In a hospital, a patient presents for care and an encounter is opened. Bedside monitors and lab systems post observations continuously against that encounter. A rules engine evaluates each observation against configured thresholds and flags abnormal values as clinical alerts. Composite scoring engines (NEWS2, SIRS/sepsis) aggregate multiple vitals into acuity scores. Clinicians acknowledge and resolve alerts. If a critical alert goes unacknowledged for five minutes, the system escalates to the on-call backup. All events flow through Kafka so the Elasticsearch dashboard, scoring engines, and data lake writer consume the same stream independently.
```
Patient ─────────────────────────── one patient = one MRN, many lifetime encounters
└── Encounter one clinical episode (inpatient, outpatient, ED)
├── Observation one measurement: vital sign, lab value, SpO₂
│ └── OutboxEvent written in the same transaction → relayed to Kafka
└── ClinicalAlert generated on threshold breach or SIRS detection
└── ClinicalAlert generated on threshold breach, SIRS detection, or NEWS2 composite score
└── OutboxEvent → Kafka → RabbitMQ → clinician page → escalation
```
@@ -37,7 +37,7 @@ An `idempotencyKey` (partial unique index) prevents duplicate observations when
### ClinicalAlert
A `ClinicalAlert` is generated when an observation breaches a threshold or when the sepsis engine detects two or more concurrent SIRS criteria. Lifecycle: `open → acknowledged → resolved` (or `escalated` after a five-minute NACK cycle through the RabbitMQ dead-letter queue). Alerts carry an audit trail: who acknowledged, when, and with what note.
A `ClinicalAlert` is generated when an observation breaches a threshold, when the sepsis engine detects two or more concurrent SIRS criteria, or when the NEWS2 engine computes a medium/high-risk composite score (or a single-parameter score of 3). Lifecycle: `open → acknowledged → resolved` (or `escalated` after a five-minute NACK cycle through the RabbitMQ dead-letter queue). Alerts carry an audit trail: who acknowledged, when, and with what note.
### OutboxEvent
@@ -56,14 +56,15 @@ An `OutboxEvent` is written in the same transaction as any observation or alert,
- **Clinical Alert Lifecycle** — paginated alert list per encounter and globally; acknowledge with clinician ID and optional note; resolve (must be acknowledged first); global list filterable by status, severity, and department
- **Outbox Relay** — `IHostedService` polling every 500ms; reads unprocessed outbox rows, publishes to Kafka, marks processed; partitioned by `encounterId` for per-encounter ordering
- **Kafka Pipeline** — three topics (`observation.recorded`, `alert.generated`, `encounter.status.changed`) with six partitions each; KRaft mode, no Zookeeper; `KAFKA_AUTO_CREATE_TOPICS_ENABLE=false` — topics are provisioned explicitly by `KafkaTopicProvisioner`
- **Elasticsearch CQRS Projection** — `EsIndexerService` consumer group upserts `patient_encounters` documents, appends to the `observations` index, and updates `openAlertCount` on alert events; patient/encounter search; per-encounter observation trend (hourly avg/min/max); alert volume summary by department and severity; population query (numeric range aggregation across all patients)
- **Elasticsearch CQRS Projection** — `EsIndexerService` consumer group upserts `patient_encounters` documents, appends to the `observations` index, increments `openAlertCount` on alert events, and stamps `news2Score` / `news2RiskLevel` when a NEWS2 alert is generated; patient/encounter search; per-encounter observation trend (hourly avg/min/max); alert volume summary by department and severity; population query (numeric range aggregation across all patients)
- **Sepsis Early Warning Engine** — `SepsisEngineService` Kafka consumer evaluates SIRS criteria (temperature, heart rate, respiratory rate, WBC) per encounter using Redis keys with a 30-minute TTL sliding window; on ≥2 active criteria, inserts a `SEPSIS_WARNING / CRITICAL` alert idempotently (`INSERT WHERE NOT EXISTS`)
- **NEWS2 Composite Scoring Engine** — `News2ScoringService` Kafka consumer (`news2-scoring`) evaluates seven vital parameters per encounter (`RESP_RATE`, `SPO2`, `SYSTOLIC_BP`, `HEART_RATE`, `AVPU`, `TEMP_C`, `SUPPLEMENTAL_O2`) using Redis keys with a 4-hour TTL; when all seven are present, computes the official NEWS2 aggregate score, persists to `news2_scores`, and creates `NEWS2_WARNING` (score 5–6 or single param = 3) or `NEWS2_EMERGENCY` (score ≥ 7) alerts idempotently; `GET /encounters/:id/news2/current` and `/history` expose score history; Prometheus `news2_scores_total` and `news2_scoring_duration_seconds`
- **RabbitMQ Notification Workers** — `NotificationPublisherService` reads `alert.generated` from Kafka and publishes paging jobs to `alerts.paging.queue`; `PagingWorkerService` sends the page and waits for acknowledgment; if no ack arrives before timeout it NACKs to `alerts.paging.dlq` with `x-message-ttl = 300000ms`; if the host is stopping, in-flight paging messages are NACKed with `requeue=true` so they are retried after restart and do not false-escalate; `EscalationWorkerService` pages the on-call backup and sets alert status to `escalated`; `DischargeSummaryWorkerService` reads `encounter.status.changed`, generates a discharge summary, and stores it in MinIO under `/discharge-summaries/{encounterId}/summary.pdf`
- **Data Lake Writer** — `DataLakeWriterService` (consumer group `data-lake-writer`) buffers `observation.recorded`, `alert.generated`, and `encounter.status.changed` events, flushes date-partitioned Parquet files to MinIO (`/observations/`, `/alerts/`, `/encounters/`), and commits Kafka offsets only after successful uploads; `kafka_partition` and `kafka_offset` columns provide audit lineage
- **Data Lake Writer** — `DataLakeWriterService` (consumer group `data-lake-writer`) buffers `observation.recorded`, `alert.generated`, and `encounter.status.changed` events, flushes date-partitioned Parquet files to MinIO (`/observations/`, `/alerts/`, `/encounters/`), and commits Kafka offsets only after at least one successful upload; shutdown flush uses an uncanceled token so MinIO writes complete on Ctrl+C; `kafka_partition` and `kafka_offset` columns provide audit lineage
- **Reconciliation Jobs** — three scheduled checks: (1) unacknowledged CRITICAL alerts older than 30 minutes, (2) pending orders without results after 4 hours, (3) active inpatients with no observation in 2 hours; each finding creates a `reconciliation_alerts` row and publishes to RabbitMQ
- **Standard Envelope** — all responses use a consistent `{ success, statusCode, data, error }` wrapper; validation errors use the same shape; `ApiBehaviorOptions` overridden so model validation also produces the standard envelope with field-level `details`
- **Input Validation** — FluentValidation validators on all request DTOs (patient registration, encounter open, observation ingest, alert acknowledge, alert thresholds, orders); invalid requests return 400 before reaching the service layer
- **Observability** — Serilog structured logging enriched with `correlationId`, `encounterId`, `patientId` on alert paths; Seq sink (`http://localhost:5345`); Prometheus (`http://localhost:9101`) scrapes `GET /metrics`; eight application metric families via `ClinicalMetrics` and three background collectors (`AlertsUnacknowledgedCollector`, `OutboxPendingCollector`, `KafkaConsumerLagCollector`); Grafana clinical dashboard (`http://localhost:3101`, admin/admin) with `alerts_unacknowledged_gauge` as the primary safety panel; per-request correlation IDs in request logs and `X-Correlation-Id` response headers
- **Observability** — Serilog structured logging enriched with `correlationId`, `encounterId`, `patientId` on alert paths; Seq sink (`http://localhost:5345`); Prometheus (`http://localhost:9101`) scrapes `GET /metrics`; ten application metric families via `ClinicalMetrics` and three background collectors (`AlertsUnacknowledgedCollector`, `OutboxPendingCollector`, `KafkaConsumerLagCollector`); Grafana clinical dashboard (`http://localhost:3101`, admin/admin) with `alerts_unacknowledged_gauge` as the primary safety panel; per-request correlation IDs in request logs and `X-Correlation-Id` response headers
- **Swagger UI** — OpenAPI spec via Swashbuckle (Development only)
---
@@ -77,7 +78,7 @@ HTTP request
→ Controllers
→ Services
├── PostgreSQL (EF Core — writes, keyed reads)
├── Redis (threshold cache, SIRS state)
├── Redis (threshold cache, SIRS state, NEWS2 parameter state)
└── OutboxEvent (same transaction as domain write)
IHostedServices (background):
@@ -89,6 +90,7 @@ IHostedServices (background):
EsIndexerService → Kafka → Elasticsearch (consumer group: es-indexer)
SepsisEngineService → Kafka → Redis SIRS state → PostgreSQL alert (consumer group: sepsis-engine)
WarningAlertService → Kafka → WarningEvaluator → PostgreSQL WARNING alert (consumer group: warning-evaluator)
News2ScoringService → Kafka → News2Detector → Redis NEWS2 state → PostgreSQL score + alert (consumer group: news2-scoring)
NotificationPublisherService → Kafka → RabbitMQ paging.queue (consumer group: notification-publisher)
PagingWorkerService → RabbitMQ paging.queue → log page → NACK on timeout (or requeue on shutdown)
EscalationWorkerService → RabbitMQ escalation.queue → update alert status
@@ -98,7 +100,7 @@ IHostedServices (background):
AlertsUnacknowledgedCollector → polls PostgreSQL every 30s → alerts_unacknowledged_gauge
OutboxPendingCollector → polls outbox every 30s → outbox_pending_events
KafkaConsumerLagCollector → polls four consumer groups every 30s → kafka_consumer_lag
ClinicalMetrics (singleton) → inline counters/histogram from ingest, SIRS, escalation paths
ClinicalMetrics (singleton) → inline counters/histogram from ingest, SIRS, NEWS2, escalation paths
```
**Why Kafka and RabbitMQ coexist:** Kafka is an append-only log — the same observation event reaches the Elasticsearch indexer, the sepsis engine, and the data lake independently without coordination. Each consumer holds its own offset and can replay from the beginning. RabbitMQ handles the action side: one message, one worker, one page. A duplicate page at 3am is a patient safety concern, not a minor inconvenience — RabbitMQ's acknowledgment-then-delete model is correct here. The DLQ TTL-based escalation has no equivalent in Kafka.
@@ -111,7 +113,7 @@ IHostedServices (background):
|---|---|
| Server | ASP.NET Core 8 (.NET 8.0) |
| Database | PostgreSQL 16 with EF Core 8 (code-first migrations) |
| Cache / SIRS state | Redis 7 |
| Cache / SIRS & NEWS2 state | Redis 7 |
| Message log | Apache Kafka 3.7 (KRaft, 6 partitions per topic) |
| Task queue | RabbitMQ 3.13 (direct exchange, DLQ escalation) |
| Search / analytics | Elasticsearch 8.13 (CQRS read projection) |
@@ -139,6 +141,7 @@ VigilCareClinicalAPI/
│ ├── AlertThresholdsController.cs # Threshold CRUD + cache invalidation
│ ├── AlertsController.cs # Alert list (global + per-encounter), acknowledge, resolve
│ ├── OrdersController.cs # Order create, list, get, status transition, record result
│ ├── News2Controller.cs # Current NEWS2 score and cursor-paginated history
│ └── AnalyticsController.cs # Elasticsearch-backed patient search, trend, alert summary, population
├── Domains/
│ ├── Entities/
@@ -148,6 +151,7 @@ VigilCareClinicalAPI/
│ │ ├── Observation.cs # Append-only; IdempotencyKey; partial unique index
│ │ ├── ClinicalAlert.cs # open → acknowledged → resolved / escalated
│ │ ├── Order.cs
│ │ ├── News2Score.cs # Composite score with seven component scores + risk level
│ │ ├── OutboxEvent.cs # topic + payload JSONB + processed_at
│ │ └── ReconciliationAlert.cs
│ └── Enums/
@@ -155,7 +159,7 @@ VigilCareClinicalAPI/
│ ├── EncounterType.cs # Inpatient, Outpatient, Emergency
│ ├── AlertSeverity.cs # Warning, Critical
│ ├── AlertStatus.cs # Open, Acknowledged, Resolved, Escalated
│ ├── AlertType.cs # Threshold breach, sepsis, warning*, systolic BP, AVPU, glucose, …
│ ├── AlertType.cs # Threshold breach, sepsis, warning*, NEWS2_*, systolic BP, AVPU, glucose, …
│ ├── BloodType.cs # A+, O-, AB-, … with ToDbString/FromDbString
│ ├── ObservationSource.cs # Device, Manual, Lab
│ └── OrderType.cs / ReconciliationCheckType.cs / Department.cs / OrderStatus.cs
@@ -172,13 +176,14 @@ VigilCareClinicalAPI/
│ ├── ObservationQueryService.cs # Cursor-paginated history
│ ├── AlertService.cs # Acknowledge, resolve, list
│ ├── OrderService.cs # Order lifecycle; status machine; ConflictException on illegal transitions
│ ├── News2Service.cs # Current score + cursor-paginated history from PostgreSQL
│ ├── WarningEvaluator.cs # Warning-range threshold evaluation; idempotent alert INSERT
│ ├── AnalyticsService.cs # Elasticsearch query wrappers
│ └── PlausibilityValidator.cs # Per-code numeric range guard
├── Validators/ # FluentValidation — RegisterPatient, OpenEncounter, IngestObservation, …
├── Observability/
│ └── Metrics/
│ └── ClinicalMetrics.cs # Eight Prometheus metric families (counters, histogram, gauges)
│ └── ClinicalMetrics.cs # Ten Prometheus metric families (counters, histograms, gauges)
├── BackgroundServices/
│ ├── ThresholdCacheLoader.cs # Pre-loads all thresholds into Redis on startup
│ ├── KafkaTopicProvisioner.cs # Creates topics with NumPartitions from config
@@ -192,6 +197,7 @@ VigilCareClinicalAPI/
│ │ └── EsIndexerService.cs # consumer group: es-indexer; upserts Elasticsearch documents
│ ├── SepsisEngineService.cs # consumer group: sepsis-engine; SIRS eval via Redis TTL keys
│ ├── WarningAlertService.cs # consumer group: warning-evaluator; observation.recorded → WARNING alerts
│ ├── News2ScoringService.cs # consumer group: news2-scoring; observation.recorded → NEWS2 score + alert
│ ├── Notifications/
│ │ ├── NotificationPublisherService.cs # consumer group: notification-publisher; alert.generated → RabbitMQ paging.queue
│ │ ├── PagingWorkerService.cs # RabbitMQ consumer; logs page; NACK on ack timeout → DLQ, requeue on graceful shutdown
@@ -210,6 +216,9 @@ VigilCareClinicalAPI/
├── Sepsis/
│ ├── SirsDetector.cs # Redis SIRS state management (SET/DEL/MGET)
│ └── SirsEvaluator.cs # Per-code criterion evaluation
├── News2/
│ ├── News2Calculator.cs # Pure static NEWS2 scoring tables (no I/O)
│ └── News2Detector.cs # Redis parameter state, score persistence, alert creation
├── Elasticsearch/Documents/
│ ├── PatientEncounterDocument.cs
│ ├── ObservationDocument.cs
@@ -220,7 +229,7 @@ VigilCareClinicalAPI/
│ └── MinioClientFactory.cs
├── DataLake/
│ ├── DataLakeOptions.cs # Flush thresholds and bucket settings
│ ├── DataLakeWriterService.cs # consumer group: data-lake-writer; Kafka → Parquet → MinIO
│ ├── DataLakeWriterService.cs # consumer group: data-lake-writer; Kafka → Parquet → MinIO; graceful shutdown flush
│ └── ParquetFileBuilder.cs # Topic row models → Parquet byte arrays
├── Models/Records/
│ ├── Observation/ObservationRow.cs # Parquet row contract for observation events
@@ -263,7 +272,9 @@ tests/
├── ClinicalDemographicsAndObservationTests.cs # Patient/encounter enrichment, expanded observation alerts
├── WarningAlertTests.cs # WarningEvaluator — warning created, normal/critical skipped, idempotent
├── OrderLifecycleTests.cs # Orders API — create, list, record result, illegal transition 409
└── ValidationTests.cs # FluentValidation — empty fields, threshold ordering, order description
├── ValidationTests.cs # FluentValidation — empty fields, threshold ordering, order description
├── News2CalculatorTests.cs # Boundary tests for all seven NEWS2 scoring tables
└── News2DetectorTests.cs # NEWS2 detector — score tiers, alerts, idempotency, incomplete set
scripts/
├── run-api-redis-tests.sh # Phase 1 — patient/encounter/threshold + Redis cache
@@ -275,10 +286,11 @@ scripts/
├── run-phase8-verification.sh # Phase 8 — Prometheus metrics, alerts_unacknowledged_gauge, correlation headers
├── run-phase9-verification.sh # Phase 9 — data lake tests, Kafka offsets, MinIO Parquet, DuckDB schema
├── run-phase10-verification.sh # Phase 10 — 12 Redis thresholds, clinical enrichment, ES pipeline, integration tests
└── run-phase11-verification.sh # Phase 11 — warning alerts, orders API, validation, integration tests
├── run-phase11-verification.sh # Phase 11 — warning alerts, orders API, validation, integration tests
└── run-phase12-verification.sh # Phase 12 — NEWS2 end-to-end pipeline, API, ES, Prometheus, integration tests
docs/
├── plans/ # Phase 1–11 implementation and verification guides
├── plans/ # Phase 1–12 implementation and verification guides
├── decisions/
│ ├── data-lake-design.md # Parquet vs JSON, partitioning, replay rationale
│ └── sepsis-engine-design.md # SIRS sliding window and idempotent alert design
@@ -302,14 +314,16 @@ This is the architectural decision that separates thinking about healthcare syst
Observations, alerts, and orders belong to an encounter, not directly to a patient. A patient's blood pressure taken during a 2022 admission belongs to that admission. This bounds queries naturally: "show me all observations for this encounter" is a bounded query. "Show me all observations ever recorded for this patient" is a cross-encounter aggregation that belongs in the data lake.
### Redis for Two Distinct Purposes
### Redis for Three Distinct Purposes
Redis serves two independent roles with different semantics:
Redis serves three independent roles with different semantics:
1. **Threshold cache:** write-through invalidation on every threshold update. Staleness here has clinical consequences — a stale threshold could suppress a critical alert. TTL expiry is not sufficient; invalidation must be immediate on write.
2. **SIRS sliding window:** `SET sirs:{encounterId}:{code} EX 1800`. The TTL does real work — a heart rate that was abnormal 31 minutes ago stops contributing to the SIRS count without any cleanup job. The 30-minute TTL is a clinical parameter, not an arbitrary cache timeout.
3. **NEWS2 parameter state:** `SET news2:{encounterId}:{code}` with a 4-hour TTL. Each of the seven NEWS2 parameters is scored individually and stored in Redis; `MGET` across all seven keys determines completeness. An incomplete set (fewer than seven present keys) does not produce a score — expired parameters must be re-recorded before the aggregate is computed.
### Outbox Pattern
Observation and alert writes use the transactional outbox: the `outbox_events` row is inserted in the same transaction as the domain record. The relay publishes to Kafka asynchronously. This prevents message loss when Kafka is temporarily unavailable and prevents phantom messages when the transaction rolls back. The relay is idempotent — re-publishing an already-processed event is safe because all downstream consumers check for duplicates.
@@ -396,7 +410,7 @@ On startup the application:
3. Pre-loads all thresholds into Redis
4. Provisions Kafka topics and Elasticsearch indices
5. Declares the RabbitMQ exchange and queue topology
6. Starts all background consumers (outbox relay, ES indexer, sepsis engine, notification workers, data lake writer, reconciliation scheduler)
6. Starts all background consumers (outbox relay, ES indexer, sepsis engine, warning evaluator, NEWS2 scoring, notification workers, data lake writer, reconciliation scheduler)
7. Starts Prometheus metric collectors (unacknowledged alerts, outbox pending, Kafka consumer lag)
Swagger UI is available at `http://localhost:5270/swagger` in Development (API binds to `0.0.0.0:5270` per `launchSettings.json`).
@@ -416,22 +430,25 @@ Integration tests use `WebApplicationFactory` with a `Testing` environment and T
| `SirsDetectorTests` / `SirsEvaluatorTests` | 5 | Redis SIRS state and per-code criterion evaluation |
| `NotificationPipelineTests` | 6 | RabbitMQ topology, DLQ routing, paging |
| `ReconciliationTests` | 7 | Three reconciliation checks, deduplication, RabbitMQ publish |
| `ObservabilityPhase8Tests` | 8 | All eight `/metrics` families, correlation headers, ingest counter increment |
| `ObservabilityPhase8Tests` | 8 | All ten `/metrics` families, correlation headers, ingest counter increment |
| `DataLakePhase9Tests` | 9 | Kafka → MinIO Parquet flow and schema checks |
| `ClinicalDemographicsAndObservationTests` | 10 | Patient clinical fields, encounter enrichment, expanded observation codes, critical glucose alert |
| `WarningAlertTests` | 11 | WarningEvaluator — warning HR alert, normal/critical skipped, duplicate idempotent |
| `OrderLifecycleTests` | 11 | Orders API create, list, record result, cancel-resulted 409 |
| `ValidationTests` | 11 | FluentValidation 400 on empty first name, invalid threshold order, empty order description |
| `News2CalculatorTests` | 12 | Boundary tests for all seven NEWS2 scoring tables and risk-level determination |
| `News2DetectorTests` | 12 | NEWS2 detector — score tiers, alert creation, incomplete parameters, idempotency |
### Verification Scripts
With the API running (`dotnet run`) and Docker Compose up:
```bash
./scripts/run-phase8-verification.sh # Prometheus target UP, eight metrics, alerts_unacknowledged_gauge live update
./scripts/run-phase8-verification.sh # Prometheus target UP, ten metrics, alerts_unacknowledged_gauge live update
./scripts/run-phase9-verification.sh # DataLakePhase9Tests, Kafka consumer group, MinIO Parquet, DuckDB schema
./scripts/run-phase10-verification.sh # 12 Redis thresholds, clinical enrichment, ES pipeline, Phase 10 integration tests
./scripts/run-phase11-verification.sh # Warning alert pipeline, orders API, FluentValidation, Phase 11 integration tests
./scripts/run-phase12-verification.sh # NEWS2 end-to-end pipeline, API, Elasticsearch, Prometheus, Phase 12 integration tests
```
Per-phase test runners (subset of `dotnet test`):
@@ -458,20 +475,22 @@ curl https://install.duckdb.org | sh
export PATH="$HOME/.duckdb/cli/latest:$HOME/.local/bin:$PATH"
```
See `docs/plans/phase-8-plan.md`, `docs/plans/phase-9-plan.md`, and `docs/plans/phase-10-plan.md` for manual Grafana, Seq, Kafka replay, and DuckDB query examples.
See `docs/plans/phase-8-plan.md` through `docs/plans/phase-12-plan.md` for manual Grafana, Seq, Kafka replay, and DuckDB query examples.
---
## Prometheus Metrics
`GET /metrics` exposes eight application metric families registered in `ClinicalMetrics`. Three background collectors poll PostgreSQL and Kafka every 30 seconds; counters and the ingest histogram are updated inline during request handling.
`GET /metrics` exposes ten application metric families registered in `ClinicalMetrics`. Three background collectors poll PostgreSQL and Kafka every 30 seconds; counters and histograms are updated inline during request handling and background processing.
| Metric | Type | Labels | Source |
|---|---|---|---|
| `observations_ingested_total` | Counter | `observation_code`, `source` | `ObservationService` on each committed observation |
| `observation_ingest_duration_seconds` | Histogram | — | `ObservationService` — full ingest transaction to COMMIT |
| `clinical_alerts_total` | Counter | `alert_type`, `severity` | `ObservationService` (threshold breach), `SirsDetector` (sepsis) |
| `clinical_alerts_total` | Counter | `alert_type`, `severity` | `ObservationService` (threshold breach), `SirsDetector`, `News2Detector` |
| `sirs_detections_total` | Counter | — | `SirsDetector` — only on successful idempotent insert |
| `news2_scores_total` | Counter | `risk_level` | `News2Detector` — on each persisted score (`LOW`, `MEDIUM`, `HIGH`, …) |
| `news2_scoring_duration_seconds` | Histogram | — | `News2Detector` — Redis update through score persistence |
| `escalations_total` | Counter | — | `EscalationWorkerService` on DLQ escalation |
| `alerts_unacknowledged_gauge` | Gauge | — | `AlertsUnacknowledgedCollector` — open CRITICAL alerts older than 5 minutes |
| `outbox_pending_events` | Gauge | — | `OutboxPendingCollector` — unprocessed outbox rows |
@@ -726,6 +745,19 @@ pending → in_progress → resulted
The `population` query uses Elasticsearch's numeric range aggregation engine — no full-text search. Running this against PostgreSQL on the operational database would compete with ingest writes under load.
### NEWS2 (National Early Warning Score 2)
| Method | Path | Description |
|---|---|---|
| GET | `/encounters/{id}/news2/current` | Latest NEWS2 score for an encounter (404 if none computed) |
| GET | `/encounters/{id}/news2/history` | Cursor-paginated score history |
**`GET /news2/current` response** includes `totalScore`, `riskLevel` (`LOW`, `LOW_MEDIUM`, `MEDIUM`, `HIGH`), all seven component scores (`respRateScore`, `spo2Score`, …), `hasSingleParamThree`, and `calculatedAt`.
**`GET /news2/history` query params:** `limit` (default 20), `cursor` (opaque token from previous response).
Scores are computed asynchronously by `News2ScoringService` after observations are ingested — allow a few seconds for the Kafka consumer to process all seven parameters before querying.
---
## Data Models
@@ -802,8 +834,8 @@ Indexes: partial unique `(idempotency_key) WHERE idempotency_key IS NOT NULL`, `
id Guid PK
encounterId Guid FK → Encounter
patientId Guid FK → Patient
observationId Guid? FK → Observation (null for SIRS alerts)
alertType string e.g. THRESHOLD_BREACH, SEPSIS_WARNING
observationId Guid? FK → Observation (null for SIRS and NEWS2 composite alerts)
alertType string e.g. THRESHOLD_BREACH, SEPSIS_WARNING, NEWS2_WARNING, NEWS2_EMERGENCY
severity string WARNING | CRITICAL
details text required
status string open | acknowledged | resolved | escalated (default: open)
@@ -815,6 +847,27 @@ triggeredAt DateTimeOffset
Indexes: `(encounter_id, triggered_at DESC)`, `(patient_id, triggered_at DESC)`, partial `(severity, triggered_at DESC) WHERE status = 'open'`
### News2Score
```
id Guid PK
encounterId Guid FK → Encounter
patientId Guid FK → Patient
totalScore int aggregate 0–20+
riskLevel string LOW | LOW_MEDIUM | MEDIUM | HIGH
respRateScore int component 0–3
spo2Score int
systolicBpScore int
heartRateScore int
consciousnessScore int
temperatureScore int
supplementalO2Score int
hasSingleParamThree bool true when any single parameter scored 3
calculatedAt DateTimeOffset
```
Indexes: `(encounter_id, calculated_at DESC)`, `(patient_id, calculated_at DESC)`
### Order
```
@@ -875,10 +928,14 @@ createdAt DateTimeOffset
"admissionReason": "Chest pain, rule out MI",
"admittedAt": "2025-01-01T08:00:00Z",
"openAlertCount": 2,
"news2Score": 6,
"news2RiskLevel": "MEDIUM",
"lastObservationAt": "2025-01-01T09:45:00Z"
}
```
`news2Score` and `news2RiskLevel` are optional — populated by `EsIndexerService` when a `NEWS2_WARNING` or `NEWS2_EMERGENCY` alert is indexed (payload fields `news2Score` / `news2RiskLevel` from `News2Detector`). LOW-risk scores with no alert are not projected to Elasticsearch.
### `observations`
```json
@@ -931,12 +988,23 @@ Exchange: `clinical.notifications.exchange` (direct)
| Topic | Partition key | Consumer groups |
|---|---|---|
| `observation.recorded` | `encounterId` | `es-indexer`, `sepsis-engine`, `warning-evaluator`, `data-lake-writer` |
| `observation.recorded` | `encounterId` | `es-indexer`, `sepsis-engine`, `warning-evaluator`, `news2-scoring`, `data-lake-writer` |
| `alert.generated` | `encounterId` | `es-indexer`, `notification-publisher`, `data-lake-writer` |
| `encounter.status.changed` | `encounterId` | `es-indexer`, `data-lake-writer` |
All topics use 6 partitions. `KAFKA_AUTO_CREATE_TOPICS_ENABLE=false` — topics are provisioned explicitly by `KafkaTopicProvisioner` to guarantee correct partition count.
**`alert.generated` payload (minimum fields for downstream consumers):**
| Field | Required by | Notes |
|---|---|---|
| `alertId`, `encounterId`, `patientId` | ES indexer, data lake, paging | UUIDs |
| `alertType`, `severity`, `triggeredAt` | All consumers | DB string literals for type/severity |
| `details` | Data lake Parquet | Human-readable breach summary; always set on new alerts |
| `department` | ES indexer | Optional; critical ingest alerts include it |
| `news2Score`, `news2RiskLevel` | ES indexer | Optional; set on `NEWS2_WARNING` / `NEWS2_EMERGENCY` alerts for encounter document projection |
| `partitionKey` | Outbox relay | Same as `encounterId` |
---
## SIRS Criteria
@@ -954,6 +1022,33 @@ When ≥ 2 criteria are active simultaneously (keys present in Redis) for the sa
---
## NEWS2 Scoring
The NEWS2 engine evaluates seven observation codes that overlap with the expanded vital-sign vocabulary from Phase 10:
| Parameter | Observation Code | Score range |
|---|---|---|
| Respiratory rate | `RESP_RATE` | 0–3 |
| Oxygen saturation (Scale 1) | `SPO2` | 0–3 |
| Systolic blood pressure | `SYSTOLIC_BP` | 0–3 |
| Heart rate | `HEART_RATE` | 0–3 |
| Consciousness (AVPU) | `AVPU` | 0 or 3 |
| Temperature | `TEMP_C` | 0–3 |
| Supplemental oxygen | `SUPPLEMENTAL_O2` | 0 or 2 |
**Risk levels** (aggregate score):
| Total score | Risk level | Alert |
|---|---|---|
| 0–4 (no single param = 3) | `LOW` | None |
| Any single param = 3 (total under 5) | `LOW_MEDIUM` | `NEWS2_WARNING` |
| 5–6 | `MEDIUM` | `NEWS2_WARNING` |
| ≥ 7 | `HIGH` | `NEWS2_EMERGENCY` (CRITICAL) |
All seven parameters must be present in Redis (4-hour TTL per key) before a score is computed. Each new observation after completeness triggers a new `news2_scores` row; alert creation is idempotent per encounter and alert type while an alert remains open.
---
## Elasticsearch Index Replay
If the Elasticsearch indices need to be rebuilt (e.g., after a mapping change or data loss):
@@ -1031,7 +1126,7 @@ Observation history uses cursor pagination on `(recorded_at DESC, id DESC)`. Off
## Implemented Phases
Eleven phases from the project roadmap are implemented. Phases 1–10 and Step 4 of Phase 11 are covered by integration tests (`dotnet test` — 64 passing). Phase 11 Step 5 manual verification (docker compose end-to-end) is documented in `docs/plans/phase-11-plan.md`.
Twelve phases from the project roadmap are implemented and verified. Integration tests (`dotnet test` — 126 passing) and per-phase verification scripts cover Phases 8–12.
| Phase | Feature | Status |
|---|---|---|
@@ -1042,9 +1137,10 @@ Eleven phases from the project roadmap are implemented. Phases 1–10 and Step 4
| 5 | Sepsis detection engine (`SepsisEngineService`); Redis SIRS state with 30-min TTL; idempotent alert creation; integration tests | Done |
| 6 | RabbitMQ exchange and queue topology; `NotificationPublisherService`; `PagingWorkerService`; DLQ escalation (`EscalationWorkerService`); discharge summary (`DischargeSummaryWorkerService` → MinIO); integration tests | Done |
| 7 | Reconciliation scheduler — unacknowledged critical alerts, stale pending orders, disconnected monitors; `reconciliation_alerts` table; RabbitMQ publish; integration tests | Done |
| 8 | Prometheus metrics (`GET /metrics`); eight metric families and three collectors; Grafana clinical dashboard; `ObservabilityPhase8Tests`; `run-phase8-verification.sh` | Done |
| 8 | Prometheus metrics (`GET /metrics`); ten metric families and three collectors; Grafana clinical dashboard; `ObservabilityPhase8Tests`; `run-phase8-verification.sh` | Done |
| 9 | Data lake writer — `data-lake-writer` consumer group; date-partitioned Parquet flush to MinIO; `DataLakePhase9Tests`; `run-phase9-verification.sh`; design doc in `docs/decisions/data-lake-design.md` | Done |
| 10 | Clinical data model expansion — `BloodType`, patient allergies/emergency contact, encounter room/bed/admission/discharge fields; five new observation codes (`SYSTOLIC_BP`, `DIASTOLIC_BP`, `LACTATE_MMOL_L`, `AVPU`, `SUPPLEMENTAL_O2`); `GLUCOSE_MG_DL` threshold fix; 12 seeded thresholds; `ClinicalDemographicsAndObservationTests`; `run-phase10-verification.sh` | Done |
| 11 | Warning alert consumer (`WarningAlertService` / `warning-evaluator`); 10 `Warning*` alert types; Orders API (`OrdersController`, `OrderService`); FluentValidation on all request DTOs; `WarningAlertTests`, `OrderLifecycleTests`, `ValidationTests`; `run-phase11-verification.sh` | Done (Step 5 E2E verification via script) |
| 11 | Warning alert consumer (`WarningAlertService` / `warning-evaluator`); 10 `Warning*` alert types; Orders API (`OrdersController`, `OrderService`); FluentValidation on all request DTOs; `WarningAlertTests`, `OrderLifecycleTests`, `ValidationTests`; `run-phase11-verification.sh` | Done |
| 12 | NEWS2 composite scoring (`News2Calculator`, `News2Detector`, `News2ScoringService`); `news2_scores` table; `NEWS2_WARNING` / `NEWS2_EMERGENCY` alert types; `News2Controller` (current + history); ES `news2Score` / `news2RiskLevel` projection; Prometheus NEWS2 metrics; `News2CalculatorTests`, `News2DetectorTests`; `run-phase12-verification.sh` | Done |
**Optional follow-up:** execute and document the Kafka replay demonstration for the data lake (reset `data-lake-writer` offsets, clear MinIO prefixes, restart API, confirm Parquet rebuild). See `docs/plans/phase-9-plan.md` § Replay demonstration.