Trent/vigilcare-clinical
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feature: Trend Detection & Alert Suppression Windows

Browse Source
This commit is contained in:
voltsrage
2026-06-18 21:18:13 +08:00
parent e6f7989298
commit 3c54feb38a
31 changed files with 2751 additions and 18 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+39 -12
View File
@@ -59,12 +59,14 @@ An `OutboxEvent` is written in the same transaction as any observation or alert,
- **Elasticsearch CQRS Projection** — `EsIndexerService` consumer group upserts `patient_encounters` documents, appends to the `observations` index, increments `openAlertCount` on alert events, and stamps `news2Score` / `news2RiskLevel` when a NEWS2 alert is generated; patient/encounter search; per-encounter observation trend (hourly avg/min/max); alert volume summary by department and severity; population query (numeric range aggregation across all patients)
- **Sepsis Early Warning Engine** — `SepsisEngineService` Kafka consumer evaluates SIRS criteria (temperature, heart rate, respiratory rate, WBC) per encounter using Redis keys with a 30-minute TTL sliding window; on ≥2 active criteria, inserts a `SEPSIS_WARNING / CRITICAL` alert idempotently (`INSERT WHERE NOT EXISTS`)
- **NEWS2 Composite Scoring Engine** — `News2ScoringService` Kafka consumer (`news2-scoring`) evaluates seven vital parameters per encounter (`RESP_RATE`, `SPO2`, `SYSTOLIC_BP`, `HEART_RATE`, `AVPU`, `TEMP_C`, `SUPPLEMENTAL_O2`) using Redis keys with a 4-hour TTL; when all seven are present, computes the official NEWS2 aggregate score, persists to `news2_scores`, and creates `NEWS2_WARNING` (score 5–6 or single param = 3) or `NEWS2_EMERGENCY` (score ≥ 7) alerts idempotently; `GET /encounters/:id/news2/current` and `/history` expose score history; Prometheus `news2_scores_total` and `news2_scoring_duration_seconds`
- **Trend Detection Engine** — `TrendAnalyzerService` Kafka consumer (`trend-analyzer`) tracks rate-of-change for five vital parameters (`HEART_RATE`, `RESP_RATE`, `SYSTOLIC_BP`, `TEMP_C`, `SPO2`) using Redis sliding-window history; when velocity exceeds configured thresholds (e.g. 72→95 bpm in 30 min), creates a `RAPID_DETERIORATION` alert even if the current value is below warning thresholds; Prometheus `trend_alerts_total` and `trend_analysis_duration_seconds`
- **Alert Suppression Windows** — acknowledging a suppressible alert (`WARNING_*`, `NEWS2_WARNING`) sets a Redis key `suppress:{encounterId}:{alertType}` with a configurable TTL (default 30 min from `AlertSuppression` config; optional per-code override via `alert_thresholds.suppression_window_minutes`); `WarningEvaluator` and `News2Detector` check suppression before creating new warning alerts; critical alerts (`CRITICAL_*`, `NEWS2_EMERGENCY`, `SEPSIS_WARNING`, `RAPID_DETERIORATION`) are never suppressed; observations and NEWS2 scores continue to persist during suppression; Prometheus `alert_suppressions_total`
- **RabbitMQ Notification Workers** — `NotificationPublisherService` reads `alert.generated` from Kafka and publishes paging jobs to `alerts.paging.queue`; `PagingWorkerService` sends the page and waits for acknowledgment; if no ack arrives before timeout it NACKs to `alerts.paging.dlq` with `x-message-ttl = 300000ms`; if the host is stopping, in-flight paging messages are NACKed with `requeue=true` so they are retried after restart and do not false-escalate; `EscalationWorkerService` pages the on-call backup and sets alert status to `escalated`; `DischargeSummaryWorkerService` reads `encounter.status.changed`, generates a discharge summary, and stores it in MinIO under `/discharge-summaries/{encounterId}/summary.pdf`
- **Data Lake Writer** — `DataLakeWriterService` (consumer group `data-lake-writer`) buffers `observation.recorded`, `alert.generated`, and `encounter.status.changed` events, flushes date-partitioned Parquet files to MinIO (`/observations/`, `/alerts/`, `/encounters/`), and commits Kafka offsets only after at least one successful upload; shutdown flush uses an uncanceled token so MinIO writes complete on Ctrl+C; `kafka_partition` and `kafka_offset` columns provide audit lineage
- **Reconciliation Jobs** — three scheduled checks: (1) unacknowledged CRITICAL alerts older than 30 minutes, (2) pending orders without results after 4 hours, (3) active inpatients with no observation in 2 hours; each finding creates a `reconciliation_alerts` row and publishes to RabbitMQ
- **Standard Envelope** — all responses use a consistent `{ success, statusCode, data, error }` wrapper; validation errors use the same shape; `ApiBehaviorOptions` overridden so model validation also produces the standard envelope with field-level `details`
- **Input Validation** — FluentValidation validators on all request DTOs (patient registration, encounter open, observation ingest, alert acknowledge, alert thresholds, orders); invalid requests return 400 before reaching the service layer
- **Observability** — Serilog structured logging enriched with `correlationId`, `encounterId`, `patientId` on alert paths; Seq sink (`http://localhost:5345`); Prometheus (`http://localhost:9101`) scrapes `GET /metrics`; ten application metric families via `ClinicalMetrics` and three background collectors (`AlertsUnacknowledgedCollector`, `OutboxPendingCollector`, `KafkaConsumerLagCollector`); Grafana clinical dashboard (`http://localhost:3101`, admin/admin) with `alerts_unacknowledged_gauge` as the primary safety panel; per-request correlation IDs in request logs and `X-Correlation-Id` response headers
- **Observability** — Serilog structured logging enriched with `correlationId`, `encounterId`, `patientId` on alert paths; Seq sink (`http://localhost:5345`); Prometheus (`http://localhost:9101`) scrapes `GET /metrics`; thirteen application metric families via `ClinicalMetrics` and three background collectors (`AlertsUnacknowledgedCollector`, `OutboxPendingCollector`, `KafkaConsumerLagCollector`); Grafana clinical dashboard (`http://localhost:3101`, admin/admin) with `alerts_unacknowledged_gauge` as the primary safety panel; per-request correlation IDs in request logs and `X-Correlation-Id` response headers
- **Swagger UI** — OpenAPI spec via Swashbuckle (Development only)
---
@@ -78,7 +80,7 @@ HTTP request
→ Controllers
→ Services
├── PostgreSQL (EF Core — writes, keyed reads)
├── Redis (threshold cache, SIRS state, NEWS2 parameter state)
├── Redis (threshold cache, SIRS state, NEWS2 parameter state, trend history, alert suppression keys)
└── OutboxEvent (same transaction as domain write)
IHostedServices (background):
@@ -91,6 +93,8 @@ IHostedServices (background):
SepsisEngineService → Kafka → Redis SIRS state → PostgreSQL alert (consumer group: sepsis-engine)
WarningAlertService → Kafka → WarningEvaluator → PostgreSQL WARNING alert (consumer group: warning-evaluator)
News2ScoringService → Kafka → News2Detector → Redis NEWS2 state → PostgreSQL score + alert (consumer group: news2-scoring)
TrendAnalyzerService → Kafka → TrendDetector → Redis trend history → PostgreSQL RAPID_DETERIORATION alert (consumer group: trend-analyzer)
AlertSuppressionService → Redis suppress:{enc}:{type} keys set on acknowledge; read by WarningEvaluator + News2Detector
NotificationPublisherService → Kafka → RabbitMQ paging.queue (consumer group: notification-publisher)
PagingWorkerService → RabbitMQ paging.queue → log page → NACK on timeout (or requeue on shutdown)
EscalationWorkerService → RabbitMQ escalation.queue → update alert status
@@ -100,7 +104,7 @@ IHostedServices (background):
AlertsUnacknowledgedCollector → polls PostgreSQL every 30s → alerts_unacknowledged_gauge
OutboxPendingCollector → polls outbox every 30s → outbox_pending_events
KafkaConsumerLagCollector → polls four consumer groups every 30s → kafka_consumer_lag
ClinicalMetrics (singleton) → inline counters/histogram from ingest, SIRS, NEWS2, escalation paths
ClinicalMetrics (singleton) → inline counters/histogram from ingest, SIRS, NEWS2, trend, suppression, escalation paths
```
**Why Kafka and RabbitMQ coexist:** Kafka is an append-only log — the same observation event reaches the Elasticsearch indexer, the sepsis engine, and the data lake independently without coordination. Each consumer holds its own offset and can replay from the beginning. RabbitMQ handles the action side: one message, one worker, one page. A duplicate page at 3am is a patient safety concern, not a minor inconvenience — RabbitMQ's acknowledgment-then-delete model is correct here. The DLQ TTL-based escalation has no equivalent in Kafka.
@@ -174,16 +178,20 @@ VigilCareClinicalAPI/
│ ├── AlertThresholdService.cs # CRUD + Redis write-through invalidation
│ ├── ObservationService.cs # Ingest transaction: idempotency → plausibility → threshold → alert → outbox; emits Prometheus counters
│ ├── ObservationQueryService.cs # Cursor-paginated history
│ ├── AlertService.cs # Acknowledge, resolve, list
│ ├── AlertService.cs # Acknowledge (sets suppression), resolve, list
│ ├── AlertSuppressionService.cs # Redis suppress:{enc}:{type} TTL keys
│ ├── OrderService.cs # Order lifecycle; status machine; ConflictException on illegal transitions
│ ├── News2Service.cs # Current score + cursor-paginated history from PostgreSQL
│ ├── WarningEvaluator.cs # Warning-range threshold evaluation; idempotent alert INSERT
│ ├── WarningEvaluator.cs # Warning-range evaluation; suppression check; idempotent alert INSERT
│ ├── AnalyticsService.cs # Elasticsearch query wrappers
│ └── PlausibilityValidator.cs # Per-code numeric range guard
├── Trend/
│ ├── TrendCalculator.cs # Pure static rate-of-change logic
│ └── TrendDetector.cs # Redis history + RAPID_DETERIORATION alert creation
├── Validators/ # FluentValidation — RegisterPatient, OpenEncounter, IngestObservation, …
├── Observability/
│ └── Metrics/
│ └── ClinicalMetrics.cs # Ten Prometheus metric families (counters, histograms, gauges)
│ └── ClinicalMetrics.cs # Thirteen Prometheus metric families (counters, histograms, gauges)
├── BackgroundServices/
│ ├── ThresholdCacheLoader.cs # Pre-loads all thresholds into Redis on startup
│ ├── KafkaTopicProvisioner.cs # Creates topics with NumPartitions from config
@@ -198,6 +206,7 @@ VigilCareClinicalAPI/
│ ├── SepsisEngineService.cs # consumer group: sepsis-engine; SIRS eval via Redis TTL keys
│ ├── WarningAlertService.cs # consumer group: warning-evaluator; observation.recorded → WARNING alerts
│ ├── News2ScoringService.cs # consumer group: news2-scoring; observation.recorded → NEWS2 score + alert
│ ├── TrendAnalyzerService.cs # consumer group: trend-analyzer; observation.recorded → RAPID_DETERIORATION alert
│ ├── Notifications/
│ │ ├── NotificationPublisherService.cs # consumer group: notification-publisher; alert.generated → RabbitMQ paging.queue
│ │ ├── PagingWorkerService.cs # RabbitMQ consumer; logs page; NACK on ack timeout → DLQ, requeue on graceful shutdown
@@ -274,7 +283,10 @@ tests/
├── OrderLifecycleTests.cs # Orders API — create, list, record result, illegal transition 409
├── ValidationTests.cs # FluentValidation — empty fields, threshold ordering, order description
├── News2CalculatorTests.cs # Boundary tests for all seven NEWS2 scoring tables
└── News2DetectorTests.cs # NEWS2 detector — score tiers, alerts, idempotency, incomplete set
├── News2DetectorTests.cs # NEWS2 detector — score tiers, alerts, idempotency, incomplete set
├── TrendCalculatorTests.cs # Pure unit tests — rate-of-change, threshold direction, describe
├── TrendDetectorTests.cs # Trend detector — rapid climb, stable, idempotent, non-trend code
└── AlertSuppressionTests.cs # Suppression on acknowledge, read-side skip, TTL expiry
scripts/
├── run-api-redis-tests.sh # Phase 1 — patient/encounter/threshold + Redis cache
@@ -287,10 +299,11 @@ scripts/
├── run-phase9-verification.sh # Phase 9 — data lake tests, Kafka offsets, MinIO Parquet, DuckDB schema
├── run-phase10-verification.sh # Phase 10 — 12 Redis thresholds, clinical enrichment, ES pipeline, integration tests
├── run-phase11-verification.sh # Phase 11 — warning alerts, orders API, validation, integration tests
└── run-phase12-verification.sh # Phase 12 — NEWS2 end-to-end pipeline, API, ES, Prometheus, integration tests
├── run-phase12-verification.sh # Phase 12 — NEWS2 end-to-end pipeline, API, ES, Prometheus, integration tests
└── run-phase13-verification.sh # Phase 13 — trend detection, alert suppression, consumer lag, integration tests
docs/
├── plans/ # Phase 1–12 implementation and verification guides
├── plans/ # Phase 1–13 implementation and verification guides
├── decisions/
│ ├── data-lake-design.md # Parquet vs JSON, partitioning, replay rationale
│ └── sepsis-engine-design.md # SIRS sliding window and idempotent alert design
@@ -438,6 +451,9 @@ Integration tests use `WebApplicationFactory` with a `Testing` environment and T
| `ValidationTests` | 11 | FluentValidation 400 on empty first name, invalid threshold order, empty order description |
| `News2CalculatorTests` | 12 | Boundary tests for all seven NEWS2 scoring tables and risk-level determination |
| `News2DetectorTests` | 12 | NEWS2 detector — score tiers, alert creation, incomplete parameters, idempotency |
| `TrendCalculatorTests` | 13 | Pure unit tests — delta/time rate, SPO2/BP decline direction, describe formatting |
| `TrendDetectorTests` | 13 | Trend detector — rapid HR climb alert, stable high HR, idempotency, non-trend code |
| `AlertSuppressionTests` | 13 | Acknowledge sets Redis key, suppressed warning skipped, critical/NEWS2 emergency never suppressed, TTL expiry |
### Verification Scripts
@@ -449,6 +465,13 @@ With the API running (`dotnet run`) and Docker Compose up:
./scripts/run-phase10-verification.sh # 12 Redis thresholds, clinical enrichment, ES pipeline, Phase 10 integration tests
./scripts/run-phase11-verification.sh # Warning alert pipeline, orders API, FluentValidation, Phase 11 integration tests
./scripts/run-phase12-verification.sh # NEWS2 end-to-end pipeline, API, Elasticsearch, Prometheus, Phase 12 integration tests
./scripts/run-phase13-verification.sh # Trend detection, alert suppression, consumer lag, Phase 13 integration tests
```
Phase 13 unit/integration tests only:
```bash
dotnet test --filter "FullyQualifiedName~Trend|FullyQualifiedName~Suppression"
```
Per-phase test runners (subset of `dotnet test`):
@@ -481,16 +504,19 @@ See `docs/plans/phase-8-plan.md` through `docs/plans/phase-12-plan.md` for manua
## Prometheus Metrics
`GET /metrics` exposes ten application metric families registered in `ClinicalMetrics`. Three background collectors poll PostgreSQL and Kafka every 30 seconds; counters and histograms are updated inline during request handling and background processing.
`GET /metrics` exposes thirteen application metric families registered in `ClinicalMetrics`. Three background collectors poll PostgreSQL and Kafka every 30 seconds; counters and histograms are updated inline during request handling and background processing.
| Metric | Type | Labels | Source |
|---|---|---|---|
| `observations_ingested_total` | Counter | `observation_code`, `source` | `ObservationService` on each committed observation |
| `observation_ingest_duration_seconds` | Histogram | — | `ObservationService` — full ingest transaction to COMMIT |
| `clinical_alerts_total` | Counter | `alert_type`, `severity` | `ObservationService` (threshold breach), `SirsDetector`, `News2Detector` |
| `clinical_alerts_total` | Counter | `alert_type`, `severity` | `ObservationService`, `SirsDetector`, `News2Detector`, `TrendDetector`, `WarningEvaluator` |
| `sirs_detections_total` | Counter | — | `SirsDetector` — only on successful idempotent insert |
| `news2_scores_total` | Counter | `risk_level` | `News2Detector` — on each persisted score (`LOW`, `MEDIUM`, `HIGH`, …) |
| `news2_scoring_duration_seconds` | Histogram | — | `News2Detector` — Redis update through score persistence |
| `trend_alerts_total` | Counter | `observation_code` | `TrendDetector` — on each `RAPID_DETERIORATION` alert created |
| `trend_analysis_duration_seconds` | Histogram | — | `TrendDetector` — per-observation trend evaluation |
| `alert_suppressions_total` | Counter | `alert_type` | `AlertSuppressionService` — on each suppression window set after acknowledge |
| `escalations_total` | Counter | — | `EscalationWorkerService` on DLQ escalation |
| `alerts_unacknowledged_gauge` | Gauge | — | `AlertsUnacknowledgedCollector` — open CRITICAL alerts older than 5 minutes |
| `outbox_pending_events` | Gauge | — | `OutboxPendingCollector` — unprocessed outbox rows |
@@ -988,7 +1014,7 @@ Exchange: `clinical.notifications.exchange` (direct)
| Topic | Partition key | Consumer groups |
|---|---|---|
| `observation.recorded` | `encounterId` | `es-indexer`, `sepsis-engine`, `warning-evaluator`, `news2-scoring`, `data-lake-writer` |
| `observation.recorded` | `encounterId` | `es-indexer`, `sepsis-engine`, `warning-evaluator`, `news2-scoring`, `trend-analyzer`, `data-lake-writer` |
| `alert.generated` | `encounterId` | `es-indexer`, `notification-publisher`, `data-lake-writer` |
| `encounter.status.changed` | `encounterId` | `es-indexer`, `data-lake-writer` |
@@ -1142,5 +1168,6 @@ Twelve phases from the project roadmap are implemented and verified. Integration
| 10 | Clinical data model expansion — `BloodType`, patient allergies/emergency contact, encounter room/bed/admission/discharge fields; five new observation codes (`SYSTOLIC_BP`, `DIASTOLIC_BP`, `LACTATE_MMOL_L`, `AVPU`, `SUPPLEMENTAL_O2`); `GLUCOSE_MG_DL` threshold fix; 12 seeded thresholds; `ClinicalDemographicsAndObservationTests`; `run-phase10-verification.sh` | Done |
| 11 | Warning alert consumer (`WarningAlertService` / `warning-evaluator`); 10 `Warning*` alert types; Orders API (`OrdersController`, `OrderService`); FluentValidation on all request DTOs; `WarningAlertTests`, `OrderLifecycleTests`, `ValidationTests`; `run-phase11-verification.sh` | Done |
| 12 | NEWS2 composite scoring (`News2Calculator`, `News2Detector`, `News2ScoringService`); `news2_scores` table; `NEWS2_WARNING` / `NEWS2_EMERGENCY` alert types; `News2Controller` (current + history); ES `news2Score` / `news2RiskLevel` projection; Prometheus NEWS2 metrics; `News2CalculatorTests`, `News2DetectorTests`; `run-phase12-verification.sh` | Done |
| 13 | Trend detection (`TrendCalculator`, `TrendDetector`, `TrendAnalyzerService`); `RAPID_DETERIORATION` alert type; alert suppression windows (`AlertSuppressionService`, Redis `suppress:{enc}:{type}`); `TrendCalculatorTests`, `TrendDetectorTests`, `AlertSuppressionTests`; `run-phase13-verification.sh` | Done |
**Optional follow-up:** execute and document the Kafka replay demonstration for the data lake (reset `data-lake-writer` offsets, clear MinIO prefixes, restart API, confirm Parquet rebuild). See `docs/plans/phase-9-plan.md` § Replay demonstration.