932 lines
65 KiB
Markdown
932 lines
65 KiB
Markdown
# PRD: VigilCare — Clinical Data Pipeline & Real-Time Alert Platform
|
||
|
||
## Overview
|
||
|
||
A production-quality clinical backend built with ASP.NET Core 8, PostgreSQL, Apache Kafka, RabbitMQ, Elasticsearch, Redis, and MinIO. The system models patient encounters, continuous observation ingest, and real-time clinical alerting with composite scoring engines (NEWS2, GCS, SOFA), Sepsis-3 two-tier detection (qSOFA screening → SOFA confirmation → treatment bundle), trend analysis, and alert suppression. It streams vital signs and lab results through Kafka, fans urgent notifications to clinicians through RabbitMQ with DLQ-based escalation, and maintains a searchable CQRS projection in Elasticsearch for ward dashboards and population analytics. A Vue 3 ward dashboard provides real-time clinical views. Ward gateway edge nodes buffer observations locally during connectivity loss. A FHIR R4 facade enables EHR integration. JWT-based RBAC with clinical audit logging gates every endpoint. Long-term data is archived as Parquet files in MinIO — a regulatory requirement in healthcare that has no equivalent in most other domains. Twenty-nine phases are implemented and verified.
|
||
|
||
The domain is deliberately different from the Digital Wallet API. Both projects use Kafka, RabbitMQ, and Elasticsearch, but the trade-off conversations are entirely different. In fintech the core question is "did the money move correctly?" In healthcare the core question is "did the right person get the right alert at the right time?" That distinction — correctness vs timeliness — produces different architectural decisions at every layer.
|
||
|
||
This project maps to `sd-mid-009` (Outbox Pattern), `sd-mid-013` (CQRS), `sd-mid-043–048` (Kafka internals), `sd-senior-008` (Real-Time Event Processing), and `sd-senior-011` (Anomaly Detection in Streams).
|
||
|
||
**Stack:** .NET 8 Web API, PostgreSQL, Apache Kafka (KRaft), RabbitMQ, Elasticsearch, Redis, MinIO (Parquet archival), Serilog → Seq, Prometheus → Grafana, Vue 3 + Vite + Pinia + Tailwind CSS + Chart.js, Hl7.Fhir.R4 (Firely SDK), JWT + BCrypt, FluentValidation, xUnit + Testcontainers, Docker Compose.
|
||
|
||
---
|
||
|
||
## Local Development Setup
|
||
|
||
Start all infrastructure services with:
|
||
|
||
```bash
|
||
docker compose up -d
|
||
```
|
||
|
||
| Service | Port | Notes |
|
||
|---|---|---|
|
||
| PostgreSQL | 5436 | Database: `vigilcare`, user: `postgres`, password: `password` |
|
||
| Redis | 6382 | No auth |
|
||
| Seq | 5345 | UI at `http://localhost:5345` — login: `admin` / `admin` |
|
||
| Kafka | 9092 | KRaft mode, no Zookeeper |
|
||
|
||
**Seq first-run:** `SEQ_FIRSTRUN_ADMINPASSWORD=admin` is set in `docker-compose.yml`. This password is only applied on the very first container start (when `/data` volume is empty). After initialization, the password is stored in the volume and this env var is ignored.
|
||
|
||
---
|
||
|
||
## Goals
|
||
|
||
- Model the observe-alert-acknowledge lifecycle that sits at the center of any clinical monitoring system
|
||
- Demonstrate Kafka's multi-consumer log model in a healthcare context where the same observation event must reach the alert engine, the Elasticsearch projection, and the data lake independently
|
||
- Show RabbitMQ's DLQ pattern as a clinical escalation mechanism — if a critical alert is not acknowledged in five minutes, the message routes through a dead-letter queue and re-delivers as an escalation to the on-call physician
|
||
- Build stateful Kafka consumers that detect sepsis (qSOFA/SOFA), compute composite scores (NEWS2, GCS), and track trends by maintaining rolling windows of recent observations per patient in Redis
|
||
- Implement a two-tier sepsis detection pathway (Sepsis-3: qSOFA screen → SOFA confirmation → treatment bundle) that demonstrates clinical workflow automation
|
||
- Provide a Vue 3 ward dashboard with real-time clinical views, scoring history charts, and clinician feedback collection
|
||
- Enable EHR integration via a FHIR R4 inbound facade with LOINC/SNOMED code mapping
|
||
- Support edge deployment via ward gateway nodes with offline buffering and central sync
|
||
- Secure all endpoints with JWT-based RBAC and maintain an append-only clinical audit trail
|
||
- Produce a project that supports senior trade-off conversations in healthcare, medtech, and any domain where real-time alerting and long-term archival coexist
|
||
|
||
## Non-Goals
|
||
|
||
- ~~HL7 FHIR compliance~~ → **Implemented in Phase 30** — FHIR R4 inbound facade with LOINC/SNOMED code mapping, transaction Bundles, and read/search endpoints; Mirth Connect integration guide
|
||
- Integration with real medical devices or lab information systems
|
||
- ~~Medication dispensing or pharmacy workflows~~ → **Partially addressed in Phase 15** — medication administration recording with drug-vital correlation annotations on alerts
|
||
- Patient billing or insurance claim adjudication
|
||
- HIPAA-compliant deployment (model the patterns; don't configure real PHI)
|
||
|
||
---
|
||
|
||
## Why Both Kafka and RabbitMQ?
|
||
|
||
This is the same architectural question as the Digital Wallet — but the healthcare context produces a different answer that is worth knowing independently.
|
||
|
||
**Kafka** is an append-only log. Every observation recorded by a bedside monitor, every lab result that arrives from the lab information system, is written to a Kafka topic and retained. Multiple independent consumer groups read the same observation stream at their own pace:
|
||
|
||
- The Elasticsearch indexer maintains a searchable patient dashboard
|
||
- The sepsis detection engine analyzes rolling windows for qSOFA/SOFA criteria (Sepsis-3)
|
||
- The data lake writer archives observations as Parquet for long-term regulatory retention
|
||
- A future billing consumer could derive charges from observation codes without touching the operational database
|
||
|
||
None of these consumers coordinate with each other. Each holds its own offset. If the sepsis engine is deployed a month after go-live, it can replay all historical observations from offset 0 to catch up. This is only possible because Kafka retains events after consumption.
|
||
|
||
**RabbitMQ** handles the action side — what must happen after a clinical event is recognized. When the alert engine detects a critical potassium value, a clinician must be paged. That page is a task: one message, one worker, one action. It must not be processed twice (a duplicate page at 3am is a patient safety concern, not a minor inconvenience). RabbitMQ's acknowledgment model — the message is deleted after exactly one worker acknowledges it — is correct here. Kafka's model is not.
|
||
|
||
The escalation pattern makes RabbitMQ's dead-letter queue uniquely valuable in this domain. If a physician does not acknowledge a critical alert within five minutes, the original message NACKs into a dead-letter queue with a `x-message-ttl` of 300 seconds. After that TTL expires, the message is re-routed to an escalation queue targeting the on-call backup. This is the DLQ pattern repurposed as a clinical escalation protocol — a design that does not exist cleanly in Kafka.
|
||
|
||
| Use Case | System | Why |
|
||
|---|---|---|
|
||
| Vital sign streams from monitors | Kafka | Continuous, high-frequency, multiple consumers |
|
||
| Lab result events from LIS | Kafka | Replayable; alert engine and data lake both need it |
|
||
| Encounter admission/discharge events | Kafka | Multiple downstream systems react independently |
|
||
| Sepsis detection analytics | Kafka → Redis | Stateful windowed analysis over the observation stream |
|
||
| Page a physician for a critical value | RabbitMQ | One task, one worker, acknowledged-then-deleted |
|
||
| Escalate if unacknowledged after 5 minutes | RabbitMQ DLQ | Delayed re-delivery is native to DLQ TTL; Kafka has no equivalent |
|
||
| Generate discharge summary PDF | RabbitMQ | Background job; one per discharge, not replayable |
|
||
| Appointment reminder SMS | RabbitMQ | Task queue; idempotent at the SMS provider level |
|
||
|
||
---
|
||
|
||
## API Conventions
|
||
|
||
Same response envelope as all other portfolio projects. Prefix: `/api/v1`.
|
||
|
||
**Success:**
|
||
```json
|
||
{
|
||
"success": true,
|
||
"statusCode": 200,
|
||
"data": {},
|
||
"error": null
|
||
}
|
||
```
|
||
|
||
**Error:**
|
||
```json
|
||
{
|
||
"success": false,
|
||
"statusCode": 422,
|
||
"data": null,
|
||
"error": {
|
||
"message": "Observation value exceeds plausible range for this code.",
|
||
"code": "OBSERVATION_OUT_OF_PLAUSIBLE_RANGE"
|
||
}
|
||
}
|
||
```
|
||
|
||
**Pagination:** Observation history and alert history use cursor pagination on `(recorded_at DESC, id DESC)` — the table is append-only and grows continuously; offset pagination shifts results as new rows arrive. All other list endpoints use offset pagination (`?page=1&pageSize=20`).
|
||
|
||
**Idempotency:** `POST /api/v1/encounters/:id/observations` accepts an `Idempotency-Key` header enforced by a unique partial index. Medical device integrations frequently retry on network failure; a duplicate reading must produce the same response without creating a duplicate observation.
|
||
|
||
---
|
||
|
||
## Domain Model
|
||
|
||
A **patient** is the central entity. Each patient has a Medical Record Number (MRN) — a stable identifier issued at first registration that never changes, even across multiple encounters.
|
||
|
||
An **encounter** is a single clinical episode — an inpatient admission, an outpatient visit, or an emergency department visit. A patient may have many encounters over their lifetime. An encounter has a status (`scheduled`, `active`, `discharged`, `cancelled`) and a department. All observations, orders, and alerts belong to an encounter, not directly to a patient.
|
||
|
||
An **alert threshold** defines the numeric boundaries that trigger a clinical alert for a given observation code. Thresholds are global (not per-patient) and are managed by clinical administrators. Each threshold has four optional bounds: `critical_low`, `warning_low`, `warning_high`, `critical_high`. A potassium value below `critical_low` is an immediate life-threatening emergency; a value below `warning_low` warrants physician review within the hour.
|
||
|
||
An **observation** is a single recorded measurement: a vital sign (heart rate, temperature, blood pressure), a lab value (potassium, glucose, white blood cell count), or a pulse oximetry reading. Observations are append-only. They are never updated or deleted. The observation stream is the primary input to both the alert engine and the Kafka pipeline.
|
||
|
||
A **clinical alert** is generated when an observation breaches a threshold or when the sepsis detection engine identifies a pattern across multiple recent observations. An alert has a lifecycle: `open` → `acknowledged` → `resolved`. An unacknowledged `CRITICAL` alert triggers the RabbitMQ escalation after five minutes.
|
||
|
||
An **order** is a clinician's instruction: run this lab test, administer this medication, perform this imaging study. Orders have a status lifecycle and a `resulted_at` timestamp. The reconciliation job uses pending orders to detect cases where a result was never returned.
|
||
|
||
An **outbox event** is written in the same transaction as any observation or alert, then relayed to Kafka by a background worker. This decouples Kafka availability from the database transaction.
|
||
|
||
---
|
||
|
||
## Features
|
||
|
||
---
|
||
|
||
### 1. Patient Registration and Encounter Management
|
||
|
||
**Description:** Patients are registered with demographic information and assigned a unique MRN. When a patient presents for care, an encounter is opened against their record. Encounters progress through a controlled status machine. A patient cannot have two active encounters of the same type simultaneously.
|
||
|
||
**Endpoints:**
|
||
- `POST /api/v1/patients` — register patient; generates MRN
|
||
- `GET /api/v1/patients` — paginated list with search by name or MRN
|
||
- `GET /api/v1/patients/:id` — patient detail with active encounter summary
|
||
- `POST /api/v1/patients/:id/encounters` — open an encounter
|
||
- `GET /api/v1/encounters/:id` — encounter detail with recent observations and open alerts
|
||
- `PATCH /api/v1/encounters/:id/status` — advance status (`active → discharged`, `scheduled → active`, etc.); illegal transitions return `409`
|
||
- `GET /api/v1/encounters/:id/timeline` — merged chronological view: status changes, observation summaries, alerts
|
||
|
||
**Encounter status machine:**
|
||
```
|
||
scheduled → active → discharged
|
||
→ cancelled
|
||
```
|
||
A `discharged` encounter triggers a RabbitMQ job to generate a discharge summary PDF.
|
||
|
||
**Concepts practiced:** Aggregate design (encounter owns observations and alerts), controlled state transitions with explicit transition matrix, 409 on illegal transitions, timeline as a composed projection across multiple tables.
|
||
|
||
---
|
||
|
||
### 2. Alert Threshold Management
|
||
|
||
**Description:** Clinical administrators configure the numeric boundaries that define normal, warning, and critical ranges for each observation code. Thresholds are cached in Redis at application startup and invalidated on write — they are read on every observation ingest and must not add database latency to the ingest path.
|
||
|
||
**Endpoints:**
|
||
- `POST /api/v1/alert-thresholds`
|
||
- `GET /api/v1/alert-thresholds`
|
||
- `GET /api/v1/alert-thresholds/:id`
|
||
- `PUT /api/v1/alert-thresholds/:id`
|
||
|
||
**Data:**
|
||
```json
|
||
{
|
||
"id": "uuid",
|
||
"observationCode": "POTASSIUM_MEQ_L",
|
||
"displayName": "Serum Potassium",
|
||
"unit": "mEq/L",
|
||
"criticalLow": 2.5,
|
||
"warningLow": 3.5,
|
||
"warningHigh": 5.0,
|
||
"criticalHigh": 6.5
|
||
}
|
||
```
|
||
|
||
**Concepts practiced:** Redis as a configuration cache (not just session/balance cache), cache-aside with write-through invalidation, the difference between data that changes per-request (patient observations) and data that changes per-configuration (thresholds).
|
||
|
||
---
|
||
|
||
### 3. Observation Ingest
|
||
|
||
**Description:** The highest-volume endpoint in the system. Bedside monitors, point-of-care devices, and lab integration systems POST observations continuously. The endpoint must be idempotent (devices retry on network failure), must validate the value against a plausibility range (no human has a heart rate of 400), and must evaluate the observation against alert thresholds on the synchronous path for critical values.
|
||
|
||
**Endpoints:**
|
||
- `POST /api/v1/encounters/:id/observations` — single or small batch (up to 10); document choice
|
||
- `GET /api/v1/encounters/:id/observations?code=&from=&to=&limit=&cursor=` — cursor-paginated observation history with optional code filter
|
||
|
||
**Ingest transaction sequence:**
|
||
```
|
||
1. Validate encounter is active (not discharged or cancelled)
|
||
2. Check idempotency key against unique index
|
||
3. Validate observation value within plausible range for the code
|
||
4. Insert observation row
|
||
5. Load alert threshold for this code from Redis cache (→ PostgreSQL on miss)
|
||
6. If value breaches CRITICAL threshold:
|
||
a. Insert clinical_alert row (status: open)
|
||
b. Insert outbox event (topic: alert.generated)
|
||
7. Insert outbox event (topic: observation.recorded, payload: full observation)
|
||
8. COMMIT
|
||
```
|
||
|
||
**Critical vs warning split:** Critical threshold breaches are detected synchronously within the ingest transaction and immediately create an alert. Warning threshold breaches are detected by the Kafka consumer asynchronously — the additional latency (milliseconds to seconds) is acceptable for a warning, but a critical potassium value must trigger a page before the API returns a response.
|
||
|
||
**Idempotency:** A device that retries an observation with the same `Idempotency-Key` receives the original `201` response without creating a duplicate row. The unique partial index enforces this at the database layer.
|
||
|
||
**Concepts practiced:** Idempotency key on high-frequency ingest (sd-mid-008), synchronous vs asynchronous alert detection (the split is a clinical safety decision, not an arbitrary one), Redis cache-aside for threshold lookup on the hot path, outbox pattern within ingest transaction.
|
||
|
||
---
|
||
|
||
### 4. Clinical Alert Lifecycle
|
||
|
||
**Description:** Alerts are the patient safety core of the system. Every open `CRITICAL` alert must be acknowledged by a clinician within five minutes or it escalates. Every alert has a documented audit trail: who acknowledged it, when, and with what note.
|
||
|
||
**Endpoints:**
|
||
- `GET /api/v1/encounters/:id/alerts` — paginated alert list for an encounter
|
||
- `GET /api/v1/alerts` — global alert list filterable by status, severity, department
|
||
- `GET /api/v1/alerts/:id` — alert detail
|
||
- `POST /api/v1/alerts/:id/acknowledge` — acknowledge with clinician ID and optional note
|
||
- `POST /api/v1/alerts/:id/resolve` — resolve (must be acknowledged first)
|
||
|
||
**Alert lifecycle:**
|
||
```
|
||
open → acknowledged → resolved
|
||
→ escalated (via RabbitMQ DLQ after 5 min unacknowledged)
|
||
```
|
||
|
||
**The escalation path:** When an alert is created, the outbox relay publishes to `alert.generated` in Kafka. The notification worker Kafka consumer reads this event and publishes a paging job to RabbitMQ. If the RabbitMQ consumer sends the page but receives no acknowledgment event within five minutes, the message NACKs to the DLQ. After the DLQ TTL expires (300 seconds), the message re-routes to the escalation queue and the on-call backup is paged. The alert status transitions to `escalated` in PostgreSQL.
|
||
|
||
**Concepts practiced:** Alert acknowledgment as a domain event (not just a status update), escalation via DLQ TTL as a healthcare-specific pattern, the difference between an alert being acknowledged in the app vs a clinician physically responding at the bedside.
|
||
|
||
---
|
||
|
||
### 5. Outbox Relay and Kafka Pipeline
|
||
|
||
**Description:** Same pattern as the Digital Wallet. The relay reads unprocessed outbox rows, publishes to Kafka, marks processed. Every observation and every alert flows through this relay to reach Elasticsearch, the sepsis engine, and the data lake independently.
|
||
|
||
**Kafka topics:**
|
||
|
||
| Topic | Producer | Consumers |
|
||
|---|---|---|
|
||
| `observation.recorded` | Outbox relay | `es-indexer`, `sepsis-engine` (qSOFA), `warning-evaluator`, `news2-scoring`, `gcs-scoring`, `sofa-scoring`, `trend-analyzer`, `data-lake-writer` |
|
||
| `alert.generated` | Outbox relay | `es-indexer`, `notification-publisher`, `data-lake-writer` |
|
||
| `encounter.status.changed` | Outbox relay | `es-indexer`, `data-lake-writer` |
|
||
| `gcs.scored` | Outbox relay (via GcsDetector) | `sofa-scoring` (CNS organ-system re-scoring) |
|
||
| `sepsis.bundle.created` | Outbox relay | `es-indexer` |
|
||
| `sepsis.bundle.updated` | Outbox relay | `es-indexer` |
|
||
|
||
All topics use 6 partitions. `KAFKA_AUTO_CREATE_TOPICS_ENABLE=false` — topics are provisioned explicitly by `KafkaTopicProvisioner`.
|
||
|
||
**Partition key:** `encounter_id` for all topics. All events for the same encounter land on the same partition, preserving per-encounter ordering. This is important for the sepsis engine: observations for the same patient must be processed in arrival order.
|
||
|
||
**Consumer group isolation:** `es-indexer`, `sepsis-engine`, and `data-lake-writer` are separate consumer groups. Each maintains its own committed offset. The sepsis engine processing slowly does not affect the Elasticsearch indexer.
|
||
|
||
**Concepts practiced:** Partition key design for per-entity ordering guarantees, consumer group independence, at-least-once delivery via outbox relay (and why consumers must be idempotent), Kafka as the backbone that allows adding new consumers without modifying the producer.
|
||
|
||
---
|
||
|
||
### 6. Elasticsearch Clinical Search and Analytics (CQRS)
|
||
|
||
**Description:** The Elasticsearch indexer maintains a denormalized, queryable projection of the clinical record. It is the read side of CQRS — PostgreSQL is always the write side and the source of truth. The index is optimized for the queries clinicians actually run: "show me all patients with a critical potassium alert in the last hour," "show me the average heart rate trend for this patient over the last 24 hours."
|
||
|
||
**Index shape — patient_encounters:**
|
||
```json
|
||
{
|
||
"encounterId": "uuid",
|
||
"patientId": "uuid",
|
||
"mrn": "MRN-000001",
|
||
"patientName": "Jane Smith",
|
||
"department": "ICU",
|
||
"status": "active",
|
||
"attendingPhysician": "Dr. Osei",
|
||
"admittedAt": "2025-01-01T08:00:00Z",
|
||
"openAlertCount": 2,
|
||
"lastObservationAt": "2025-01-01T09:45:00Z"
|
||
}
|
||
```
|
||
|
||
**Index shape — observations:**
|
||
```json
|
||
{
|
||
"observationId": "uuid",
|
||
"encounterId": "uuid",
|
||
"patientId": "uuid",
|
||
"mrn": "MRN-000001",
|
||
"observationCode": "HEART_RATE",
|
||
"value": 118.0,
|
||
"unit": "bpm",
|
||
"source": "DEVICE",
|
||
"recordedAt": "2025-01-01T09:45:00Z"
|
||
}
|
||
```
|
||
|
||
**Endpoints:**
|
||
- `GET /api/v1/analytics/patients?q=&department=&status=` — patient/encounter search across MRN, name, department
|
||
- `GET /api/v1/analytics/observations/trend?encounterId=&code=&from=&to=` — time-series aggregation (hourly average, min, max) for a specific observation code
|
||
- `GET /api/v1/analytics/alerts/summary?severity=&from=&to=&department=` — alert volume by department and severity over a time window
|
||
- `GET /api/v1/analytics/population?code=&threshold=&from=&to=` — how many patients had a value above or below a threshold in a given window
|
||
|
||
**The replay demo:** Stop the indexer → delete the Elasticsearch index → reset the `es-indexer` consumer group offset to 0 → restart → watch both indices rebuild from Kafka history. This is only possible because Kafka retains events. Document this procedure in the project README. It is the most important operational proof-of-concept in the project.
|
||
|
||
**Why Elasticsearch here and not PostgreSQL:** The `population` query — "how many active patients have a heart rate above 100 in the last hour across all departments" — is an aggregation across potentially millions of observation rows. Running this against PostgreSQL on the operational database would compete with ingest writes and introduce latency for both. Elasticsearch's aggregation engine is purpose-built for this pattern. PostgreSQL remains untouched for this query.
|
||
|
||
**Concepts practiced:** CQRS read projection design (sd-mid-013), Elasticsearch aggregations as a distinct use case from full-text search (the `population` endpoint uses no full-text search at all — it is a numeric range aggregation), eventual consistency between PostgreSQL and Elasticsearch, replay as a recovery mechanism.
|
||
|
||
---
|
||
|
||
### 7. Sepsis Detection Engine (Sepsis-3: qSOFA Screen → SOFA Confirmation)
|
||
|
||
**Description:** A two-tier sepsis detection pathway following the Sepsis-3 consensus (2016), replacing the original SIRS-based approach. The first tier is a bedside qSOFA screening engine (`SepsisEngineService`) that evaluates three organ-dysfunction criteria per patient in near real-time. The second tier is the SOFA organ-dysfunction scoring engine (`SofaScoringService`) that confirms sepsis and triggers the treatment bundle. State is maintained in Redis as rolling windows of recent observations per encounter.
|
||
|
||
> **Historical note:** The original PRD specified SIRS criteria (temperature, heart rate, respiratory rate, WBC). Phase 27 replaced SIRS with qSOFA/SOFA per Sepsis-3 consensus — SIRS criteria were too non-specific, triggering bundles for post-surgical inflammation, anxiety, and viral infections. The legacy `SEPSIS_WARNING` alert type is retained `[Obsolete]` for historical queries but can no longer be created.
|
||
|
||
**Tier 1 — qSOFA screening criteria:**
|
||
|
||
| Criterion | Observation Code | Trigger |
|
||
|---|---|---|
|
||
| Tachypnea | `RESP_RATE` | ≥ 22 breaths/min |
|
||
| Hypotension | `SYSTOLIC_BP` | ≤ 100 mmHg |
|
||
| Altered mentation | `GCS` / `AVPU` | GCS < 15 or AVPU ≥ 1 |
|
||
|
||
**Redis state per encounter (qSOFA):**
|
||
```
|
||
qsofa:{encounterId}:RESP_RATE → "1" (TTL: 30 minutes)
|
||
qsofa:{encounterId}:SYSTOLIC_BP → "1" (TTL: 30 minutes)
|
||
qsofa:{encounterId}:MENTATION → "1" (TTL: 30 minutes)
|
||
```
|
||
|
||
**Tier 1 detection logic per observation event:**
|
||
```
|
||
1. Evaluate the incoming observation against qSOFA criteria
|
||
2. If criterion met: SET qsofa:{encounterId}:{code} EX 1800
|
||
3. If criterion normalized: DEL qsofa:{encounterId}:{code}
|
||
4. MGET all three qSOFA keys for this encounter
|
||
5. Persist evaluation to qsofa_evaluations table
|
||
6. If count >= 2 and no open QSOFA_SCREEN alert exists for this encounter:
|
||
a. Write clinical_alert (QSOFA_SCREEN, WARNING) — recommends ordering SOFA labs
|
||
b. Write outbox event → Kafka alert.generated
|
||
```
|
||
|
||
**Tier 2 — SOFA organ-dysfunction scoring:**
|
||
|
||
`SofaScoringService` subscribes to `observation.recorded` and `gcs.scored` Kafka topics. It scores six organ systems (respiratory, coagulation, liver, cardiovascular, CNS, renal) from Redis lab cache with carry-forward semantics. When SOFA delta ≥ 2 from baseline, a `SOFA_SEPSIS` (CRITICAL) alert fires and triggers a four-element sepsis treatment bundle via `SepsisAlertHandler` → `SepsisBundleService`. Delta = 1 creates `SOFA_WARNING`.
|
||
|
||
**Sepsis bundle compliance (SEP-1):** On `SOFA_SEPSIS`, four treatment orders (blood cultures, serum lactate, broad-spectrum antibiotics, IV fluid resuscitation) are auto-created with a one-hour compliance deadline. As orders are resulted, bundle elements complete. `SepsisBundleMonitorService` marks overdue bundles `NON_COMPLIANT`.
|
||
|
||
**Why Redis here and not PostgreSQL:** The qSOFA evaluation runs on every observation event, potentially multiple times per minute per patient. Redis's O(1) key operations with TTL-based expiry are correct and fast. The 30-minute TTL is a clinical parameter — a respiratory rate that was abnormal 31 minutes ago stops contributing to the qSOFA count without any cleanup job.
|
||
|
||
**Idempotency:** Alert creation uses `INSERT WHERE NOT EXISTS` — a duplicate is impossible even with at-least-once Kafka delivery. Only one in-progress sepsis bundle can exist per encounter, enforced by a partial unique index.
|
||
|
||
**Concepts practiced:** Stateful stream processing with Redis as the state store (sd-senior-011), TTL as a sliding window mechanism, two-tier clinical detection (screen → confirm → treat), idempotent alert creation, why Kafka consumer + Redis is appropriate here vs a dedicated stream processor like Flink (at the scale of a single hospital, the overhead of a full stream processing framework is not justified).
|
||
|
||
---
|
||
|
||
### 8. RabbitMQ Notification Workers and Escalation
|
||
|
||
**Description:** The notification worker reads `alert.generated` events from Kafka and dispatches paging jobs to RabbitMQ. The RabbitMQ consumer sends the page and waits for acknowledgment. If no acknowledgment arrives within five minutes, the dead-letter queue escalates to the on-call backup. If the API host is stopping while a page is in flight, the cancellation path requeues the message rather than escalating it.
|
||
|
||
**Exchange topology:**
|
||
```
|
||
clinical.notifications.exchange (direct)
|
||
├── alerts.paging.queue (physician paging, prefetch=3)
|
||
├── alerts.paging.dlq (unacknowledged pages → escalation, x-message-ttl=300000ms)
|
||
├── alerts.escalation.queue (on-call backup paging)
|
||
├── notifications.discharge.queue (discharge summary PDF → MinIO)
|
||
├── notifications.reconciliation.queue (reconciliation safety findings)
|
||
└── notifications.appointment.queue (appointment reminder SMS)
|
||
```
|
||
|
||
**Escalation flow:**
|
||
```
|
||
1. alert.generated event arrives from Kafka
|
||
2. Notification worker publishes to alerts.paging.queue
|
||
3. Paging worker sends page to attending physician
|
||
4. If no POST /alerts/:id/acknowledge within 5 minutes:
|
||
a. NACK with requeue=false → message goes to alerts.paging.dlq
|
||
b. DLQ has x-message-ttl = 300000ms (5 min)
|
||
c. After TTL: message routes back to alerts.escalation.queue
|
||
d. Escalation worker pages the on-call backup
|
||
e. clinical_alert.status → 'escalated' in PostgreSQL
|
||
5. If host shutdown occurs during paging wait:
|
||
a. Cancellation is treated as graceful stop, not failure
|
||
b. NACK with requeue=true → message returns to alerts.paging.queue
|
||
c. No DLQ route, so no false escalation during restart/deploy
|
||
```
|
||
|
||
**Discharge summary job:** When an encounter status changes to `discharged`, the outbox relay publishes to Kafka `encounter.status.changed`. The notification Kafka consumer reads this and publishes to `notifications.discharge.queue`. The worker generates a PDF summary (log the content; no real PDF library required), stores it in MinIO under `/discharge-summaries/{encounterId}/summary.pdf`, and marks the job complete.
|
||
|
||
**Concepts practiced:** RabbitMQ exchange-to-queue binding topology, DLQ TTL as a delayed retry and escalation mechanism, prefetch count and worker concurrency, why this escalation pattern is not replicable in Kafka (Kafka has no concept of per-message TTL or conditional re-routing based on consumer acknowledgment).
|
||
|
||
---
|
||
|
||
### 9. Reconciliation Jobs
|
||
|
||
**Description:** Three scheduled checks that verify the system is behaving correctly. Unlike the Digital Wallet where reconciliation checks that money did not disappear, clinical reconciliation checks that actions were taken — alerts were acknowledged, orders were resulted, encounters were closed.
|
||
|
||
**Check 1 — Unacknowledged critical alerts:**
|
||
```sql
|
||
SELECT id, encounter_id, triggered_at
|
||
FROM clinical_alerts
|
||
WHERE severity = 'CRITICAL'
|
||
AND status = 'open'
|
||
AND triggered_at < NOW() - INTERVAL '30 minutes';
|
||
```
|
||
Any row here means a critical alert sat open for 30 minutes without acknowledgment or escalation — this is a patient safety failure.
|
||
|
||
**Check 2 — Pending orders without results:**
|
||
```sql
|
||
SELECT id, encounter_id, order_type, ordered_at
|
||
FROM orders
|
||
WHERE status IN ('pending', 'in_progress')
|
||
AND ordered_at < NOW() - INTERVAL '4 hours';
|
||
```
|
||
A lab order that has been pending for four hours without a result may indicate a lost sample or a system integration failure.
|
||
|
||
**Check 3 — Active encounters without recent observations:**
|
||
```sql
|
||
SELECT e.id, e.patient_id, MAX(o.recorded_at) AS last_observation
|
||
FROM encounters e
|
||
LEFT JOIN observations o ON o.encounter_id = e.id
|
||
WHERE e.status = 'active'
|
||
AND e.encounter_type = 'INPATIENT'
|
||
GROUP BY e.id, e.patient_id
|
||
HAVING MAX(o.recorded_at) < NOW() - INTERVAL '2 hours'
|
||
OR MAX(o.recorded_at) IS NULL;
|
||
```
|
||
An active inpatient without any observation in two hours may indicate a disconnected monitor or a patient who was physically moved without a system update.
|
||
|
||
Each check creates a `reconciliation_alerts` row and publishes a job to RabbitMQ for operator notification.
|
||
|
||
**Concepts practiced:** Reconciliation as a patient safety mechanism (not just a data integrity mechanism), the difference between "did the data record correctly" (Digital Wallet) and "did the required action happen" (VigilCare), scheduled background jobs in .NET.
|
||
|
||
---
|
||
|
||
### 10. Observability
|
||
|
||
**Metrics (Prometheus → Grafana):**
|
||
|
||
| Metric | Type | Labels | Description |
|
||
|---|---|---|---|
|
||
| `observations_ingested_total` | Counter | `observation_code`, `source` | Per committed observation |
|
||
| `observation_ingest_duration_seconds` | Histogram | — | Full ingest transaction to COMMIT |
|
||
| `clinical_alerts_total` | Counter | `alert_type`, `severity` | All alert sources (threshold, qSOFA, NEWS2, GCS, SOFA, trend, warning) |
|
||
| `alerts_unacknowledged_gauge` | Gauge | — | Open CRITICAL alerts older than 5 minutes |
|
||
| `kafka_consumer_lag` | Gauge | `consumer_group` | Per consumer group lag |
|
||
| `outbox_pending_events` | Gauge | — | Unprocessed outbox rows |
|
||
| `qsofa_detections_total` | Counter | — | Successful qSOFA SCREEN alert inserts |
|
||
| `sepsis_bundle_compliance_total` | Counter | `status` | Bundle completion (`COMPLIANT`, `NON_COMPLIANT`) |
|
||
| `news2_scores_total` | Counter | `risk_level` | Per persisted NEWS2 score |
|
||
| `news2_scoring_duration_seconds` | Histogram | — | Redis update through score persistence |
|
||
| `gcs_scores_total` | Counter | `classification` | Per persisted GCS score (`MILD`, `MODERATE`, `SEVERE`) |
|
||
| `sofa_scores_total` | Counter | `has_delta_alert` | Per persisted SOFA score |
|
||
| `sofa_scoring_duration_seconds` | Histogram | — | Full SOFA compose + persist |
|
||
| `trend_alerts_total` | Counter | `observation_code` | Per `RAPID_DETERIORATION` alert |
|
||
| `trend_analysis_duration_seconds` | Histogram | — | Per-observation trend evaluation |
|
||
| `alert_suppressions_total` | Counter | `alert_type` | Per suppression window set |
|
||
| `escalations_total` | Counter | — | DLQ escalation pages |
|
||
| `fhir_ingest_total` | Counter | `resource_type`, `outcome` | Per FHIR resource ingest |
|
||
| `fhir_read_total` | Counter | `resource_type`, `interaction`, `outcome` | Per FHIR read/search |
|
||
| `fhir_mapping_errors_total` | Counter | `resource_type` | FHIR mapping/validation failures |
|
||
| `authorization_failures_total` | Counter | `permission`, `role` | RBAC authorization denials |
|
||
| `ward_gateways_offline_gauge` | Gauge | `site_code` | Offline/degraded gateways per site |
|
||
| `ward_gateway_buffer_depth` | Gauge | `gateway_code`, `department` | Unsynced events per gateway |
|
||
| `kafka_poison_pills_skipped_total` | Counter | `consumer_group`, `topic` | Permanently un-processable messages |
|
||
|
||
**The `alerts_unacknowledged_gauge` panel** is the most clinically significant metric. If this gauge rises, a nurse station monitor or alerting dashboard must surface it immediately. In a real deployment, this panel would be connected to a paging system. In the portfolio, it demonstrates that you understand which metrics have patient safety implications vs which are purely operational.
|
||
|
||
**Background collectors:** `AlertsUnacknowledgedCollector` (open CRITICAL alerts > 5 min), `OutboxPendingCollector` (unprocessed outbox rows), `KafkaConsumerLagCollector` (four consumer groups), `WardGatewayMetricsCollector` (gateway status/buffer depth) — all poll every 30–60 seconds.
|
||
|
||
**Concepts practiced:** The four golden signals in a clinical context, which metrics are operational (Kafka lag, outbox pending) vs which are patient safety indicators (unacknowledged critical alerts), log enrichment with `correlationId`, `encounterId`, `patientId` on every alert path log line.
|
||
|
||
---
|
||
|
||
### 11. Data Lake Writer
|
||
|
||
**Description:** A Kafka consumer that reads all three topics and writes partitioned Parquet files to MinIO. In healthcare, long-term retention is not optional — medical records must be retained for 7–25 years depending on jurisdiction. The data lake is the tier that satisfies this requirement without keeping the operational PostgreSQL database at 10-year scale.
|
||
|
||
**File structure:**
|
||
```
|
||
/observations/2025/01/15/partition-0-offset-0000001.parquet
|
||
/alerts/2025/01/15/partition-0-offset-0000001.parquet
|
||
/encounters/2025/01/15/partition-0-offset-0000001.parquet
|
||
```
|
||
|
||
**Flush policy:** Buffer 1,000 events or 5 minutes, whichever comes first.
|
||
|
||
**Why Parquet:** Columnar storage compresses repetitive observation data (many rows with the same `observation_code` and `unit`) at ratios of 5–10× vs JSON. A population health query — "give me all heart rate values for patients in the ICU in 2024" — reads only the `observation_code` and `value` columns without deserializing the rest of each row. This matters for 10 years of data at a multi-hospital scale.
|
||
|
||
**Concepts practiced:** Data lake as a separate retention tier from the operational database, Parquet's columnar advantage over row-based formats for analytics workloads, partition structure as the basis for future query tools (Spark, Athena, DuckDB), regulatory retention as an architectural driver.
|
||
|
||
---
|
||
|
||
### 12. Clinical Data Expansion and Warning Alerts (Phases 10–11)
|
||
|
||
**Description:** Expands the clinical data model and alert pipeline. Patient entities gain optional clinical fields (blood type, allergies, emergency contact). Encounters gain room/bed assignment, admission reason, and discharge diagnosis. Five new observation codes (`SYSTOLIC_BP`, `DIASTOLIC_BP`, `LACTATE_MMOL_L`, `AVPU`, `SUPPLEMENTAL_O2`) and `GLUCOSE_MG_DL` join the original six for 12 total seeded thresholds.
|
||
|
||
The `WarningAlertService` (consumer group `warning-evaluator`) reads `observation.recorded` from Kafka and creates `WARNING`-severity alerts for values that breach warning thresholds but not critical thresholds. Warning alerts are idempotent (`INSERT WHERE NOT EXISTS`) per encounter and alert type while status is `OPEN` or `ACKNOWLEDGED`. Warning alerts are indexed in Elasticsearch but not published to the RabbitMQ paging queue.
|
||
|
||
An Orders API (`OrdersController`, `OrderService`) supports clinical order management with a status machine (`Pending → InProgress → Resulted`, terminal `Cancelled`). FluentValidation is applied to all request DTOs.
|
||
|
||
---
|
||
|
||
### 13. NEWS2 Composite Scoring Engine (Phase 12)
|
||
|
||
**Description:** The NEWS2 (National Early Warning Score 2) engine evaluates seven vital parameters per encounter using Redis keys with a 4-hour TTL. When all seven are present, computes the official NEWS2 aggregate score, persists to `news2_scores`, and creates `NEWS2_WARNING` (score 5–6 or single param = 3) or `NEWS2_EMERGENCY` (score ≥ 7) alerts idempotently.
|
||
|
||
**Parameters:** `RESP_RATE`, `SPO2`, `SYSTOLIC_BP`, `HEART_RATE`, `AVPU`, `TEMP_C`, `SUPPLEMENTAL_O2` — each scored 0–3 per official lookup tables. Consciousness resolves GCS-first with AVPU fallback. `GET /encounters/:id/news2/current` and `/history` expose score history.
|
||
|
||
---
|
||
|
||
### 14. Trend Detection and Alert Suppression (Phase 13)
|
||
|
||
**Description:** `TrendAnalyzerService` (consumer group `trend-analyzer`) tracks rate-of-change for five vital parameters using Redis sliding-window history. When velocity exceeds configured thresholds (e.g. 72→95 bpm in 30 min), creates a `RAPID_DETERIORATION` alert even if the current value is below warning thresholds.
|
||
|
||
Alert suppression windows prevent warning fatigue. Acknowledging a suppressible alert sets a Redis key `suppress:{encounterId}:{alertType}` with a configurable TTL (default 30 min). `WarningEvaluator` and `News2Detector` check suppression before creating new warning alerts. Critical alerts are never suppressed.
|
||
|
||
---
|
||
|
||
### 15. Medication Administration and Correlation (Phase 15)
|
||
|
||
**Description:** Records drug administrations per encounter. `MedicationCorrelationHelper` annotates warning and NEWS2 alert details when a correlated drug was administered within a configurable window (default 90 min). Annotations provide clinical context — e.g. `— note: metoprolol 25mg (PO) administered 45 min ago` — but never suppress alerts.
|
||
|
||
---
|
||
|
||
### 16. Glasgow Coma Scale and SOFA Scoring (Phases 25–26)
|
||
|
||
**Description:** GCS scores three components (Eye 1–4, Verbal 1–5, Motor 1–6) tracked in Redis. When all three are present, computes total (3–15), classification (`MILD`/`MODERATE`/`SEVERE`), creates alerts (≤ 8 `GCS_CRITICAL`, 9–12 `GCS_WARNING`), and publishes `gcs.scored` via outbox for SOFA CNS re-scoring.
|
||
|
||
SOFA scoring evaluates six organ systems (respiratory, coagulation, liver, cardiovascular, CNS, renal) with Redis lab cache, carry-forward semantics, MAP derivation, SpO₂/FiO₂ fallback, and vasopressor detection. Baseline established when ≥ 4 organ systems have data. Delta ≥ 2 from baseline triggers `SOFA_SEPSIS` → sepsis bundle.
|
||
|
||
---
|
||
|
||
### 17. FHIR R4 Integration (Phase 30)
|
||
|
||
**Description:** Inbound FHIR R4 facade accepts resources from integration engines (Mirth Connect, Rhapsody). Per-resource endpoints (`POST /fhir/R4/{Patient,Encounter,Observation,MedicationAdministration}`) and transaction Bundles (Patient → Encounter → Observation in dependency order). 19 LOINC codes + 3 SNOMED CT fallbacks mapped to internal observation codes. Fahrenheit-to-Celsius conversion. `ExternalResourceIdentifier` table links hospital MRNs and visit numbers to internal UUIDs for idempotent upserts. Read/search endpoints (`GET /fhir/R4/Patient/{id}`, `GET /fhir/R4/Patient`, `GET /fhir/R4/Encounter/{id}`, `GET /fhir/R4/Encounter`) return FHIR R4 JSON.
|
||
|
||
---
|
||
|
||
### 18. RBAC and Clinical Audit Logging (Phase 31)
|
||
|
||
**Description:** JWT bearer authentication with four clinical roles (`Nurse`, `Physician`, `Admin`, `Integration`) and 17 granular permissions. `AuthorizePermission` attribute on every controller action. `PermissionAuthorizationHandler` resolves role → permission at runtime. `CurrentUserService` extracts authenticated identity from JWT claims.
|
||
|
||
Append-only `clinical_audit_logs` table records write actions with user identity, entity type/ID, before/after state (JSONB), reason, IP address, and correlation ID. Ten audit actions tracked. FHIR endpoints accept both JWT and `X-Api-Key` authentication for integration engine compatibility.
|
||
|
||
---
|
||
|
||
### 19. Site & Gateway Registry and Ward Gateway (Phases 20–21)
|
||
|
||
**Description:** `ClinicalSite` and `WardGateway` entities model ward edge nodes. Dual authentication — JWT + RBAC for admin CRUD, `GatewayApiKeyAuthenticationHandler` for gateway heartbeat and sync. `VigilCare.ClinicalContracts` shared class library defines sync DTOs.
|
||
|
||
`VigilCare.WardGateway` is a standalone ASP.NET Core deployable with its own PostgreSQL, Redis, and RabbitMQ. Observations are ingested locally with threshold evaluation and critical alert creation, then buffered for upload to central API when the network link recovers. Background services replicate encounter/patient data, report heartbeat status, and batch-upload buffered sync items.
|
||
|
||
---
|
||
|
||
### 20. Ward Dashboard (Phases 17–19, 22, 28)
|
||
|
||
**Description:** Vue 3 SPA (`vigilcare-dashboard/`) with Vite, Pinia, Tailwind CSS v4, and Chart.js. Virtual ward table (NEWS2-sorted, department filter), patient detail view (vitals, scores, alerts, orders, sepsis bundle, GCS entry form, SOFA score panel, patient banner with demographics, encounter timeline), and alert center (global acknowledge/resolve).
|
||
|
||
Chart components: five vital sign trend charts with medication administration markers, NEWS2 history, SOFA history with organ-system breakdown, GCS history with component tracking, qSOFA evaluation history. Local replay scrubbing, alert reasoning with medication context, and clinician feedback mode (six ratings per alert, Feedback Summary with JSON/CSV export).
|
||
|
||
Phase 22 gap analysis fixes: `SofaHistory.vue`, `GcsHistory.vue`, `QsofaHistory.vue`, `PatientBanner.vue`, `EncounterTimeline.vue`, medication marker Chart.js plugin. Backend additions: `GET /gcs/history`, `GET /qsofa/history` APIs, `qsofa_evaluations` table.
|
||
|
||
---
|
||
|
||
### 21. Console Replay Simulator (Phase 16, 29)
|
||
|
||
**Description:** Standalone `VigilCare.Simulator` .NET console app replays JSON scenario files against the live API with configurable speed. Commands: `replay`, `replay-all`, `validate`, `dry-run`. Optional `--poll` shows alerts, NEWS2, GCS, SOFA, and sepsis bundle state during replay. Eleven sample scenarios including GCS neurological decline, SOFA sepsis progression, and SpO₂/FiO₂ fallback.
|
||
|
||
---
|
||
|
||
## Database Schema and Indexing Plan
|
||
|
||
```sql
|
||
CREATE TABLE patients (
|
||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||
mrn VARCHAR(20) NOT NULL UNIQUE,
|
||
first_name VARCHAR(100) NOT NULL,
|
||
last_name VARCHAR(100) NOT NULL,
|
||
date_of_birth DATE NOT NULL,
|
||
gender VARCHAR(10) NOT NULL,
|
||
blood_type VARCHAR(5) NULL, -- Phase 10: A+, O-, AB-, etc.
|
||
allergies TEXT NULL, -- Phase 10: free-text allergy list
|
||
emergency_contact_name VARCHAR(200) NULL, -- Phase 10
|
||
emergency_contact_phone VARCHAR(30) NULL, -- Phase 10
|
||
status VARCHAR(20) NOT NULL DEFAULT 'active',
|
||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||
);
|
||
|
||
CREATE TABLE encounters (
|
||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||
patient_id UUID NOT NULL REFERENCES patients(id),
|
||
encounter_type VARCHAR(20) NOT NULL,
|
||
status VARCHAR(20) NOT NULL DEFAULT 'scheduled',
|
||
department VARCHAR(100) NOT NULL,
|
||
attending_physician VARCHAR(200) NOT NULL,
|
||
room_bed VARCHAR(50) NULL, -- Phase 10: ward/bed assignment
|
||
admission_reason TEXT NULL, -- Phase 10
|
||
discharge_diagnosis TEXT NULL, -- Phase 10: set on discharge
|
||
admitted_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||
discharged_at TIMESTAMPTZ NULL,
|
||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||
);
|
||
|
||
CREATE INDEX idx_encounters_patient ON encounters (patient_id, admitted_at DESC);
|
||
CREATE INDEX idx_encounters_active ON encounters (status, admitted_at DESC) WHERE status = 'active';
|
||
CREATE UNIQUE INDEX ix_encounters_patient_active_type
|
||
ON encounters (patient_id, encounter_type) WHERE status = 'ACTIVE';
|
||
|
||
CREATE TABLE alert_thresholds (
|
||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||
observation_code VARCHAR(50) NOT NULL UNIQUE,
|
||
display_name VARCHAR(200) NOT NULL,
|
||
unit VARCHAR(20) NOT NULL,
|
||
critical_low DECIMAL(10, 3) NULL,
|
||
warning_low DECIMAL(10, 3) NULL,
|
||
warning_high DECIMAL(10, 3) NULL,
|
||
critical_high DECIMAL(10, 3) NULL,
|
||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||
);
|
||
|
||
CREATE TABLE observations (
|
||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||
encounter_id UUID NOT NULL REFERENCES encounters(id),
|
||
observation_code VARCHAR(50) NOT NULL,
|
||
value DECIMAL(10, 3) NOT NULL,
|
||
unit VARCHAR(20) NOT NULL,
|
||
source VARCHAR(20) NOT NULL DEFAULT 'MANUAL',
|
||
idempotency_key VARCHAR(100) NULL,
|
||
recorded_at TIMESTAMPTZ NOT NULL,
|
||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||
);
|
||
|
||
CREATE UNIQUE INDEX idx_observations_idempotency
|
||
ON observations (idempotency_key) WHERE idempotency_key IS NOT NULL;
|
||
|
||
CREATE INDEX idx_observations_encounter_time
|
||
ON observations (encounter_id, observation_code, recorded_at DESC);
|
||
|
||
CREATE TABLE clinical_alerts (
|
||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||
encounter_id UUID NOT NULL REFERENCES encounters(id),
|
||
patient_id UUID NOT NULL REFERENCES patients(id),
|
||
observation_id UUID NULL REFERENCES observations(id),
|
||
alert_type VARCHAR(50) NOT NULL,
|
||
severity VARCHAR(20) NOT NULL,
|
||
details TEXT NOT NULL,
|
||
observation_code VARCHAR(50) NULL, -- enables direct lookups without LIKE pattern matching
|
||
status VARCHAR(20) NOT NULL DEFAULT 'open',
|
||
acknowledged_at TIMESTAMPTZ NULL,
|
||
acknowledged_by VARCHAR(200) NULL,
|
||
resolved_at TIMESTAMPTZ NULL,
|
||
triggered_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||
);
|
||
|
||
CREATE INDEX idx_alerts_encounter ON clinical_alerts (encounter_id, triggered_at DESC);
|
||
CREATE INDEX idx_alerts_patient ON clinical_alerts (patient_id, triggered_at DESC);
|
||
CREATE INDEX idx_alerts_open ON clinical_alerts (severity, triggered_at DESC)
|
||
WHERE status = 'open';
|
||
CREATE INDEX idx_alerts_enc_code ON clinical_alerts (encounter_id, observation_code, status);
|
||
|
||
CREATE TABLE orders (
|
||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||
encounter_id UUID NOT NULL REFERENCES encounters(id),
|
||
order_type VARCHAR(20) NOT NULL,
|
||
description VARCHAR(500) NOT NULL,
|
||
ordered_by VARCHAR(200) NOT NULL,
|
||
status VARCHAR(20) NOT NULL DEFAULT 'pending',
|
||
ordered_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||
resulted_at TIMESTAMPTZ NULL,
|
||
result_summary TEXT NULL -- free-text result summary
|
||
);
|
||
|
||
CREATE INDEX idx_orders_encounter ON orders (encounter_id, ordered_at DESC);
|
||
CREATE INDEX idx_orders_pending ON orders (status, ordered_at)
|
||
WHERE status IN ('pending', 'in_progress');
|
||
|
||
CREATE TABLE outbox_events (
|
||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||
topic VARCHAR(200) NOT NULL,
|
||
partition_key VARCHAR(100) NULL, -- encounterId for per-encounter ordering
|
||
payload JSONB NOT NULL,
|
||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||
processed_at TIMESTAMPTZ NULL,
|
||
retry_count INT NOT NULL DEFAULT 0, -- Kafka produce attempt counter
|
||
last_error TEXT NULL, -- last failure reason
|
||
failed_at TIMESTAMPTZ NULL -- set when retryCount exceeds OutboxMaxRetries
|
||
);
|
||
|
||
CREATE INDEX idx_outbox_pending ON outbox_events (created_at)
|
||
WHERE processed_at IS NULL AND failed_at IS NULL;
|
||
|
||
CREATE TABLE reconciliation_alerts (
|
||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||
check_type VARCHAR(64) NOT NULL,
|
||
encounter_id UUID NULL REFERENCES encounters(id),
|
||
patient_id UUID NULL REFERENCES patients(id),
|
||
details TEXT NOT NULL,
|
||
resolved_at TIMESTAMPTZ NULL,
|
||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||
);
|
||
```
|
||
|
||
**Tables added in Phases 10–31** (managed by EF Core migrations — see `README.md` Data Models for full column definitions):
|
||
|
||
| Table | Phase | Purpose |
|
||
|---|---|---|
|
||
| `news2_scores` | 12 | NEWS2 composite scores with seven component scores and risk level |
|
||
| `gcs_scores` | 25 | GCS eye/verbal/motor components, total score, classification |
|
||
| `sofa_scores` | 26 | Six organ-system scores, baseline flag, delta from baseline, staleness metadata |
|
||
| `qsofa_evaluations` | 22 | Per-evaluation qSOFA record: criteria count, values, screen alert fired |
|
||
| `sepsis_bundles` | 14 | Four-element treatment bundles with 1-hour compliance deadline |
|
||
| `sepsis_bundle_elements` | 14 | Individual bundle elements linked to clinical orders |
|
||
| `medication_administrations` | 15 | Drug administration records per encounter |
|
||
| `external_resource_identifiers` | 30 | Links hospital MRNs and visit numbers to internal UUIDs for FHIR |
|
||
| `clinical_users` | 31 | Username, BCrypt password hash, display name, role |
|
||
| `clinical_audit_logs` | 31 | Append-only audit trail: action, entity, user, before/after JSONB, IP, correlation ID |
|
||
| `clinical_sites` | 20 | Hospital sites with site code, name, address |
|
||
| `ward_gateways` | 20 | Ward edge nodes with status, buffer depth, heartbeat, sync timestamps |
|
||
|
||
---
|
||
|
||
## Design Decisions
|
||
|
||
### Synchronous vs Asynchronous Alert Detection — The Split
|
||
|
||
The ingest endpoint evaluates critical thresholds synchronously and warning thresholds asynchronously via the Kafka consumer. This is a deliberate patient safety decision.
|
||
|
||
A critical potassium of 2.1 mEq/L (normal: 3.5–5.0) is immediately life-threatening. If the API returns `201 Created` before generating the alert, and the Kafka consumer is lagging by 30 seconds, a patient could deteriorate during that window. The synchronous check costs one additional Redis read per observation on the critical path — acceptable for correctness.
|
||
|
||
A warning heart rate of 95 bpm (warning threshold: 90) warrants attention but is not an emergency. The additional latency of Kafka consumer processing (milliseconds to seconds) is clinically acceptable for a warning.
|
||
|
||
This is the architectural decision that separates thinking about healthcare systems from thinking about financial systems. In fintech, milliseconds of latency matter for user experience. In healthcare, the right tradeoff is latency for correctness — and the correctness definition is clinical, not technical.
|
||
|
||
### Observation Codes as Strings (Not an Enum)
|
||
|
||
Observation codes are stored as VARCHAR rather than a database enum. This allows new device types and lab panels to be registered by inserting a threshold row without a schema migration. The trade-off is that typos in observation codes produce silent mismatches (an observation with code `HEART_RATE` and a threshold for `HEARTRATE` would never trigger an alert). The application layer validates incoming codes against the `alert_thresholds` table on ingest.
|
||
|
||
In production this would use LOINC codes — an international standard for lab and clinical observations. Knowing that LOINC exists and why it exists (interoperability between systems, not just a naming convention) is a senior talking point.
|
||
|
||
### Why Not a Time-Series Database for Observations?
|
||
|
||
A medium hospital with 200 concurrent inpatients generating five observations per patient per minute produces approximately 17 observations per second at steady state, peaking near 50/second during shift changes. PostgreSQL with the composite index `(encounter_id, observation_code, recorded_at DESC)` handles this volume with headroom on any modern server.
|
||
|
||
A time-series database (InfluxDB, TimescaleDB) would be warranted at sustained 10,000+ observations/second — a large hospital network, not a single facility. TimescaleDB specifically is worth mentioning: it is PostgreSQL with automatic time-based partitioning, meaning it shares the operational model of this project and could be swapped in without changing the query layer. The decision to use vanilla PostgreSQL is correct at this scale and defensible at interview.
|
||
|
||
The Parquet data lake handles the analytics workload that would otherwise stress the operational database over a 10-year horizon.
|
||
|
||
### Encounter as the Aggregate Root (Not Patient)
|
||
|
||
Observations, alerts, and orders belong to an encounter, not directly to a patient. This mirrors clinical reality — a patient's blood pressure taken during a 2022 inpatient admission belongs to that admission, not floating freely on the patient record. It also bounds queries naturally: "show me all observations for this encounter" is a bounded query; "show me all observations ever recorded for this patient" is an expensive cross-encounter aggregation that belongs in the data lake, not the operational path.
|
||
|
||
---
|
||
|
||
## Non-Functional Requirements
|
||
|
||
| Concern | Target |
|
||
|---|---|
|
||
| Critical alert latency | A CRITICAL threshold breach must generate an alert before the ingest response returns |
|
||
| Observation idempotency | Duplicate device submissions with the same key must not create duplicate observations or alerts |
|
||
| Alert acknowledgment | All open CRITICAL alerts must be detectable via the reconciliation job within 30 minutes |
|
||
| Replay | Deleting and rebuilding the Elasticsearch index from Kafka offset 0 must be demonstrable |
|
||
| Retention | Data lake writer must write observations to MinIO; nothing is deleted from the lake |
|
||
| Testing | Integration tests: critical value ingest → alert created; qSOFA criteria met → screening alert; SOFA delta ≥ 2 → sepsis bundle; duplicate idempotency key → no duplicate; encounter discharge → RabbitMQ job published |
|
||
|
||
---
|
||
|
||
## Build Order
|
||
|
||
| Phase | Focus | Status |
|
||
|---|---|---|
|
||
| 1 | Schema, migrations, patient/encounter CRUD, alert threshold CRUD, seed data | Done |
|
||
| 2 | Observation ingest + synchronous critical value detection + alert lifecycle API | Done |
|
||
| 3 | Outbox relay + Kafka topics + producer | Done |
|
||
| 4 | Elasticsearch CQRS projection + clinical search + analytics endpoints | Done |
|
||
| 5 | Sepsis detection engine (Kafka consumer + Redis qSOFA state) | Done |
|
||
| 6 | RabbitMQ notification workers + DLQ escalation | Done |
|
||
| 7 | Reconciliation jobs (three checks) | Done |
|
||
| 8 | Prometheus metrics + Grafana dashboards + Seq logging | Done |
|
||
| 9 | MinIO data lake writer (Parquet, partitioned) | Done |
|
||
| 10 | Clinical data model expansion — patient demographics, encounter enrichment, 12 observation codes | Done |
|
||
| 11 | Warning alert consumer (`warning-evaluator`) + Orders API + FluentValidation | Done |
|
||
| 12 | NEWS2 composite scoring engine (seven vitals → aggregate score → alerts) | Done |
|
||
| 13 | Trend detection (rate-of-change alerts) + alert suppression windows | Done |
|
||
| 14 | qSOFA bedside screening + sepsis bundle compliance (SEP-1) | Done |
|
||
| 15 | Medication administration + drug-vital correlation annotations on alerts | Done |
|
||
| 16 | Console replay simulator (scenario JSON files, speed multiplier, API polling) | Done |
|
||
| 17 | Ward dashboard shell — Vue 3 + Vite + Pinia + Tailwind; virtual ward table, patient detail, alert center | Done |
|
||
| 18 | Clinical review mode — vital trend charts, NEWS2 history, replay controls, alert reasoning | Done |
|
||
| 19 | Clinician feedback mode — six ratings per alert, Feedback Summary with export | Done |
|
||
| 20 | Site & Gateway Registry + Clinical Sync Contracts (shared class library) | Done |
|
||
| 21 | Ward Gateway Service — local-first clinical path with offline buffering and central sync | Done |
|
||
| 22 | Dashboard gap analysis fixes — SOFA/GCS/qSOFA history charts, patient banner, encounter timeline, medication markers | Done |
|
||
| 25 | Glasgow Coma Scale (GCS) scoring — three components → total → alerts → SOFA CNS | Done |
|
||
| 26 | SOFA organ-dysfunction scoring — six organ systems, baseline tracking, delta sepsis alerts | Done |
|
||
| 27 | Sepsis-3 clinical refactor — SIRS removed, qSOFA screening, SOFA bundle trigger | Done |
|
||
| 28 | Frontend GCS entry form + SOFA score panel + sepsis UI refactor | Done |
|
||
| 29 | Simulator scenario expansion + clinical validation (end-to-end qSOFA → SOFA → bundle) | Done |
|
||
| 30 | FHIR R4 Inbound Facade — per-resource ingest, transaction Bundles, read/search, LOINC mapping | Done |
|
||
| 31 | RBAC + Clinical Audit Logging — JWT auth, four roles, 17 permissions, append-only audit trail | Done |
|
||
|
||
---
|
||
|
||
## Step-by-Step Guide
|
||
|
||
Complete phases in order. The synchronous alert path in Phase 2 must be correct before Kafka is introduced in Phase 3 — mixing the two failure modes early makes debugging very difficult.
|
||
|
||
---
|
||
|
||
### Phase 1 — Schema, Migrations, and Core CRUD
|
||
|
||
**What to do:**
|
||
1. Model all tables in EF Core with migrations matching this PRD.
|
||
2. Enforce the encounter status machine at the service layer — build an explicit transition matrix and return `409` with a stable error code on illegal transitions.
|
||
3. Seed: two patients, one active inpatient encounter each, four alert thresholds (heart rate, temperature, potassium, SpO₂), a set of observations covering normal, warning, and critical ranges.
|
||
4. Implement patient search supporting both MRN (exact match) and name (partial match via `ILIKE`). Explain in a comment why MRN uses an exact-match index and name uses a prefix scan.
|
||
5. Pre-load all alert thresholds into Redis on application startup using `IHostedService`. Verify that a threshold update via the API invalidates the cache.
|
||
|
||
**Why:**
|
||
The threshold cache design is worth getting right in Phase 1. Every observation ingest will read from it. Understanding that it is a write-through invalidation (not a TTL expiry) is the correct design for data where staleness has clinical consequences.
|
||
|
||
---
|
||
|
||
### Phase 2 — Observation Ingest and Synchronous Alert Detection
|
||
|
||
**What to do:**
|
||
1. Implement `POST /encounters/:id/observations` following the transaction sequence in the Features section exactly.
|
||
2. Load the alert threshold from Redis (not PostgreSQL) inside the ingest transaction. Measure the latency difference with `EXPLAIN ANALYZE` on the PostgreSQL path for comparison.
|
||
3. On `CRITICAL` breach: insert the `clinical_alerts` row and outbox event within the same transaction. Do not return `201` until the alert is written.
|
||
4. On `WARNING` breach: insert the outbox event only — alert creation is deferred to the Kafka consumer.
|
||
5. Implement cursor-paginated observation history. Verify the composite index `(encounter_id, observation_code, recorded_at DESC)` is used.
|
||
6. Write integration tests: normal observation (no alert), critical breach (alert created in same transaction), duplicate idempotency key (no duplicate), discharged encounter (reject ingest with `409`).
|
||
|
||
**Why:**
|
||
The split between synchronous (critical) and asynchronous (warning) detection is the most clinically significant decision in the codebase. Test both paths and articulate why they are different. A reviewer or interviewer who asks "why not do all alerts asynchronously?" should get a clinical safety answer, not a technical one.
|
||
|
||
---
|
||
|
||
### Phase 3 — Outbox Relay and Kafka
|
||
|
||
**What to do:**
|
||
1. Implement the outbox relay as an `IHostedService` polling every 500ms.
|
||
2. Create Kafka topics: `observation.recorded`, `alert.generated`, `encounter.status.changed`.
|
||
3. Partition all topics by `encounterId` to guarantee per-encounter ordering.
|
||
4. Verify the relay survives a Kafka restart: observations commit to PostgreSQL while Kafka is down; the relay catches up when Kafka recovers.
|
||
5. Introduce the outbox bug deliberately: make two separate commits (one for the observation, one for the outbox event) and observe the data loss when the process crashes between them. Fix it. This step is not optional — seeing the failure mode is the fastest path to internalizing the pattern.
|
||
|
||
**Why:**
|
||
The per-encounter partition key is important for the sepsis engine. If observations from the same patient land on different partitions, they may be processed out of order, and qSOFA criteria that arrived simultaneously could be missed. Document this in the code.
|
||
|
||
---
|
||
|
||
### Phase 4 — Elasticsearch CQRS Projection
|
||
|
||
**What to do:**
|
||
1. Build the `es-indexer` consumer group. Upsert `patient_encounters` documents on `encounter.status.changed`; append to `observations` index on `observation.recorded`; update `openAlertCount` on `alert.generated`.
|
||
2. Implement the analytics endpoints using Elasticsearch aggregations. The `population` query is a numeric range filter aggregation — no full-text search at all. Write this query first to make explicit that Elasticsearch is being used here for its aggregation engine, not its search engine.
|
||
3. Write the replay procedure to the README: stop the consumer → delete both indices → reset consumer group offset to 0 → restart → wait for rebuild → verify document count matches PostgreSQL row count.
|
||
4. Run the replay. Verify it completes and the counts match.
|
||
|
||
**Why:**
|
||
The replay is the proof that Elasticsearch is a projection and not a source of truth. It is also the clearest demonstration of why Kafka's event retention matters. Practice running it until it takes less than two minutes to explain what is happening and why it is significant.
|
||
|
||
---
|
||
|
||
### Phase 5 — Sepsis Detection Engine
|
||
|
||
> **Updated for Sepsis-3 (Phase 27 refactor):** The original Phase 5 implemented SIRS-based detection. Phase 27 replaced SIRS with the two-tier qSOFA → SOFA pathway per the 2016 Sepsis-3 consensus. The steps below reflect the current implementation.
|
||
|
||
**What to do:**
|
||
1. Build the `sepsis-engine` consumer group reading `observation.recorded`.
|
||
2. Implement the Redis qSOFA state: `SET qsofa:{encounterId}:{code} EX 1800` on criterion met, `DEL` on criterion normalized.
|
||
3. Use `MGET` on all three qSOFA keys per encounter after each observation — three O(1) operations, not a scan.
|
||
4. Persist every evaluation to `qsofa_evaluations` table with criteria values and screen-alert-fired flag.
|
||
5. On qSOFA count >= 2: check for an existing open `QSOFA_SCREEN` alert for this encounter before inserting. The check and insert are one round-trip: `INSERT INTO clinical_alerts ... WHERE NOT EXISTS (SELECT 1 FROM clinical_alerts WHERE encounter_id = ? AND alert_type = 'QSOFA_SCREEN' AND status IN ('open', 'acknowledged'))`.
|
||
6. SOFA scoring (`sofa-scoring` consumer) scores six organ systems and triggers `SOFA_SEPSIS` on delta ≥ 2 from baseline → sepsis bundle creation.
|
||
7. Write integration tests: ingest observations meeting two qSOFA criteria → verify `QSOFA_SCREEN` alert created. Verify normalization deletes Redis key. Verify SOFA delta ≥ 2 triggers sepsis bundle.
|
||
|
||
**Why:**
|
||
The 30-minute TTL matches the clinical window for qSOFA evaluation — a respiratory rate that was abnormal 31 minutes ago stops contributing without any cleanup job. SIRS was removed because it was too non-specific (triggering on post-surgical inflammation, anxiety, viral infections). qSOFA measures organ dysfunction at the bedside; SOFA confirms it with lab values. This two-tier approach prevents false-positive bundle activations.
|
||
|
||
---
|
||
|
||
### Phase 6 — RabbitMQ Notifications and Escalation
|
||
|
||
**What to do:**
|
||
1. Create the exchange and queues from the topology in the Features section. Set `x-dead-letter-exchange` on `alerts.paging.queue` pointing to `alerts.paging.dlq`. Set `x-message-ttl = 300000` on `alerts.paging.dlq`.
|
||
2. Build a Kafka consumer (`notification-publisher` group) reading `alert.generated`. For `CRITICAL` severity alerts: publish a paging job to `alerts.paging.queue`.
|
||
3. Build the paging worker: log the page (no real pager required), wait for an `acknowledged` webhook or a timeout, then NACK on timeout.
|
||
4. Build the escalation worker on `alerts.escalation.queue`: log the escalation, update `clinical_alerts.status = 'escalated'` in PostgreSQL.
|
||
5. Test the full escalation path: create a critical alert → verify page is published → do not acknowledge → wait for TTL → verify escalation fires → verify alert status is `escalated` in the database.
|
||
|
||
**Why:**
|
||
The escalation test requires actually waiting 5 minutes (or temporarily setting TTL to 5 seconds in the test environment). Run it. Watching the message appear in the DLQ, wait, and then re-appear in the escalation queue is the moment the DLQ pattern becomes intuitive. It is also the answer to "how would you build escalation in a paging system?" in an interview — a Kafka-native answer does not exist for this pattern.
|
||
|
||
---
|
||
|
||
### Phase 7 — Reconciliation Jobs
|
||
|
||
**What to do:**
|
||
1. Implement the three reconciliation queries as scheduled `IHostedService` jobs (every 30 minutes in development).
|
||
2. For each check: if rows are found, insert `reconciliation_alerts` and publish a job to RabbitMQ.
|
||
3. Test Check 1 by creating a critical alert and not acknowledging it for 31 minutes (advance the `triggered_at` timestamp in the database directly to simulate time passing).
|
||
4. Test Check 3 by creating an active inpatient encounter and not posting any observations — verify the job detects it.
|
||
|
||
**Why:**
|
||
Check 3 is the one unique to clinical systems. A financial reconciliation job checks that data is correct. This check verifies that the real-world process (a nurse checking vitals) actually happened and was recorded. The system cannot verify that the nurse physically took the measurement — only that a reading was posted. Understanding this limitation is part of the senior conversation.
|
||
|
||
---
|
||
|
||
### Phases 8 and 9 — Observability and Data Lake
|
||
|
||
Follow the same Prometheus/Grafana and MinIO/Parquet approach as described in the Features section. The `alerts_unacknowledged_gauge` panel is the single most important panel in the Grafana dashboard — build it first and make sure it updates in near real-time (poll the database every 30 seconds).
|
||
|
||
---
|
||
|
||
### Phases 10–31 — Extended Feature Phases
|
||
|
||
See the [Build Order](#build-order) table for all implemented phases and the [Features](#features) section (items 12–21) for detailed descriptions. Per-phase implementation plans and verification guides are in `docs/plans/`. Integration tests and verification scripts cover all phases — see `README.md` for the full test table and script listing.
|