# PRD: VigilCare — Clinical Data Pipeline & Real-Time Alert Platform ## Overview A production-style clinical backend that models patient encounters, continuous observation ingest, and real-time clinical alerting. The system streams vital signs and lab results through Kafka, fans urgent notifications to clinicians through RabbitMQ, and maintains a searchable CQRS projection in Elasticsearch for patient dashboards and population analytics. Long-term data is archived as Parquet files in an S3-compatible object store — a regulatory requirement in healthcare that has no equivalent in most other domains. The domain is deliberately different from the Digital Wallet API. Both projects use Kafka, RabbitMQ, and Elasticsearch, but the trade-off conversations are entirely different. In fintech the core question is "did the money move correctly?" In healthcare the core question is "did the right person get the right alert at the right time?" That distinction — correctness vs timeliness — produces different architectural decisions at every layer. This project maps to `sd-mid-009` (Outbox Pattern), `sd-mid-013` (CQRS), `sd-mid-043–048` (Kafka internals), `sd-senior-008` (Real-Time Event Processing), and `sd-senior-011` (Anomaly Detection in Streams). **Stack:** .NET 8 Web API, PostgreSQL, Apache Kafka (KRaft), RabbitMQ, Elasticsearch, Redis, MinIO (Parquet archival), Serilog → Seq, Prometheus → Grafana, xUnit, Docker Compose. --- ## Local Development Setup Start all infrastructure services with: ```bash docker compose up -d ``` | Service | Port | Notes | |---|---|---| | PostgreSQL | 5436 | Database: `vigilcare`, user: `postgres`, password: `password` | | Redis | 6382 | No auth | | Seq | 5345 | UI at `http://localhost:5345` — login: `admin` / `admin` | | Kafka | 9092 | KRaft mode, no Zookeeper | **Seq first-run:** `SEQ_FIRSTRUN_ADMINPASSWORD=admin` is set in `docker-compose.yml`. This password is only applied on the very first container start (when `/data` volume is empty). After initialization, the password is stored in the volume and this env var is ignored. --- ## Goals - Model the observe-alert-acknowledge lifecycle that sits at the center of any clinical monitoring system - Demonstrate Kafka's multi-consumer log model in a healthcare context where the same observation event must reach the alert engine, the Elasticsearch projection, and the data lake independently - Show RabbitMQ's DLQ pattern as a clinical escalation mechanism — if a critical alert is not acknowledged in five minutes, the message routes through a dead-letter queue and re-delivers as an escalation to the on-call physician - Build a stateful Kafka consumer that detects sepsis early warning signs by maintaining rolling windows of recent observations per patient in Redis - Produce a project that supports senior trade-off conversations in healthcare, medtech, and any domain where real-time alerting and long-term archival coexist ## Non-Goals - HL7 FHIR compliance (reference the standard; do not implement it) - Integration with real medical devices or lab information systems - Medication dispensing or pharmacy workflows - Patient billing or insurance claim adjudication - HIPAA-compliant deployment (model the patterns; don't configure real PHI) --- ## Why Both Kafka and RabbitMQ? This is the same architectural question as the Digital Wallet — but the healthcare context produces a different answer that is worth knowing independently. **Kafka** is an append-only log. Every observation recorded by a bedside monitor, every lab result that arrives from the lab information system, is written to a Kafka topic and retained. Multiple independent consumer groups read the same observation stream at their own pace: - The Elasticsearch indexer maintains a searchable patient dashboard - The sepsis detection engine analyzes rolling windows for SIRS criteria - The data lake writer archives observations as Parquet for long-term regulatory retention - A future billing consumer could derive charges from observation codes without touching the operational database None of these consumers coordinate with each other. Each holds its own offset. If the sepsis engine is deployed a month after go-live, it can replay all historical observations from offset 0 to catch up. This is only possible because Kafka retains events after consumption. **RabbitMQ** handles the action side — what must happen after a clinical event is recognized. When the alert engine detects a critical potassium value, a clinician must be paged. That page is a task: one message, one worker, one action. It must not be processed twice (a duplicate page at 3am is a patient safety concern, not a minor inconvenience). RabbitMQ's acknowledgment model — the message is deleted after exactly one worker acknowledges it — is correct here. Kafka's model is not. The escalation pattern makes RabbitMQ's dead-letter queue uniquely valuable in this domain. If a physician does not acknowledge a critical alert within five minutes, the original message NACKs into a dead-letter queue with a `x-message-ttl` of 300 seconds. After that TTL expires, the message is re-routed to an escalation queue targeting the on-call backup. This is the DLQ pattern repurposed as a clinical escalation protocol — a design that does not exist cleanly in Kafka. | Use Case | System | Why | |---|---|---| | Vital sign streams from monitors | Kafka | Continuous, high-frequency, multiple consumers | | Lab result events from LIS | Kafka | Replayable; alert engine and data lake both need it | | Encounter admission/discharge events | Kafka | Multiple downstream systems react independently | | Sepsis detection analytics | Kafka → Redis | Stateful windowed analysis over the observation stream | | Page a physician for a critical value | RabbitMQ | One task, one worker, acknowledged-then-deleted | | Escalate if unacknowledged after 5 minutes | RabbitMQ DLQ | Delayed re-delivery is native to DLQ TTL; Kafka has no equivalent | | Generate discharge summary PDF | RabbitMQ | Background job; one per discharge, not replayable | | Appointment reminder SMS | RabbitMQ | Task queue; idempotent at the SMS provider level | --- ## API Conventions Same response envelope as all other portfolio projects. Prefix: `/api/v1`. **Success:** ```json { "success": true, "statusCode": 200, "data": {}, "error": null } ``` **Error:** ```json { "success": false, "statusCode": 422, "data": null, "error": { "message": "Observation value exceeds plausible range for this code.", "code": "OBSERVATION_OUT_OF_PLAUSIBLE_RANGE" } } ``` **Pagination:** Observation history and alert history use cursor pagination on `(recorded_at DESC, id DESC)` — the table is append-only and grows continuously; offset pagination shifts results as new rows arrive. All other list endpoints use offset pagination (`?page=1&pageSize=20`). **Idempotency:** `POST /api/v1/encounters/:id/observations` accepts an `Idempotency-Key` header enforced by a unique partial index. Medical device integrations frequently retry on network failure; a duplicate reading must produce the same response without creating a duplicate observation. --- ## Domain Model A **patient** is the central entity. Each patient has a Medical Record Number (MRN) — a stable identifier issued at first registration that never changes, even across multiple encounters. An **encounter** is a single clinical episode — an inpatient admission, an outpatient visit, or an emergency department visit. A patient may have many encounters over their lifetime. An encounter has a status (`scheduled`, `active`, `discharged`, `cancelled`) and a department. All observations, orders, and alerts belong to an encounter, not directly to a patient. An **alert threshold** defines the numeric boundaries that trigger a clinical alert for a given observation code. Thresholds are global (not per-patient) and are managed by clinical administrators. Each threshold has four optional bounds: `critical_low`, `warning_low`, `warning_high`, `critical_high`. A potassium value below `critical_low` is an immediate life-threatening emergency; a value below `warning_low` warrants physician review within the hour. An **observation** is a single recorded measurement: a vital sign (heart rate, temperature, blood pressure), a lab value (potassium, glucose, white blood cell count), or a pulse oximetry reading. Observations are append-only. They are never updated or deleted. The observation stream is the primary input to both the alert engine and the Kafka pipeline. A **clinical alert** is generated when an observation breaches a threshold or when the sepsis detection engine identifies a pattern across multiple recent observations. An alert has a lifecycle: `open` → `acknowledged` → `resolved`. An unacknowledged `CRITICAL` alert triggers the RabbitMQ escalation after five minutes. An **order** is a clinician's instruction: run this lab test, administer this medication, perform this imaging study. Orders have a status lifecycle and a `resulted_at` timestamp. The reconciliation job uses pending orders to detect cases where a result was never returned. An **outbox event** is written in the same transaction as any observation or alert, then relayed to Kafka by a background worker. This decouples Kafka availability from the database transaction. --- ## Features --- ### 1. Patient Registration and Encounter Management **Description:** Patients are registered with demographic information and assigned a unique MRN. When a patient presents for care, an encounter is opened against their record. Encounters progress through a controlled status machine. A patient cannot have two active encounters of the same type simultaneously. **Endpoints:** - `POST /api/v1/patients` — register patient; generates MRN - `GET /api/v1/patients` — paginated list with search by name or MRN - `GET /api/v1/patients/:id` — patient detail with active encounter summary - `POST /api/v1/patients/:id/encounters` — open an encounter - `GET /api/v1/encounters/:id` — encounter detail with recent observations and open alerts - `PATCH /api/v1/encounters/:id/status` — advance status (`active → discharged`, `scheduled → active`, etc.); illegal transitions return `409` - `GET /api/v1/encounters/:id/timeline` — merged chronological view: status changes, observation summaries, alerts **Encounter status machine:** ``` scheduled → active → discharged → cancelled ``` A `discharged` encounter triggers a RabbitMQ job to generate a discharge summary PDF. **Concepts practiced:** Aggregate design (encounter owns observations and alerts), controlled state transitions with explicit transition matrix, 409 on illegal transitions, timeline as a composed projection across multiple tables. --- ### 2. Alert Threshold Management **Description:** Clinical administrators configure the numeric boundaries that define normal, warning, and critical ranges for each observation code. Thresholds are cached in Redis at application startup and invalidated on write — they are read on every observation ingest and must not add database latency to the ingest path. **Endpoints:** - `POST /api/v1/alert-thresholds` - `GET /api/v1/alert-thresholds` - `GET /api/v1/alert-thresholds/:id` - `PUT /api/v1/alert-thresholds/:id` **Data:** ```json { "id": "uuid", "observationCode": "POTASSIUM_MEQ_L", "displayName": "Serum Potassium", "unit": "mEq/L", "criticalLow": 2.5, "warningLow": 3.5, "warningHigh": 5.0, "criticalHigh": 6.5 } ``` **Concepts practiced:** Redis as a configuration cache (not just session/balance cache), cache-aside with write-through invalidation, the difference between data that changes per-request (patient observations) and data that changes per-configuration (thresholds). --- ### 3. Observation Ingest **Description:** The highest-volume endpoint in the system. Bedside monitors, point-of-care devices, and lab integration systems POST observations continuously. The endpoint must be idempotent (devices retry on network failure), must validate the value against a plausibility range (no human has a heart rate of 400), and must evaluate the observation against alert thresholds on the synchronous path for critical values. **Endpoints:** - `POST /api/v1/encounters/:id/observations` — single or small batch (up to 10); document choice - `GET /api/v1/encounters/:id/observations?code=&from=&to=&limit=&cursor=` — cursor-paginated observation history with optional code filter **Ingest transaction sequence:** ``` 1. Validate encounter is active (not discharged or cancelled) 2. Check idempotency key against unique index 3. Validate observation value within plausible range for the code 4. Insert observation row 5. Load alert threshold for this code from Redis cache (→ PostgreSQL on miss) 6. If value breaches CRITICAL threshold: a. Insert clinical_alert row (status: open) b. Insert outbox event (topic: alert.generated) 7. Insert outbox event (topic: observation.recorded, payload: full observation) 8. COMMIT ``` **Critical vs warning split:** Critical threshold breaches are detected synchronously within the ingest transaction and immediately create an alert. Warning threshold breaches are detected by the Kafka consumer asynchronously — the additional latency (milliseconds to seconds) is acceptable for a warning, but a critical potassium value must trigger a page before the API returns a response. **Idempotency:** A device that retries an observation with the same `Idempotency-Key` receives the original `201` response without creating a duplicate row. The unique partial index enforces this at the database layer. **Concepts practiced:** Idempotency key on high-frequency ingest (sd-mid-008), synchronous vs asynchronous alert detection (the split is a clinical safety decision, not an arbitrary one), Redis cache-aside for threshold lookup on the hot path, outbox pattern within ingest transaction. --- ### 4. Clinical Alert Lifecycle **Description:** Alerts are the patient safety core of the system. Every open `CRITICAL` alert must be acknowledged by a clinician within five minutes or it escalates. Every alert has a documented audit trail: who acknowledged it, when, and with what note. **Endpoints:** - `GET /api/v1/encounters/:id/alerts` — paginated alert list for an encounter - `GET /api/v1/alerts` — global alert list filterable by status, severity, department - `GET /api/v1/alerts/:id` — alert detail - `POST /api/v1/alerts/:id/acknowledge` — acknowledge with clinician ID and optional note - `POST /api/v1/alerts/:id/resolve` — resolve (must be acknowledged first) **Alert lifecycle:** ``` open → acknowledged → resolved → escalated (via RabbitMQ DLQ after 5 min unacknowledged) ``` **The escalation path:** When an alert is created, the outbox relay publishes to `alert.generated` in Kafka. The notification worker Kafka consumer reads this event and publishes a paging job to RabbitMQ. If the RabbitMQ consumer sends the page but receives no acknowledgment event within five minutes, the message NACKs to the DLQ. After the DLQ TTL expires (300 seconds), the message re-routes to the escalation queue and the on-call backup is paged. The alert status transitions to `escalated` in PostgreSQL. **Concepts practiced:** Alert acknowledgment as a domain event (not just a status update), escalation via DLQ TTL as a healthcare-specific pattern, the difference between an alert being acknowledged in the app vs a clinician physically responding at the bedside. --- ### 5. Outbox Relay and Kafka Pipeline **Description:** Same pattern as the Digital Wallet. The relay reads unprocessed outbox rows, publishes to Kafka, marks processed. Every observation and every alert flows through this relay to reach Elasticsearch, the sepsis engine, and the data lake independently. **Kafka topics:** | Topic | Producer | Consumers | |---|---|---| | `observation.recorded` | Outbox relay | Elasticsearch indexer, Sepsis engine, Data lake writer | | `alert.generated` | Outbox relay | Elasticsearch indexer, Notification worker, Data lake writer | | `encounter.status.changed` | Outbox relay | Elasticsearch indexer, Data lake writer | **Partition key:** `encounter_id` for `observation.recorded` and `alert.generated`. All events for the same encounter land on the same partition, preserving per-encounter ordering. This is important for the sepsis engine: observations for the same patient must be processed in arrival order. **Consumer group isolation:** `es-indexer`, `sepsis-engine`, and `data-lake-writer` are separate consumer groups. Each maintains its own committed offset. The sepsis engine processing slowly does not affect the Elasticsearch indexer. **Concepts practiced:** Partition key design for per-entity ordering guarantees, consumer group independence, at-least-once delivery via outbox relay (and why consumers must be idempotent), Kafka as the backbone that allows adding new consumers without modifying the producer. --- ### 6. Elasticsearch Clinical Search and Analytics (CQRS) **Description:** The Elasticsearch indexer maintains a denormalized, queryable projection of the clinical record. It is the read side of CQRS — PostgreSQL is always the write side and the source of truth. The index is optimized for the queries clinicians actually run: "show me all patients with a critical potassium alert in the last hour," "show me the average heart rate trend for this patient over the last 24 hours." **Index shape — patient_encounters:** ```json { "encounterId": "uuid", "patientId": "uuid", "mrn": "MRN-000001", "patientName": "Jane Smith", "department": "ICU", "status": "active", "attendingPhysician": "Dr. Osei", "admittedAt": "2025-01-01T08:00:00Z", "openAlertCount": 2, "lastObservationAt": "2025-01-01T09:45:00Z" } ``` **Index shape — observations:** ```json { "observationId": "uuid", "encounterId": "uuid", "patientId": "uuid", "mrn": "MRN-000001", "observationCode": "HEART_RATE", "value": 118.0, "unit": "bpm", "source": "DEVICE", "recordedAt": "2025-01-01T09:45:00Z" } ``` **Endpoints:** - `GET /api/v1/analytics/patients?q=&department=&status=` — patient/encounter search across MRN, name, department - `GET /api/v1/analytics/observations/trend?encounterId=&code=&from=&to=` — time-series aggregation (hourly average, min, max) for a specific observation code - `GET /api/v1/analytics/alerts/summary?severity=&from=&to=&department=` — alert volume by department and severity over a time window - `GET /api/v1/analytics/population?code=&threshold=&from=&to=` — how many patients had a value above or below a threshold in a given window **The replay demo:** Stop the indexer → delete the Elasticsearch index → reset the `es-indexer` consumer group offset to 0 → restart → watch both indices rebuild from Kafka history. This is only possible because Kafka retains events. Document this procedure in the project README. It is the most important operational proof-of-concept in the project. **Why Elasticsearch here and not PostgreSQL:** The `population` query — "how many active patients have a heart rate above 100 in the last hour across all departments" — is an aggregation across potentially millions of observation rows. Running this against PostgreSQL on the operational database would compete with ingest writes and introduce latency for both. Elasticsearch's aggregation engine is purpose-built for this pattern. PostgreSQL remains untouched for this query. **Concepts practiced:** CQRS read projection design (sd-mid-013), Elasticsearch aggregations as a distinct use case from full-text search (the `population` endpoint uses no full-text search at all — it is a numeric range aggregation), eventual consistency between PostgreSQL and Elasticsearch, replay as a recovery mechanism. --- ### 7. Sepsis Early Warning Engine **Description:** A Kafka consumer that reads the `observation.recorded` stream and detects SIRS (Systemic Inflammatory Response Syndrome) criteria per patient in near real-time. SIRS is a simplified clinical proxy for sepsis risk — when two or more criteria are met simultaneously, a `SEPSIS_WARNING` alert is generated. State is maintained in Redis as a rolling window of recent observations per encounter. **SIRS criteria (simplified for this project):** | Criterion | Observation Code | Trigger | |---|---|---| | Fever or hypothermia | `TEMP_C` | > 38.3°C or < 36.0°C | | Tachycardia | `HEART_RATE` | > 90 bpm | | Tachypnea | `RESP_RATE` | > 20 breaths/min | | Abnormal WBC | `WBC_K_UL` | > 12.0 or < 4.0 k/µL | **Redis state per encounter:** ``` sirs:{encounterId}:TEMP_C → "1" (TTL: 30 minutes) sirs:{encounterId}:HEART_RATE → "1" (TTL: 30 minutes) sirs:{encounterId}:RESP_RATE → "1" (TTL: 30 minutes) sirs:{encounterId}:WBC_K_UL → "1" (TTL: 30 minutes) ``` **Detection logic per observation event:** ``` 1. Evaluate the incoming observation against SIRS criteria 2. If criterion met: SET sirs:{encounterId}:{code} = "1" EX 1800 3. If criterion not met: DEL sirs:{encounterId}:{code} 4. Count active SIRS keys for this encounter (KEYS pattern or MGET) 5. If count >= 2 and no open SEPSIS_WARNING alert exists for this encounter: a. Write clinical_alert to PostgreSQL (SEPSIS_WARNING, CRITICAL) b. Write outbox event → Kafka alert.generated ``` **Why Redis here and not PostgreSQL:** The SIRS evaluation runs on every observation event, potentially multiple times per minute per patient. Checking "which SIRS criteria were met in the last 30 minutes" against PostgreSQL on every event would require a query against the observations table with a time range filter per encounter — under load, this creates read pressure that competes with ingest writes. Redis's O(1) key operations with TTL-based expiry are correct and fast. The TTL handles the sliding window automatically: a heart rate measurement that was abnormal 31 minutes ago stops contributing to the SIRS count without any cleanup job. **Idempotency:** If the consumer crashes between detecting SIRS and committing the Kafka offset, it will re-process the same observation on restart. The alert creation query checks for an existing open `SEPSIS_WARNING` alert before inserting — a duplicate is impossible even with at-least-once delivery. **Concepts practiced:** Stateful stream processing with Redis as the state store (sd-senior-011), TTL as a sliding window mechanism, idempotent alert creation, why Kafka consumer + Redis is appropriate here vs a dedicated stream processor like Flink (at the scale of a single hospital, the overhead of a full stream processing framework is not justified — this is a defensible trade-off to articulate in an interview). --- ### 8. RabbitMQ Notification Workers and Escalation **Description:** The notification worker reads `alert.generated` events from Kafka and dispatches paging jobs to RabbitMQ. The RabbitMQ consumer sends the page and waits for acknowledgment. If no acknowledgment arrives within five minutes, the dead-letter queue escalates to the on-call backup. If the API host is stopping while a page is in flight, the cancellation path requeues the message rather than escalating it. **Exchange topology:** ``` clinical.notifications.exchange (direct) ├── alerts.paging.queue (physician paging, prefetch=3) ├── alerts.paging.dlq (unacknowledged pages → escalation) ├── alerts.escalation.queue (on-call backup paging) ├── notifications.discharge.queue (discharge summary PDF jobs) └── notifications.appointment.queue (appointment reminders) ``` **Escalation flow:** ``` 1. alert.generated event arrives from Kafka 2. Notification worker publishes to alerts.paging.queue 3. Paging worker sends page to attending physician 4. If no POST /alerts/:id/acknowledge within 5 minutes: a. NACK with requeue=false → message goes to alerts.paging.dlq b. DLQ has x-message-ttl = 300000ms (5 min) c. After TTL: message routes back to alerts.escalation.queue d. Escalation worker pages the on-call backup e. clinical_alert.status → 'escalated' in PostgreSQL 5. If host shutdown occurs during paging wait: a. Cancellation is treated as graceful stop, not failure b. NACK with requeue=true → message returns to alerts.paging.queue c. No DLQ route, so no false escalation during restart/deploy ``` **Discharge summary job:** When an encounter status changes to `discharged`, the outbox relay publishes to Kafka `encounter.status.changed`. The notification Kafka consumer reads this and publishes to `notifications.discharge.queue`. The worker generates a PDF summary (log the content; no real PDF library required), stores it in MinIO under `/discharge-summaries/{encounterId}/summary.pdf`, and marks the job complete. **Concepts practiced:** RabbitMQ exchange-to-queue binding topology, DLQ TTL as a delayed retry and escalation mechanism, prefetch count and worker concurrency, why this escalation pattern is not replicable in Kafka (Kafka has no concept of per-message TTL or conditional re-routing based on consumer acknowledgment). --- ### 9. Reconciliation Jobs **Description:** Three scheduled checks that verify the system is behaving correctly. Unlike the Digital Wallet where reconciliation checks that money did not disappear, clinical reconciliation checks that actions were taken — alerts were acknowledged, orders were resulted, encounters were closed. **Check 1 — Unacknowledged critical alerts:** ```sql SELECT id, encounter_id, triggered_at FROM clinical_alerts WHERE severity = 'CRITICAL' AND status = 'open' AND triggered_at < NOW() - INTERVAL '30 minutes'; ``` Any row here means a critical alert sat open for 30 minutes without acknowledgment or escalation — this is a patient safety failure. **Check 2 — Pending orders without results:** ```sql SELECT id, encounter_id, order_type, ordered_at FROM orders WHERE status IN ('pending', 'in_progress') AND ordered_at < NOW() - INTERVAL '4 hours'; ``` A lab order that has been pending for four hours without a result may indicate a lost sample or a system integration failure. **Check 3 — Active encounters without recent observations:** ```sql SELECT e.id, e.patient_id, MAX(o.recorded_at) AS last_observation FROM encounters e LEFT JOIN observations o ON o.encounter_id = e.id WHERE e.status = 'active' AND e.encounter_type = 'INPATIENT' GROUP BY e.id, e.patient_id HAVING MAX(o.recorded_at) < NOW() - INTERVAL '2 hours' OR MAX(o.recorded_at) IS NULL; ``` An active inpatient without any observation in two hours may indicate a disconnected monitor or a patient who was physically moved without a system update. Each check creates a `reconciliation_alerts` row and publishes a job to RabbitMQ for operator notification. **Concepts practiced:** Reconciliation as a patient safety mechanism (not just a data integrity mechanism), the difference between "did the data record correctly" (Digital Wallet) and "did the required action happen" (VigilCare), scheduled background jobs in .NET. --- ### 10. Observability **Metrics (Prometheus → Grafana):** | Metric | Description | |---|---| | `observations_ingested_total` | Counter, labeled by source and observation_code | | `observation_ingest_duration_seconds` | Histogram of ingest latency (includes threshold evaluation) | | `clinical_alerts_total` | Counter, labeled by alert_type and severity | | `alerts_unacknowledged_gauge` | Gauge — open CRITICAL alerts older than 5 minutes | | `kafka_consumer_lag` | Per consumer group (es-indexer, sepsis-engine, data-lake-writer) | | `outbox_pending_events` | Gauge — unprocessed outbox rows | | `sirs_detections_total` | Counter — how many SEPSIS_WARNING alerts the engine generated | | `escalations_total` | Counter — how many pages went through DLQ escalation | **The `alerts_unacknowledged_gauge` panel** is the most clinically significant metric. If this gauge rises, a nurse station monitor or alerting dashboard must surface it immediately. In a real deployment, this panel would be connected to a paging system. In the portfolio, it demonstrates that you understand which metrics have patient safety implications vs which are purely operational. **Concepts practiced:** The four golden signals in a clinical context, which metrics are operational (Kafka lag, outbox pending) vs which are patient safety indicators (unacknowledged critical alerts), log enrichment with `correlationId`, `encounterId`, `patientId` on every alert path log line. --- ### 11. Data Lake Writer **Description:** A Kafka consumer that reads all three topics and writes partitioned Parquet files to MinIO. In healthcare, long-term retention is not optional — medical records must be retained for 7–25 years depending on jurisdiction. The data lake is the tier that satisfies this requirement without keeping the operational PostgreSQL database at 10-year scale. **File structure:** ``` /observations/2025/01/15/partition-0-offset-0000001.parquet /alerts/2025/01/15/partition-0-offset-0000001.parquet /encounters/2025/01/15/partition-0-offset-0000001.parquet ``` **Flush policy:** Buffer 1,000 events or 5 minutes, whichever comes first. **Why Parquet:** Columnar storage compresses repetitive observation data (many rows with the same `observation_code` and `unit`) at ratios of 5–10× vs JSON. A population health query — "give me all heart rate values for patients in the ICU in 2024" — reads only the `observation_code` and `value` columns without deserializing the rest of each row. This matters for 10 years of data at a multi-hospital scale. **Concepts practiced:** Data lake as a separate retention tier from the operational database, Parquet's columnar advantage over row-based formats for analytics workloads, partition structure as the basis for future query tools (Spark, Athena, DuckDB), regulatory retention as an architectural driver. --- ## Database Schema and Indexing Plan ```sql CREATE TABLE patients ( id UUID PRIMARY KEY DEFAULT gen_random_uuid(), mrn VARCHAR(20) NOT NULL UNIQUE, first_name VARCHAR(100) NOT NULL, last_name VARCHAR(100) NOT NULL, date_of_birth DATE NOT NULL, gender VARCHAR(10) NOT NULL, status VARCHAR(20) NOT NULL DEFAULT 'active', created_at TIMESTAMPTZ NOT NULL DEFAULT NOW() ); CREATE TABLE encounters ( id UUID PRIMARY KEY DEFAULT gen_random_uuid(), patient_id UUID NOT NULL REFERENCES patients(id), encounter_type VARCHAR(20) NOT NULL, status VARCHAR(20) NOT NULL DEFAULT 'scheduled', department VARCHAR(100) NOT NULL, attending_physician VARCHAR(200) NOT NULL, admitted_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), discharged_at TIMESTAMPTZ NULL, created_at TIMESTAMPTZ NOT NULL DEFAULT NOW() ); CREATE INDEX idx_encounters_patient ON encounters (patient_id, admitted_at DESC); CREATE INDEX idx_encounters_active ON encounters (status, admitted_at DESC) WHERE status = 'active'; CREATE TABLE alert_thresholds ( id UUID PRIMARY KEY DEFAULT gen_random_uuid(), observation_code VARCHAR(50) NOT NULL UNIQUE, display_name VARCHAR(200) NOT NULL, unit VARCHAR(20) NOT NULL, critical_low DECIMAL(10, 3) NULL, warning_low DECIMAL(10, 3) NULL, warning_high DECIMAL(10, 3) NULL, critical_high DECIMAL(10, 3) NULL, created_at TIMESTAMPTZ NOT NULL DEFAULT NOW() ); CREATE TABLE observations ( id UUID PRIMARY KEY DEFAULT gen_random_uuid(), encounter_id UUID NOT NULL REFERENCES encounters(id), observation_code VARCHAR(50) NOT NULL, value DECIMAL(10, 3) NOT NULL, unit VARCHAR(20) NOT NULL, source VARCHAR(20) NOT NULL DEFAULT 'MANUAL', idempotency_key VARCHAR(100) NULL, recorded_at TIMESTAMPTZ NOT NULL, created_at TIMESTAMPTZ NOT NULL DEFAULT NOW() ); CREATE UNIQUE INDEX idx_observations_idempotency ON observations (idempotency_key) WHERE idempotency_key IS NOT NULL; CREATE INDEX idx_observations_encounter_time ON observations (encounter_id, observation_code, recorded_at DESC); CREATE TABLE clinical_alerts ( id UUID PRIMARY KEY DEFAULT gen_random_uuid(), encounter_id UUID NOT NULL REFERENCES encounters(id), patient_id UUID NOT NULL REFERENCES patients(id), observation_id UUID NULL REFERENCES observations(id), alert_type VARCHAR(50) NOT NULL, severity VARCHAR(20) NOT NULL, details TEXT NOT NULL, status VARCHAR(20) NOT NULL DEFAULT 'open', acknowledged_at TIMESTAMPTZ NULL, acknowledged_by VARCHAR(200) NULL, resolved_at TIMESTAMPTZ NULL, triggered_at TIMESTAMPTZ NOT NULL DEFAULT NOW() ); CREATE INDEX idx_alerts_encounter ON clinical_alerts (encounter_id, triggered_at DESC); CREATE INDEX idx_alerts_patient ON clinical_alerts (patient_id, triggered_at DESC); CREATE INDEX idx_alerts_open ON clinical_alerts (severity, triggered_at DESC) WHERE status = 'open'; CREATE TABLE orders ( id UUID PRIMARY KEY DEFAULT gen_random_uuid(), encounter_id UUID NOT NULL REFERENCES encounters(id), order_type VARCHAR(20) NOT NULL, description VARCHAR(500) NOT NULL, ordered_by VARCHAR(200) NOT NULL, status VARCHAR(20) NOT NULL DEFAULT 'pending', ordered_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), resulted_at TIMESTAMPTZ NULL ); CREATE INDEX idx_orders_encounter ON orders (encounter_id, ordered_at DESC); CREATE INDEX idx_orders_pending ON orders (status, ordered_at) WHERE status IN ('pending', 'in_progress'); CREATE TABLE outbox_events ( id UUID PRIMARY KEY DEFAULT gen_random_uuid(), topic VARCHAR(200) NOT NULL, payload JSONB NOT NULL, created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), processed_at TIMESTAMPTZ NULL ); CREATE INDEX idx_outbox_pending ON outbox_events (created_at) WHERE processed_at IS NULL; CREATE TABLE reconciliation_alerts ( id UUID PRIMARY KEY DEFAULT gen_random_uuid(), check_type VARCHAR(64) NOT NULL, encounter_id UUID NULL REFERENCES encounters(id), patient_id UUID NULL REFERENCES patients(id), details TEXT NOT NULL, resolved_at TIMESTAMPTZ NULL, created_at TIMESTAMPTZ NOT NULL DEFAULT NOW() ); ``` --- ## Design Decisions ### Synchronous vs Asynchronous Alert Detection — The Split The ingest endpoint evaluates critical thresholds synchronously and warning thresholds asynchronously via the Kafka consumer. This is a deliberate patient safety decision. A critical potassium of 2.1 mEq/L (normal: 3.5–5.0) is immediately life-threatening. If the API returns `201 Created` before generating the alert, and the Kafka consumer is lagging by 30 seconds, a patient could deteriorate during that window. The synchronous check costs one additional Redis read per observation on the critical path — acceptable for correctness. A warning heart rate of 95 bpm (warning threshold: 90) warrants attention but is not an emergency. The additional latency of Kafka consumer processing (milliseconds to seconds) is clinically acceptable for a warning. This is the architectural decision that separates thinking about healthcare systems from thinking about financial systems. In fintech, milliseconds of latency matter for user experience. In healthcare, the right tradeoff is latency for correctness — and the correctness definition is clinical, not technical. ### Observation Codes as Strings (Not an Enum) Observation codes are stored as VARCHAR rather than a database enum. This allows new device types and lab panels to be registered by inserting a threshold row without a schema migration. The trade-off is that typos in observation codes produce silent mismatches (an observation with code `HEART_RATE` and a threshold for `HEARTRATE` would never trigger an alert). The application layer validates incoming codes against the `alert_thresholds` table on ingest. In production this would use LOINC codes — an international standard for lab and clinical observations. Knowing that LOINC exists and why it exists (interoperability between systems, not just a naming convention) is a senior talking point. ### Why Not a Time-Series Database for Observations? A medium hospital with 200 concurrent inpatients generating five observations per patient per minute produces approximately 17 observations per second at steady state, peaking near 50/second during shift changes. PostgreSQL with the composite index `(encounter_id, observation_code, recorded_at DESC)` handles this volume with headroom on any modern server. A time-series database (InfluxDB, TimescaleDB) would be warranted at sustained 10,000+ observations/second — a large hospital network, not a single facility. TimescaleDB specifically is worth mentioning: it is PostgreSQL with automatic time-based partitioning, meaning it shares the operational model of this project and could be swapped in without changing the query layer. The decision to use vanilla PostgreSQL is correct at this scale and defensible at interview. The Parquet data lake handles the analytics workload that would otherwise stress the operational database over a 10-year horizon. ### Encounter as the Aggregate Root (Not Patient) Observations, alerts, and orders belong to an encounter, not directly to a patient. This mirrors clinical reality — a patient's blood pressure taken during a 2022 inpatient admission belongs to that admission, not floating freely on the patient record. It also bounds queries naturally: "show me all observations for this encounter" is a bounded query; "show me all observations ever recorded for this patient" is an expensive cross-encounter aggregation that belongs in the data lake, not the operational path. --- ## Non-Functional Requirements | Concern | Target | |---|---| | Critical alert latency | A CRITICAL threshold breach must generate an alert before the ingest response returns | | Observation idempotency | Duplicate device submissions with the same key must not create duplicate observations or alerts | | Alert acknowledgment | All open CRITICAL alerts must be detectable via the reconciliation job within 30 minutes | | Replay | Deleting and rebuilding the Elasticsearch index from Kafka offset 0 must be demonstrable | | Retention | Data lake writer must write observations to MinIO; nothing is deleted from the lake | | Testing | Integration tests: critical value ingest → alert created; SIRS criteria met across 3 observations → sepsis alert; duplicate idempotency key → no duplicate; encounter discharge → RabbitMQ job published | --- ## Build Order | Phase | Focus | |---|---| | 1 | Schema, migrations, patient/encounter CRUD, alert threshold CRUD, seed data | | 2 | Observation ingest + synchronous critical value detection + alert lifecycle API | | 3 | Outbox relay + Kafka topics + producer | | 4 | Elasticsearch CQRS projection + clinical search + analytics endpoints | | 5 | Sepsis detection engine (Kafka consumer + Redis SIRS state) | | 6 | RabbitMQ notification workers + DLQ escalation | | 7 | Reconciliation jobs (three checks) | | 8 | Prometheus metrics + Grafana dashboards + Seq logging | | 9 | MinIO data lake writer (Parquet, partitioned) | --- ## Step-by-Step Guide Complete phases in order. The synchronous alert path in Phase 2 must be correct before Kafka is introduced in Phase 3 — mixing the two failure modes early makes debugging very difficult. --- ### Phase 1 — Schema, Migrations, and Core CRUD **What to do:** 1. Model all tables in EF Core with migrations matching this PRD. 2. Enforce the encounter status machine at the service layer — build an explicit transition matrix and return `409` with a stable error code on illegal transitions. 3. Seed: two patients, one active inpatient encounter each, four alert thresholds (heart rate, temperature, potassium, SpO₂), a set of observations covering normal, warning, and critical ranges. 4. Implement patient search supporting both MRN (exact match) and name (partial match via `ILIKE`). Explain in a comment why MRN uses an exact-match index and name uses a prefix scan. 5. Pre-load all alert thresholds into Redis on application startup using `IHostedService`. Verify that a threshold update via the API invalidates the cache. **Why:** The threshold cache design is worth getting right in Phase 1. Every observation ingest will read from it. Understanding that it is a write-through invalidation (not a TTL expiry) is the correct design for data where staleness has clinical consequences. --- ### Phase 2 — Observation Ingest and Synchronous Alert Detection **What to do:** 1. Implement `POST /encounters/:id/observations` following the transaction sequence in the Features section exactly. 2. Load the alert threshold from Redis (not PostgreSQL) inside the ingest transaction. Measure the latency difference with `EXPLAIN ANALYZE` on the PostgreSQL path for comparison. 3. On `CRITICAL` breach: insert the `clinical_alerts` row and outbox event within the same transaction. Do not return `201` until the alert is written. 4. On `WARNING` breach: insert the outbox event only — alert creation is deferred to the Kafka consumer. 5. Implement cursor-paginated observation history. Verify the composite index `(encounter_id, observation_code, recorded_at DESC)` is used. 6. Write integration tests: normal observation (no alert), critical breach (alert created in same transaction), duplicate idempotency key (no duplicate), discharged encounter (reject ingest with `409`). **Why:** The split between synchronous (critical) and asynchronous (warning) detection is the most clinically significant decision in the codebase. Test both paths and articulate why they are different. A reviewer or interviewer who asks "why not do all alerts asynchronously?" should get a clinical safety answer, not a technical one. --- ### Phase 3 — Outbox Relay and Kafka **What to do:** 1. Implement the outbox relay as an `IHostedService` polling every 500ms. 2. Create Kafka topics: `observation.recorded`, `alert.generated`, `encounter.status.changed`. 3. Partition all topics by `encounterId` to guarantee per-encounter ordering. 4. Verify the relay survives a Kafka restart: observations commit to PostgreSQL while Kafka is down; the relay catches up when Kafka recovers. 5. Introduce the outbox bug deliberately: make two separate commits (one for the observation, one for the outbox event) and observe the data loss when the process crashes between them. Fix it. This step is not optional — seeing the failure mode is the fastest path to internalizing the pattern. **Why:** The per-encounter partition key is important for the sepsis engine. If observations from the same patient land on different partitions, they may be processed out of order, and SIRS criteria that arrived simultaneously could be missed. Document this in the code. --- ### Phase 4 — Elasticsearch CQRS Projection **What to do:** 1. Build the `es-indexer` consumer group. Upsert `patient_encounters` documents on `encounter.status.changed`; append to `observations` index on `observation.recorded`; update `openAlertCount` on `alert.generated`. 2. Implement the analytics endpoints using Elasticsearch aggregations. The `population` query is a numeric range filter aggregation — no full-text search at all. Write this query first to make explicit that Elasticsearch is being used here for its aggregation engine, not its search engine. 3. Write the replay procedure to the README: stop the consumer → delete both indices → reset consumer group offset to 0 → restart → wait for rebuild → verify document count matches PostgreSQL row count. 4. Run the replay. Verify it completes and the counts match. **Why:** The replay is the proof that Elasticsearch is a projection and not a source of truth. It is also the clearest demonstration of why Kafka's event retention matters. Practice running it until it takes less than two minutes to explain what is happening and why it is significant. --- ### Phase 5 — Sepsis Detection Engine **What to do:** 1. Build the `sepsis-engine` consumer group reading `observation.recorded`. 2. Implement the Redis SIRS state as described in the Features section: `SET sirs:{encounterId}:{code} EX 1800` on criterion met, `DEL` on criterion not met. 3. Use `MGET` on all four SIRS keys per encounter after each observation — four O(1) operations, not a scan. 4. On SIRS count >= 2: check for an existing open `SEPSIS_WARNING` alert for this encounter before inserting. The check and insert are one round-trip: `INSERT INTO clinical_alerts ... WHERE NOT EXISTS (SELECT 1 FROM clinical_alerts WHERE encounter_id = ? AND alert_type = 'SEPSIS_WARNING' AND status = 'open')`. 5. Write an integration test: ingest three observations that meet two SIRS criteria for the same encounter within 30 minutes → verify one `SEPSIS_WARNING` alert is created. Ingest a normal temperature immediately after → verify the TTL key is deleted but the alert remains open until acknowledged. **Why:** The TTL is doing real work here. Without it, a patient who had a fever yesterday would still have `sirs:{encounterId}:TEMP_C = "1"` in Redis today and could trigger a false sepsis alert from a fast heart rate alone. The 30-minute TTL matches the clinical window for SIRS evaluation. Understand this before the interview — the TTL is not an arbitrary expiry, it is a clinical parameter encoded in the data layer. --- ### Phase 6 — RabbitMQ Notifications and Escalation **What to do:** 1. Create the exchange and queues from the topology in the Features section. Set `x-dead-letter-exchange` on `alerts.paging.queue` pointing to `alerts.paging.dlq`. Set `x-message-ttl = 300000` on `alerts.paging.dlq`. 2. Build a Kafka consumer (`notification-publisher` group) reading `alert.generated`. For `CRITICAL` severity alerts: publish a paging job to `alerts.paging.queue`. 3. Build the paging worker: log the page (no real pager required), wait for an `acknowledged` webhook or a timeout, then NACK on timeout. 4. Build the escalation worker on `alerts.escalation.queue`: log the escalation, update `clinical_alerts.status = 'escalated'` in PostgreSQL. 5. Test the full escalation path: create a critical alert → verify page is published → do not acknowledge → wait for TTL → verify escalation fires → verify alert status is `escalated` in the database. **Why:** The escalation test requires actually waiting 5 minutes (or temporarily setting TTL to 5 seconds in the test environment). Run it. Watching the message appear in the DLQ, wait, and then re-appear in the escalation queue is the moment the DLQ pattern becomes intuitive. It is also the answer to "how would you build escalation in a paging system?" in an interview — a Kafka-native answer does not exist for this pattern. --- ### Phase 7 — Reconciliation Jobs **What to do:** 1. Implement the three reconciliation queries as scheduled `IHostedService` jobs (every 30 minutes in development). 2. For each check: if rows are found, insert `reconciliation_alerts` and publish a job to RabbitMQ. 3. Test Check 1 by creating a critical alert and not acknowledging it for 31 minutes (advance the `triggered_at` timestamp in the database directly to simulate time passing). 4. Test Check 3 by creating an active inpatient encounter and not posting any observations — verify the job detects it. **Why:** Check 3 is the one unique to clinical systems. A financial reconciliation job checks that data is correct. This check verifies that the real-world process (a nurse checking vitals) actually happened and was recorded. The system cannot verify that the nurse physically took the measurement — only that a reading was posted. Understanding this limitation is part of the senior conversation. --- ### Phases 8 and 9 — Observability and Data Lake Follow the same Prometheus/Grafana and MinIO/Parquet approach as described in the Features section. The `alerts_unacknowledged_gauge` panel is the single most important panel in the Grafana dashboard — build it first and make sure it updates in near real-time (poll the database every 30 seconds).