Trent/vigilcare-clinical

Fork 0

Files

T

voltsrage df99bf3c91 feature: Observability: Prometheus Metrics and Grafana

2026-06-17 16:25:27 +08:00

47 KiB

Raw Blame History

PRD: VigilCare — Clinical Data Pipeline & Real-Time Alert Platform

Overview

A production-style clinical backend that models patient encounters, continuous observation ingest, and real-time clinical alerting. The system streams vital signs and lab results through Kafka, fans urgent notifications to clinicians through RabbitMQ, and maintains a searchable CQRS projection in Elasticsearch for patient dashboards and population analytics. Long-term data is archived as Parquet files in an S3-compatible object store — a regulatory requirement in healthcare that has no equivalent in most other domains.

The domain is deliberately different from the Digital Wallet API. Both projects use Kafka, RabbitMQ, and Elasticsearch, but the trade-off conversations are entirely different. In fintech the core question is "did the money move correctly?" In healthcare the core question is "did the right person get the right alert at the right time?" That distinction — correctness vs timeliness — produces different architectural decisions at every layer.

This project maps to sd-mid-009 (Outbox Pattern), sd-mid-013 (CQRS), sd-mid-043–048 (Kafka internals), sd-senior-008 (Real-Time Event Processing), and sd-senior-011 (Anomaly Detection in Streams).

Stack: .NET 8 Web API, PostgreSQL, Apache Kafka (KRaft), RabbitMQ, Elasticsearch, Redis, MinIO (Parquet archival), Serilog → Seq, Prometheus → Grafana, xUnit, Docker Compose.

Local Development Setup

Start all infrastructure services with:

docker compose up -d

Service	Port	Notes
PostgreSQL	5436	Database: `vigilcare`, user: `postgres`, password: `password`
Redis	6382	No auth
Seq	5345	UI at `http://localhost:5345` — login: `admin` / `admin`
Kafka	9092	KRaft mode, no Zookeeper

Seq first-run: SEQ_FIRSTRUN_ADMINPASSWORD=admin is set in docker-compose.yml. This password is only applied on the very first container start (when /data volume is empty). After initialization, the password is stored in the volume and this env var is ignored.

Goals

Model the observe-alert-acknowledge lifecycle that sits at the center of any clinical monitoring system
Demonstrate Kafka's multi-consumer log model in a healthcare context where the same observation event must reach the alert engine, the Elasticsearch projection, and the data lake independently
Show RabbitMQ's DLQ pattern as a clinical escalation mechanism — if a critical alert is not acknowledged in five minutes, the message routes through a dead-letter queue and re-delivers as an escalation to the on-call physician
Build a stateful Kafka consumer that detects sepsis early warning signs by maintaining rolling windows of recent observations per patient in Redis
Produce a project that supports senior trade-off conversations in healthcare, medtech, and any domain where real-time alerting and long-term archival coexist

Non-Goals

HL7 FHIR compliance (reference the standard; do not implement it)
Integration with real medical devices or lab information systems
Medication dispensing or pharmacy workflows
Patient billing or insurance claim adjudication
HIPAA-compliant deployment (model the patterns; don't configure real PHI)

Why Both Kafka and RabbitMQ?

This is the same architectural question as the Digital Wallet — but the healthcare context produces a different answer that is worth knowing independently.

Kafka is an append-only log. Every observation recorded by a bedside monitor, every lab result that arrives from the lab information system, is written to a Kafka topic and retained. Multiple independent consumer groups read the same observation stream at their own pace:

The Elasticsearch indexer maintains a searchable patient dashboard
The sepsis detection engine analyzes rolling windows for SIRS criteria
The data lake writer archives observations as Parquet for long-term regulatory retention
A future billing consumer could derive charges from observation codes without touching the operational database

None of these consumers coordinate with each other. Each holds its own offset. If the sepsis engine is deployed a month after go-live, it can replay all historical observations from offset 0 to catch up. This is only possible because Kafka retains events after consumption.

RabbitMQ handles the action side — what must happen after a clinical event is recognized. When the alert engine detects a critical potassium value, a clinician must be paged. That page is a task: one message, one worker, one action. It must not be processed twice (a duplicate page at 3am is a patient safety concern, not a minor inconvenience). RabbitMQ's acknowledgment model — the message is deleted after exactly one worker acknowledges it — is correct here. Kafka's model is not.

The escalation pattern makes RabbitMQ's dead-letter queue uniquely valuable in this domain. If a physician does not acknowledge a critical alert within five minutes, the original message NACKs into a dead-letter queue with a x-message-ttl of 300 seconds. After that TTL expires, the message is re-routed to an escalation queue targeting the on-call backup. This is the DLQ pattern repurposed as a clinical escalation protocol — a design that does not exist cleanly in Kafka.

Use Case	System	Why
Vital sign streams from monitors	Kafka	Continuous, high-frequency, multiple consumers
Lab result events from LIS	Kafka	Replayable; alert engine and data lake both need it
Encounter admission/discharge events	Kafka	Multiple downstream systems react independently
Sepsis detection analytics	Kafka → Redis	Stateful windowed analysis over the observation stream
Page a physician for a critical value	RabbitMQ	One task, one worker, acknowledged-then-deleted
Escalate if unacknowledged after 5 minutes	RabbitMQ DLQ	Delayed re-delivery is native to DLQ TTL; Kafka has no equivalent
Generate discharge summary PDF	RabbitMQ	Background job; one per discharge, not replayable
Appointment reminder SMS	RabbitMQ	Task queue; idempotent at the SMS provider level

API Conventions

Same response envelope as all other portfolio projects. Prefix: /api/v1.

Success:

{
  "success": true,
  "statusCode": 200,
  "data": {},
  "error": null
}

Error:

{
  "success": false,
  "statusCode": 422,
  "data": null,
  "error": {
    "message": "Observation value exceeds plausible range for this code.",
    "code": "OBSERVATION_OUT_OF_PLAUSIBLE_RANGE"
  }
}

Pagination: Observation history and alert history use cursor pagination on (recorded_at DESC, id DESC) — the table is append-only and grows continuously; offset pagination shifts results as new rows arrive. All other list endpoints use offset pagination (?page=1&pageSize=20).

Idempotency: POST /api/v1/encounters/:id/observations accepts an Idempotency-Key header enforced by a unique partial index. Medical device integrations frequently retry on network failure; a duplicate reading must produce the same response without creating a duplicate observation.

Domain Model

A patient is the central entity. Each patient has a Medical Record Number (MRN) — a stable identifier issued at first registration that never changes, even across multiple encounters.

An encounter is a single clinical episode — an inpatient admission, an outpatient visit, or an emergency department visit. A patient may have many encounters over their lifetime. An encounter has a status (scheduled, active, discharged, cancelled) and a department. All observations, orders, and alerts belong to an encounter, not directly to a patient.

An alert threshold defines the numeric boundaries that trigger a clinical alert for a given observation code. Thresholds are global (not per-patient) and are managed by clinical administrators. Each threshold has four optional bounds: critical_low, warning_low, warning_high, critical_high. A potassium value below critical_low is an immediate life-threatening emergency; a value below warning_low warrants physician review within the hour.

An observation is a single recorded measurement: a vital sign (heart rate, temperature, blood pressure), a lab value (potassium, glucose, white blood cell count), or a pulse oximetry reading. Observations are append-only. They are never updated or deleted. The observation stream is the primary input to both the alert engine and the Kafka pipeline.

A clinical alert is generated when an observation breaches a threshold or when the sepsis detection engine identifies a pattern across multiple recent observations. An alert has a lifecycle: open → acknowledged → resolved. An unacknowledged CRITICAL alert triggers the RabbitMQ escalation after five minutes.

An order is a clinician's instruction: run this lab test, administer this medication, perform this imaging study. Orders have a status lifecycle and a resulted_at timestamp. The reconciliation job uses pending orders to detect cases where a result was never returned.

An outbox event is written in the same transaction as any observation or alert, then relayed to Kafka by a background worker. This decouples Kafka availability from the database transaction.

Features

1. Patient Registration and Encounter Management

Description: Patients are registered with demographic information and assigned a unique MRN. When a patient presents for care, an encounter is opened against their record. Encounters progress through a controlled status machine. A patient cannot have two active encounters of the same type simultaneously.

Endpoints:

POST /api/v1/patients — register patient; generates MRN
GET /api/v1/patients — paginated list with search by name or MRN
GET /api/v1/patients/:id — patient detail with active encounter summary
POST /api/v1/patients/:id/encounters — open an encounter
GET /api/v1/encounters/:id — encounter detail with recent observations and open alerts
PATCH /api/v1/encounters/:id/status — advance status (active → discharged, scheduled → active, etc.); illegal transitions return 409
GET /api/v1/encounters/:id/timeline — merged chronological view: status changes, observation summaries, alerts

Encounter status machine:

scheduled → active → discharged
          → cancelled

A discharged encounter triggers a RabbitMQ job to generate a discharge summary PDF.

Concepts practiced: Aggregate design (encounter owns observations and alerts), controlled state transitions with explicit transition matrix, 409 on illegal transitions, timeline as a composed projection across multiple tables.

2. Alert Threshold Management

Description: Clinical administrators configure the numeric boundaries that define normal, warning, and critical ranges for each observation code. Thresholds are cached in Redis at application startup and invalidated on write — they are read on every observation ingest and must not add database latency to the ingest path.

Endpoints:

POST /api/v1/alert-thresholds
GET /api/v1/alert-thresholds
GET /api/v1/alert-thresholds/:id
PUT /api/v1/alert-thresholds/:id

Data:

{
  "id": "uuid",
  "observationCode": "POTASSIUM_MEQ_L",
  "displayName": "Serum Potassium",
  "unit": "mEq/L",
  "criticalLow": 2.5,
  "warningLow": 3.5,
  "warningHigh": 5.0,
  "criticalHigh": 6.5
}

Concepts practiced: Redis as a configuration cache (not just session/balance cache), cache-aside with write-through invalidation, the difference between data that changes per-request (patient observations) and data that changes per-configuration (thresholds).

3. Observation Ingest

Description: The highest-volume endpoint in the system. Bedside monitors, point-of-care devices, and lab integration systems POST observations continuously. The endpoint must be idempotent (devices retry on network failure), must validate the value against a plausibility range (no human has a heart rate of 400), and must evaluate the observation against alert thresholds on the synchronous path for critical values.

Endpoints:

POST /api/v1/encounters/:id/observations — single or small batch (up to 10); document choice
GET /api/v1/encounters/:id/observations?code=&from=&to=&limit=&cursor= — cursor-paginated observation history with optional code filter

Ingest transaction sequence:

1. Validate encounter is active (not discharged or cancelled)
2. Check idempotency key against unique index
3. Validate observation value within plausible range for the code
4. Insert observation row
5. Load alert threshold for this code from Redis cache (→ PostgreSQL on miss)
6. If value breaches CRITICAL threshold:
   a. Insert clinical_alert row (status: open)
   b. Insert outbox event (topic: alert.generated)
7. Insert outbox event (topic: observation.recorded, payload: full observation)
8. COMMIT

Critical vs warning split: Critical threshold breaches are detected synchronously within the ingest transaction and immediately create an alert. Warning threshold breaches are detected by the Kafka consumer asynchronously — the additional latency (milliseconds to seconds) is acceptable for a warning, but a critical potassium value must trigger a page before the API returns a response.

Idempotency: A device that retries an observation with the same Idempotency-Key receives the original 201 response without creating a duplicate row. The unique partial index enforces this at the database layer.

Concepts practiced: Idempotency key on high-frequency ingest (sd-mid-008), synchronous vs asynchronous alert detection (the split is a clinical safety decision, not an arbitrary one), Redis cache-aside for threshold lookup on the hot path, outbox pattern within ingest transaction.

4. Clinical Alert Lifecycle

Description: Alerts are the patient safety core of the system. Every open CRITICAL alert must be acknowledged by a clinician within five minutes or it escalates. Every alert has a documented audit trail: who acknowledged it, when, and with what note.

Endpoints:

GET /api/v1/encounters/:id/alerts — paginated alert list for an encounter
GET /api/v1/alerts — global alert list filterable by status, severity, department
GET /api/v1/alerts/:id — alert detail
POST /api/v1/alerts/:id/acknowledge — acknowledge with clinician ID and optional note
POST /api/v1/alerts/:id/resolve — resolve (must be acknowledged first)

Alert lifecycle:

open → acknowledged → resolved
     → escalated (via RabbitMQ DLQ after 5 min unacknowledged)

The escalation path: When an alert is created, the outbox relay publishes to alert.generated in Kafka. The notification worker Kafka consumer reads this event and publishes a paging job to RabbitMQ. If the RabbitMQ consumer sends the page but receives no acknowledgment event within five minutes, the message NACKs to the DLQ. After the DLQ TTL expires (300 seconds), the message re-routes to the escalation queue and the on-call backup is paged. The alert status transitions to escalated in PostgreSQL.

Concepts practiced: Alert acknowledgment as a domain event (not just a status update), escalation via DLQ TTL as a healthcare-specific pattern, the difference between an alert being acknowledged in the app vs a clinician physically responding at the bedside.

5. Outbox Relay and Kafka Pipeline

Description: Same pattern as the Digital Wallet. The relay reads unprocessed outbox rows, publishes to Kafka, marks processed. Every observation and every alert flows through this relay to reach Elasticsearch, the sepsis engine, and the data lake independently.

Kafka topics:

Topic	Producer	Consumers
`observation.recorded`	Outbox relay	Elasticsearch indexer, Sepsis engine, Data lake writer
`alert.generated`	Outbox relay	Elasticsearch indexer, Notification worker, Data lake writer
`encounter.status.changed`	Outbox relay	Elasticsearch indexer, Data lake writer

Partition key: encounter_id for observation.recorded and alert.generated. All events for the same encounter land on the same partition, preserving per-encounter ordering. This is important for the sepsis engine: observations for the same patient must be processed in arrival order.

Consumer group isolation: es-indexer, sepsis-engine, and data-lake-writer are separate consumer groups. Each maintains its own committed offset. The sepsis engine processing slowly does not affect the Elasticsearch indexer.

Concepts practiced: Partition key design for per-entity ordering guarantees, consumer group independence, at-least-once delivery via outbox relay (and why consumers must be idempotent), Kafka as the backbone that allows adding new consumers without modifying the producer.

6. Elasticsearch Clinical Search and Analytics (CQRS)

Description: The Elasticsearch indexer maintains a denormalized, queryable projection of the clinical record. It is the read side of CQRS — PostgreSQL is always the write side and the source of truth. The index is optimized for the queries clinicians actually run: "show me all patients with a critical potassium alert in the last hour," "show me the average heart rate trend for this patient over the last 24 hours."

Index shape — patient_encounters:

{
  "encounterId": "uuid",
  "patientId": "uuid",
  "mrn": "MRN-000001",
  "patientName": "Jane Smith",
  "department": "ICU",
  "status": "active",
  "attendingPhysician": "Dr. Osei",
  "admittedAt": "2025-01-01T08:00:00Z",
  "openAlertCount": 2,
  "lastObservationAt": "2025-01-01T09:45:00Z"
}

Index shape — observations:

{
  "observationId": "uuid",
  "encounterId": "uuid",
  "patientId": "uuid",
  "mrn": "MRN-000001",
  "observationCode": "HEART_RATE",
  "value": 118.0,
  "unit": "bpm",
  "source": "DEVICE",
  "recordedAt": "2025-01-01T09:45:00Z"
}

Endpoints:

GET /api/v1/analytics/patients?q=&department=&status= — patient/encounter search across MRN, name, department
GET /api/v1/analytics/observations/trend?encounterId=&code=&from=&to= — time-series aggregation (hourly average, min, max) for a specific observation code
GET /api/v1/analytics/alerts/summary?severity=&from=&to=&department= — alert volume by department and severity over a time window
GET /api/v1/analytics/population?code=&threshold=&from=&to= — how many patients had a value above or below a threshold in a given window

The replay demo: Stop the indexer → delete the Elasticsearch index → reset the es-indexer consumer group offset to 0 → restart → watch both indices rebuild from Kafka history. This is only possible because Kafka retains events. Document this procedure in the project README. It is the most important operational proof-of-concept in the project.

Why Elasticsearch here and not PostgreSQL: The population query — "how many active patients have a heart rate above 100 in the last hour across all departments" — is an aggregation across potentially millions of observation rows. Running this against PostgreSQL on the operational database would compete with ingest writes and introduce latency for both. Elasticsearch's aggregation engine is purpose-built for this pattern. PostgreSQL remains untouched for this query.

Concepts practiced: CQRS read projection design (sd-mid-013), Elasticsearch aggregations as a distinct use case from full-text search (the population endpoint uses no full-text search at all — it is a numeric range aggregation), eventual consistency between PostgreSQL and Elasticsearch, replay as a recovery mechanism.

7. Sepsis Early Warning Engine

Description: A Kafka consumer that reads the observation.recorded stream and detects SIRS (Systemic Inflammatory Response Syndrome) criteria per patient in near real-time. SIRS is a simplified clinical proxy for sepsis risk — when two or more criteria are met simultaneously, a SEPSIS_WARNING alert is generated. State is maintained in Redis as a rolling window of recent observations per encounter.

SIRS criteria (simplified for this project):

Criterion	Observation Code	Trigger
Fever or hypothermia	`TEMP_C`	> 38.3°C or < 36.0°C
Tachycardia	`HEART_RATE`	> 90 bpm
Tachypnea	`RESP_RATE`	> 20 breaths/min
Abnormal WBC	`WBC_K_UL`	> 12.0 or < 4.0 k/µL

Redis state per encounter:

sirs:{encounterId}:TEMP_C      → "1" (TTL: 30 minutes)
sirs:{encounterId}:HEART_RATE  → "1" (TTL: 30 minutes)
sirs:{encounterId}:RESP_RATE   → "1" (TTL: 30 minutes)
sirs:{encounterId}:WBC_K_UL    → "1" (TTL: 30 minutes)

Detection logic per observation event:

1. Evaluate the incoming observation against SIRS criteria
2. If criterion met: SET sirs:{encounterId}:{code} = "1" EX 1800
3. If criterion not met: DEL sirs:{encounterId}:{code}
4. Count active SIRS keys for this encounter (KEYS pattern or MGET)
5. If count >= 2 and no open SEPSIS_WARNING alert exists for this encounter:
   a. Write clinical_alert to PostgreSQL (SEPSIS_WARNING, CRITICAL)
   b. Write outbox event → Kafka alert.generated

Why Redis here and not PostgreSQL: The SIRS evaluation runs on every observation event, potentially multiple times per minute per patient. Checking "which SIRS criteria were met in the last 30 minutes" against PostgreSQL on every event would require a query against the observations table with a time range filter per encounter — under load, this creates read pressure that competes with ingest writes. Redis's O(1) key operations with TTL-based expiry are correct and fast. The TTL handles the sliding window automatically: a heart rate measurement that was abnormal 31 minutes ago stops contributing to the SIRS count without any cleanup job.

Idempotency: If the consumer crashes between detecting SIRS and committing the Kafka offset, it will re-process the same observation on restart. The alert creation query checks for an existing open SEPSIS_WARNING alert before inserting — a duplicate is impossible even with at-least-once delivery.

Concepts practiced: Stateful stream processing with Redis as the state store (sd-senior-011), TTL as a sliding window mechanism, idempotent alert creation, why Kafka consumer + Redis is appropriate here vs a dedicated stream processor like Flink (at the scale of a single hospital, the overhead of a full stream processing framework is not justified — this is a defensible trade-off to articulate in an interview).

8. RabbitMQ Notification Workers and Escalation

Description: The notification worker reads alert.generated events from Kafka and dispatches paging jobs to RabbitMQ. The RabbitMQ consumer sends the page and waits for acknowledgment. If no acknowledgment arrives within five minutes, the dead-letter queue escalates to the on-call backup. If the API host is stopping while a page is in flight, the cancellation path requeues the message rather than escalating it.

Exchange topology:

clinical.notifications.exchange (direct)
    ├── alerts.paging.queue          (physician paging, prefetch=3)
    ├── alerts.paging.dlq            (unacknowledged pages → escalation)
    ├── alerts.escalation.queue      (on-call backup paging)
    ├── notifications.discharge.queue (discharge summary PDF jobs)
    └── notifications.appointment.queue (appointment reminders)

Escalation flow:

1. alert.generated event arrives from Kafka
2. Notification worker publishes to alerts.paging.queue
3. Paging worker sends page to attending physician
4. If no POST /alerts/:id/acknowledge within 5 minutes:
   a. NACK with requeue=false → message goes to alerts.paging.dlq
   b. DLQ has x-message-ttl = 300000ms (5 min)
   c. After TTL: message routes back to alerts.escalation.queue
   d. Escalation worker pages the on-call backup
   e. clinical_alert.status → 'escalated' in PostgreSQL
5. If host shutdown occurs during paging wait:
   a. Cancellation is treated as graceful stop, not failure
   b. NACK with requeue=true → message returns to alerts.paging.queue
   c. No DLQ route, so no false escalation during restart/deploy

Discharge summary job: When an encounter status changes to discharged, the outbox relay publishes to Kafka encounter.status.changed. The notification Kafka consumer reads this and publishes to notifications.discharge.queue. The worker generates a PDF summary (log the content; no real PDF library required), stores it in MinIO under /discharge-summaries/{encounterId}/summary.pdf, and marks the job complete.

Concepts practiced: RabbitMQ exchange-to-queue binding topology, DLQ TTL as a delayed retry and escalation mechanism, prefetch count and worker concurrency, why this escalation pattern is not replicable in Kafka (Kafka has no concept of per-message TTL or conditional re-routing based on consumer acknowledgment).

9. Reconciliation Jobs

Description: Three scheduled checks that verify the system is behaving correctly. Unlike the Digital Wallet where reconciliation checks that money did not disappear, clinical reconciliation checks that actions were taken — alerts were acknowledged, orders were resulted, encounters were closed.

Check 1 — Unacknowledged critical alerts:

SELECT id, encounter_id, triggered_at
FROM clinical_alerts
WHERE severity = 'CRITICAL'
  AND status = 'open'
  AND triggered_at < NOW() - INTERVAL '30 minutes';

Any row here means a critical alert sat open for 30 minutes without acknowledgment or escalation — this is a patient safety failure.

Check 2 — Pending orders without results:

SELECT id, encounter_id, order_type, ordered_at
FROM orders
WHERE status IN ('pending', 'in_progress')
  AND ordered_at < NOW() - INTERVAL '4 hours';

A lab order that has been pending for four hours without a result may indicate a lost sample or a system integration failure.

Check 3 — Active encounters without recent observations:

SELECT e.id, e.patient_id, MAX(o.recorded_at) AS last_observation
FROM encounters e
LEFT JOIN observations o ON o.encounter_id = e.id
WHERE e.status = 'active'
  AND e.encounter_type = 'INPATIENT'
GROUP BY e.id, e.patient_id
HAVING MAX(o.recorded_at) < NOW() - INTERVAL '2 hours'
    OR MAX(o.recorded_at) IS NULL;

An active inpatient without any observation in two hours may indicate a disconnected monitor or a patient who was physically moved without a system update.

Each check creates a reconciliation_alerts row and publishes a job to RabbitMQ for operator notification.

Concepts practiced: Reconciliation as a patient safety mechanism (not just a data integrity mechanism), the difference between "did the data record correctly" (Digital Wallet) and "did the required action happen" (VigilCare), scheduled background jobs in .NET.

10. Observability

Metrics (Prometheus → Grafana):

Metric	Description
`observations_ingested_total`	Counter, labeled by source and observation_code
`observation_ingest_duration_seconds`	Histogram of ingest latency (includes threshold evaluation)
`clinical_alerts_total`	Counter, labeled by alert_type and severity
`alerts_unacknowledged_gauge`	Gauge — open CRITICAL alerts older than 5 minutes
`kafka_consumer_lag`	Per consumer group (es-indexer, sepsis-engine, data-lake-writer)
`outbox_pending_events`	Gauge — unprocessed outbox rows
`sirs_detections_total`	Counter — how many SEPSIS_WARNING alerts the engine generated
`escalations_total`	Counter — how many pages went through DLQ escalation

The alerts_unacknowledged_gauge panel is the most clinically significant metric. If this gauge rises, a nurse station monitor or alerting dashboard must surface it immediately. In a real deployment, this panel would be connected to a paging system. In the portfolio, it demonstrates that you understand which metrics have patient safety implications vs which are purely operational.

Concepts practiced: The four golden signals in a clinical context, which metrics are operational (Kafka lag, outbox pending) vs which are patient safety indicators (unacknowledged critical alerts), log enrichment with correlationId, encounterId, patientId on every alert path log line.

11. Data Lake Writer

Description: A Kafka consumer that reads all three topics and writes partitioned Parquet files to MinIO. In healthcare, long-term retention is not optional — medical records must be retained for 7–25 years depending on jurisdiction. The data lake is the tier that satisfies this requirement without keeping the operational PostgreSQL database at 10-year scale.

File structure:

/observations/2025/01/15/partition-0-offset-0000001.parquet
/alerts/2025/01/15/partition-0-offset-0000001.parquet
/encounters/2025/01/15/partition-0-offset-0000001.parquet

Flush policy: Buffer 1,000 events or 5 minutes, whichever comes first.

Why Parquet: Columnar storage compresses repetitive observation data (many rows with the same observation_code and unit) at ratios of 5–10× vs JSON. A population health query — "give me all heart rate values for patients in the ICU in 2024" — reads only the observation_code and value columns without deserializing the rest of each row. This matters for 10 years of data at a multi-hospital scale.

Concepts practiced: Data lake as a separate retention tier from the operational database, Parquet's columnar advantage over row-based formats for analytics workloads, partition structure as the basis for future query tools (Spark, Athena, DuckDB), regulatory retention as an architectural driver.

Database Schema and Indexing Plan

CREATE TABLE patients (
  id            UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  mrn           VARCHAR(20) NOT NULL UNIQUE,
  first_name    VARCHAR(100) NOT NULL,
  last_name     VARCHAR(100) NOT NULL,
  date_of_birth DATE NOT NULL,
  gender        VARCHAR(10) NOT NULL,
  status        VARCHAR(20) NOT NULL DEFAULT 'active',
  created_at    TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

CREATE TABLE encounters (
  id                  UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  patient_id          UUID NOT NULL REFERENCES patients(id),
  encounter_type      VARCHAR(20) NOT NULL,
  status              VARCHAR(20) NOT NULL DEFAULT 'scheduled',
  department          VARCHAR(100) NOT NULL,
  attending_physician VARCHAR(200) NOT NULL,
  admitted_at         TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  discharged_at       TIMESTAMPTZ NULL,
  created_at          TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

CREATE INDEX idx_encounters_patient ON encounters (patient_id, admitted_at DESC);
CREATE INDEX idx_encounters_active  ON encounters (status, admitted_at DESC) WHERE status = 'active';

CREATE TABLE alert_thresholds (
  id               UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  observation_code VARCHAR(50) NOT NULL UNIQUE,
  display_name     VARCHAR(200) NOT NULL,
  unit             VARCHAR(20) NOT NULL,
  critical_low     DECIMAL(10, 3) NULL,
  warning_low      DECIMAL(10, 3) NULL,
  warning_high     DECIMAL(10, 3) NULL,
  critical_high    DECIMAL(10, 3) NULL,
  created_at       TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

CREATE TABLE observations (
  id                UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  encounter_id      UUID NOT NULL REFERENCES encounters(id),
  observation_code  VARCHAR(50) NOT NULL,
  value             DECIMAL(10, 3) NOT NULL,
  unit              VARCHAR(20) NOT NULL,
  source            VARCHAR(20) NOT NULL DEFAULT 'MANUAL',
  idempotency_key   VARCHAR(100) NULL,
  recorded_at       TIMESTAMPTZ NOT NULL,
  created_at        TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

CREATE UNIQUE INDEX idx_observations_idempotency
  ON observations (idempotency_key) WHERE idempotency_key IS NOT NULL;

CREATE INDEX idx_observations_encounter_time
  ON observations (encounter_id, observation_code, recorded_at DESC);

CREATE TABLE clinical_alerts (
  id              UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  encounter_id    UUID NOT NULL REFERENCES encounters(id),
  patient_id      UUID NOT NULL REFERENCES patients(id),
  observation_id  UUID NULL REFERENCES observations(id),
  alert_type      VARCHAR(50) NOT NULL,
  severity        VARCHAR(20) NOT NULL,
  details         TEXT NOT NULL,
  status          VARCHAR(20) NOT NULL DEFAULT 'open',
  acknowledged_at TIMESTAMPTZ NULL,
  acknowledged_by VARCHAR(200) NULL,
  resolved_at     TIMESTAMPTZ NULL,
  triggered_at    TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

CREATE INDEX idx_alerts_encounter ON clinical_alerts (encounter_id, triggered_at DESC);
CREATE INDEX idx_alerts_patient   ON clinical_alerts (patient_id, triggered_at DESC);
CREATE INDEX idx_alerts_open      ON clinical_alerts (severity, triggered_at DESC)
  WHERE status = 'open';

CREATE TABLE orders (
  id            UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  encounter_id  UUID NOT NULL REFERENCES encounters(id),
  order_type    VARCHAR(20) NOT NULL,
  description   VARCHAR(500) NOT NULL,
  ordered_by    VARCHAR(200) NOT NULL,
  status        VARCHAR(20) NOT NULL DEFAULT 'pending',
  ordered_at    TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  resulted_at   TIMESTAMPTZ NULL
);

CREATE INDEX idx_orders_encounter ON orders (encounter_id, ordered_at DESC);
CREATE INDEX idx_orders_pending   ON orders (status, ordered_at)
  WHERE status IN ('pending', 'in_progress');

CREATE TABLE outbox_events (
  id            UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  topic         VARCHAR(200) NOT NULL,
  payload       JSONB NOT NULL,
  created_at    TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  processed_at  TIMESTAMPTZ NULL
);

CREATE INDEX idx_outbox_pending ON outbox_events (created_at)
  WHERE processed_at IS NULL;

CREATE TABLE reconciliation_alerts (
  id            UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  check_type    VARCHAR(64) NOT NULL,
  encounter_id  UUID NULL REFERENCES encounters(id),
  patient_id    UUID NULL REFERENCES patients(id),
  details       TEXT NOT NULL,
  resolved_at   TIMESTAMPTZ NULL,
  created_at    TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

Design Decisions

Synchronous vs Asynchronous Alert Detection — The Split

The ingest endpoint evaluates critical thresholds synchronously and warning thresholds asynchronously via the Kafka consumer. This is a deliberate patient safety decision.

A critical potassium of 2.1 mEq/L (normal: 3.5–5.0) is immediately life-threatening. If the API returns 201 Created before generating the alert, and the Kafka consumer is lagging by 30 seconds, a patient could deteriorate during that window. The synchronous check costs one additional Redis read per observation on the critical path — acceptable for correctness.

A warning heart rate of 95 bpm (warning threshold: 90) warrants attention but is not an emergency. The additional latency of Kafka consumer processing (milliseconds to seconds) is clinically acceptable for a warning.

This is the architectural decision that separates thinking about healthcare systems from thinking about financial systems. In fintech, milliseconds of latency matter for user experience. In healthcare, the right tradeoff is latency for correctness — and the correctness definition is clinical, not technical.

Observation Codes as Strings (Not an Enum)

Observation codes are stored as VARCHAR rather than a database enum. This allows new device types and lab panels to be registered by inserting a threshold row without a schema migration. The trade-off is that typos in observation codes produce silent mismatches (an observation with code HEART_RATE and a threshold for HEARTRATE would never trigger an alert). The application layer validates incoming codes against the alert_thresholds table on ingest.

In production this would use LOINC codes — an international standard for lab and clinical observations. Knowing that LOINC exists and why it exists (interoperability between systems, not just a naming convention) is a senior talking point.

Why Not a Time-Series Database for Observations?

A medium hospital with 200 concurrent inpatients generating five observations per patient per minute produces approximately 17 observations per second at steady state, peaking near 50/second during shift changes. PostgreSQL with the composite index (encounter_id, observation_code, recorded_at DESC) handles this volume with headroom on any modern server.

A time-series database (InfluxDB, TimescaleDB) would be warranted at sustained 10,000+ observations/second — a large hospital network, not a single facility. TimescaleDB specifically is worth mentioning: it is PostgreSQL with automatic time-based partitioning, meaning it shares the operational model of this project and could be swapped in without changing the query layer. The decision to use vanilla PostgreSQL is correct at this scale and defensible at interview.

The Parquet data lake handles the analytics workload that would otherwise stress the operational database over a 10-year horizon.

Encounter as the Aggregate Root (Not Patient)

Observations, alerts, and orders belong to an encounter, not directly to a patient. This mirrors clinical reality — a patient's blood pressure taken during a 2022 inpatient admission belongs to that admission, not floating freely on the patient record. It also bounds queries naturally: "show me all observations for this encounter" is a bounded query; "show me all observations ever recorded for this patient" is an expensive cross-encounter aggregation that belongs in the data lake, not the operational path.

Non-Functional Requirements

Concern	Target
Critical alert latency	A CRITICAL threshold breach must generate an alert before the ingest response returns
Observation idempotency	Duplicate device submissions with the same key must not create duplicate observations or alerts
Alert acknowledgment	All open CRITICAL alerts must be detectable via the reconciliation job within 30 minutes
Replay	Deleting and rebuilding the Elasticsearch index from Kafka offset 0 must be demonstrable
Retention	Data lake writer must write observations to MinIO; nothing is deleted from the lake
Testing	Integration tests: critical value ingest → alert created; SIRS criteria met across 3 observations → sepsis alert; duplicate idempotency key → no duplicate; encounter discharge → RabbitMQ job published

Build Order

Phase	Focus
1	Schema, migrations, patient/encounter CRUD, alert threshold CRUD, seed data
2	Observation ingest + synchronous critical value detection + alert lifecycle API
3	Outbox relay + Kafka topics + producer
4	Elasticsearch CQRS projection + clinical search + analytics endpoints
5	Sepsis detection engine (Kafka consumer + Redis SIRS state)
6	RabbitMQ notification workers + DLQ escalation
7	Reconciliation jobs (three checks)
8	Prometheus metrics + Grafana dashboards + Seq logging
9	MinIO data lake writer (Parquet, partitioned)

Step-by-Step Guide

Complete phases in order. The synchronous alert path in Phase 2 must be correct before Kafka is introduced in Phase 3 — mixing the two failure modes early makes debugging very difficult.

Phase 1 — Schema, Migrations, and Core CRUD

What to do:

Model all tables in EF Core with migrations matching this PRD.
Enforce the encounter status machine at the service layer — build an explicit transition matrix and return 409 with a stable error code on illegal transitions.
Seed: two patients, one active inpatient encounter each, four alert thresholds (heart rate, temperature, potassium, SpO₂), a set of observations covering normal, warning, and critical ranges.
Implement patient search supporting both MRN (exact match) and name (partial match via ILIKE). Explain in a comment why MRN uses an exact-match index and name uses a prefix scan.
Pre-load all alert thresholds into Redis on application startup using IHostedService. Verify that a threshold update via the API invalidates the cache.

Why: The threshold cache design is worth getting right in Phase 1. Every observation ingest will read from it. Understanding that it is a write-through invalidation (not a TTL expiry) is the correct design for data where staleness has clinical consequences.

Phase 2 — Observation Ingest and Synchronous Alert Detection

What to do:

Implement POST /encounters/:id/observations following the transaction sequence in the Features section exactly.
Load the alert threshold from Redis (not PostgreSQL) inside the ingest transaction. Measure the latency difference with EXPLAIN ANALYZE on the PostgreSQL path for comparison.
On CRITICAL breach: insert the clinical_alerts row and outbox event within the same transaction. Do not return 201 until the alert is written.
On WARNING breach: insert the outbox event only — alert creation is deferred to the Kafka consumer.
Implement cursor-paginated observation history. Verify the composite index (encounter_id, observation_code, recorded_at DESC) is used.
Write integration tests: normal observation (no alert), critical breach (alert created in same transaction), duplicate idempotency key (no duplicate), discharged encounter (reject ingest with 409).

Why: The split between synchronous (critical) and asynchronous (warning) detection is the most clinically significant decision in the codebase. Test both paths and articulate why they are different. A reviewer or interviewer who asks "why not do all alerts asynchronously?" should get a clinical safety answer, not a technical one.

Phase 3 — Outbox Relay and Kafka

What to do:

Implement the outbox relay as an IHostedService polling every 500ms.
Create Kafka topics: observation.recorded, alert.generated, encounter.status.changed.
Partition all topics by encounterId to guarantee per-encounter ordering.
Verify the relay survives a Kafka restart: observations commit to PostgreSQL while Kafka is down; the relay catches up when Kafka recovers.
Introduce the outbox bug deliberately: make two separate commits (one for the observation, one for the outbox event) and observe the data loss when the process crashes between them. Fix it. This step is not optional — seeing the failure mode is the fastest path to internalizing the pattern.

Why: The per-encounter partition key is important for the sepsis engine. If observations from the same patient land on different partitions, they may be processed out of order, and SIRS criteria that arrived simultaneously could be missed. Document this in the code.

Phase 4 — Elasticsearch CQRS Projection

What to do:

Build the es-indexer consumer group. Upsert patient_encounters documents on encounter.status.changed; append to observations index on observation.recorded; update openAlertCount on alert.generated.
Implement the analytics endpoints using Elasticsearch aggregations. The population query is a numeric range filter aggregation — no full-text search at all. Write this query first to make explicit that Elasticsearch is being used here for its aggregation engine, not its search engine.
Write the replay procedure to the README: stop the consumer → delete both indices → reset consumer group offset to 0 → restart → wait for rebuild → verify document count matches PostgreSQL row count.
Run the replay. Verify it completes and the counts match.

Why: The replay is the proof that Elasticsearch is a projection and not a source of truth. It is also the clearest demonstration of why Kafka's event retention matters. Practice running it until it takes less than two minutes to explain what is happening and why it is significant.

Phase 5 — Sepsis Detection Engine

What to do:

Build the sepsis-engine consumer group reading observation.recorded.
Implement the Redis SIRS state as described in the Features section: SET sirs:{encounterId}:{code} EX 1800 on criterion met, DEL on criterion not met.
Use MGET on all four SIRS keys per encounter after each observation — four O(1) operations, not a scan.
On SIRS count >= 2: check for an existing open SEPSIS_WARNING alert for this encounter before inserting. The check and insert are one round-trip: INSERT INTO clinical_alerts ... WHERE NOT EXISTS (SELECT 1 FROM clinical_alerts WHERE encounter_id = ? AND alert_type = 'SEPSIS_WARNING' AND status = 'open').
Write an integration test: ingest three observations that meet two SIRS criteria for the same encounter within 30 minutes → verify one SEPSIS_WARNING alert is created. Ingest a normal temperature immediately after → verify the TTL key is deleted but the alert remains open until acknowledged.

Why: The TTL is doing real work here. Without it, a patient who had a fever yesterday would still have sirs:{encounterId}:TEMP_C = "1" in Redis today and could trigger a false sepsis alert from a fast heart rate alone. The 30-minute TTL matches the clinical window for SIRS evaluation. Understand this before the interview — the TTL is not an arbitrary expiry, it is a clinical parameter encoded in the data layer.

Phase 6 — RabbitMQ Notifications and Escalation

What to do:

Create the exchange and queues from the topology in the Features section. Set x-dead-letter-exchange on alerts.paging.queue pointing to alerts.paging.dlq. Set x-message-ttl = 300000 on alerts.paging.dlq.
Build a Kafka consumer (notification-publisher group) reading alert.generated. For CRITICAL severity alerts: publish a paging job to alerts.paging.queue.
Build the paging worker: log the page (no real pager required), wait for an acknowledged webhook or a timeout, then NACK on timeout.
Build the escalation worker on alerts.escalation.queue: log the escalation, update clinical_alerts.status = 'escalated' in PostgreSQL.
Test the full escalation path: create a critical alert → verify page is published → do not acknowledge → wait for TTL → verify escalation fires → verify alert status is escalated in the database.

Why: The escalation test requires actually waiting 5 minutes (or temporarily setting TTL to 5 seconds in the test environment). Run it. Watching the message appear in the DLQ, wait, and then re-appear in the escalation queue is the moment the DLQ pattern becomes intuitive. It is also the answer to "how would you build escalation in a paging system?" in an interview — a Kafka-native answer does not exist for this pattern.

Phase 7 — Reconciliation Jobs

What to do:

Implement the three reconciliation queries as scheduled IHostedService jobs (every 30 minutes in development).
For each check: if rows are found, insert reconciliation_alerts and publish a job to RabbitMQ.
Test Check 1 by creating a critical alert and not acknowledging it for 31 minutes (advance the triggered_at timestamp in the database directly to simulate time passing).
Test Check 3 by creating an active inpatient encounter and not posting any observations — verify the job detects it.

Why: Check 3 is the one unique to clinical systems. A financial reconciliation job checks that data is correct. This check verifies that the real-world process (a nurse checking vitals) actually happened and was recorded. The system cannot verify that the nurse physically took the measurement — only that a reading was posted. Understanding this limitation is part of the senior conversation.

Phases 8 and 9 — Observability and Data Lake

Follow the same Prometheus/Grafana and MinIO/Parquet approach as described in the Features section. The alerts_unacknowledged_gauge panel is the single most important panel in the Grafana dashboard — build it first and make sure it updates in near real-time (poll the database every 30 seconds).

47 KiB Raw Blame History Unescape Escape

PRD: VigilCare — Clinical Data Pipeline & Real-Time Alert Platform

Overview

Local Development Setup

Goals

Non-Goals

Why Both Kafka and RabbitMQ?

API Conventions

Domain Model

Features

1. Patient Registration and Encounter Management

2. Alert Threshold Management

3. Observation Ingest

4. Clinical Alert Lifecycle

5. Outbox Relay and Kafka Pipeline

6. Elasticsearch Clinical Search and Analytics (CQRS)

7. Sepsis Early Warning Engine

8. RabbitMQ Notification Workers and Escalation

9. Reconciliation Jobs

10. Observability

11. Data Lake Writer

Database Schema and Indexing Plan

Design Decisions

Synchronous vs Asynchronous Alert Detection — The Split

Observation Codes as Strings (Not an Enum)

Why Not a Time-Series Database for Observations?

Encounter as the Aggregate Root (Not Patient)

Non-Functional Requirements

Build Order

Step-by-Step Guide

Phase 1 — Schema, Migrations, and Core CRUD

Phase 2 — Observation Ingest and Synchronous Alert Detection

Phase 3 — Outbox Relay and Kafka

Phase 4 — Elasticsearch CQRS Projection

Phase 5 — Sepsis Detection Engine

Phase 6 — RabbitMQ Notifications and Escalation

Phase 7 — Reconciliation Jobs

Phases 8 and 9 — Observability and Data Lake

47 KiB

Raw Blame History