Trent/vigilcare-clinical
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feature: Explainable Alerts

Browse Source
This commit is contained in:
voltsrage
2026-06-25 00:25:31 +08:00
parent 279add1e45
commit 666d683d67
61 changed files with 9553 additions and 125 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/guides/01-docker-compose-orchestration.md
+467
View File
@@ -0,0 +1,467 @@
# Guide 1: Docker Compose for Multi-Service Orchestration
## What is Docker and Docker Compose?
**Docker** lets you run software inside isolated packages called **containers**. A container bundles an application with everything it needs — the operating system libraries, configuration files, and runtime — so it works the same on every machine. Instead of installing PostgreSQL on your laptop directly (and dealing with version conflicts, path issues, and OS differences), you run a PostgreSQL container that always behaves identically.
A Docker **image** is the blueprint; a **container** is a running instance of that image. You pull images from registries (like Docker Hub), and Docker runs them as containers.
**Docker Compose** is a tool that lets you define and run **multiple containers together** using a single YAML file (`docker-compose.yml`). You describe each service (which image, which ports, which settings) and Compose starts them all with one command: `docker compose up`.
---
## Why Docker Compose?
VigilCareClinical runs 10+ infrastructure services (PostgreSQL, Redis, Kafka, Elasticsearch, RabbitMQ, MinIO, Prometheus, Grafana, Seq) alongside the application API. Docker Compose defines the entire stack in a single `docker-compose.yml` file, allowing a developer to bring up the full environment with one command. Without it, each service would need manual installation, port configuration, and startup ordering — a process that could take hours and differ across machines.
---
## The Compose File Structure
The file lives at the project root: `docker-compose.yml`. It defines three top-level sections:
- **Services** — each container you want to run (which image, which ports to expose, which environment variables to set, where to store data)
- **Volumes** — named storage locations that persist data even when containers are stopped or deleted
- **Networks** — a virtual network so containers can talk to each other
```yaml
volumes:
pg_data:
seq_data:
kafka_data:
es_data:
minio_data:
prometheus_data:
grafana_data:
ward_pg_data:
networks:
vigilcare_net:
driver: bridge
```
**What is a bridge network?** A bridge network is a private network that Docker creates on your machine. Only containers attached to the same bridge can communicate with each other. Think of it like plugging all your services into the same network switch.
All services join `vigilcare_net`, which means inside any container, you can reach PostgreSQL at `postgres:5432`, Redis at `redis:6379`, etc. — Docker's built-in DNS resolves service names to container IPs, so you never need to know the actual IP address.
---
## Service Definitions
### Core Infrastructure
#### PostgreSQL 16 — Primary Database
```yaml
postgres:
image: postgres:16
environment:
POSTGRES_DB: vigilcare
POSTGRES_USER: postgres
POSTGRES_PASSWORD: password
ports:
- "5436:5432"
volumes:
- pg_data:/var/lib/postgresql/data
networks:
- vigilcare_net
```
**Port mapping**: The `"5436:5432"` syntax means "when I connect to port 5436 on my laptop (the host), forward it to port 5432 inside the container." Port 5432 is PostgreSQL's standard port. We use 5436 on the host side to avoid collisions if you already have PostgreSQL installed locally on 5432.
**Named volume**: `pg_data` is a storage location managed by Docker that persists database files across container restarts. Think of it as a hard drive for the container — without this, every time you stop the container with `docker compose down`, all your data would be deleted.
#### Redis 7 — Cache and State Store
```yaml
redis:
image: redis:7-alpine
ports:
- "6382:6379"
networks:
- vigilcare_net
```
Uses the `alpine` variant — Alpine Linux is a tiny Linux distribution, so the image is much smaller (~30MB vs ~130MB for the full image). Smaller images download faster and use less disk space. Port 6382 avoids collision with a local Redis on 6379.
#### Apache Kafka 3.7 — Event Streaming
```yaml
kafka:
image: apache/kafka:3.7.0
ports:
- "9092:9092"
environment:
KAFKA_NODE_ID: 1
KAFKA_PROCESS_ROLES: broker,controller
KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT
KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
KAFKA_AUTO_CREATE_TOPICS_ENABLE: "false"
CLUSTER_ID: "MkU3OEVBNTcwNTJENDM2Qk"
volumes:
- kafka_data:/var/lib/kafka/data
networks:
- vigilcare_net
```
Key design decisions:
- **KRaft mode** (`KAFKA_PROCESS_ROLES: broker,controller`): Older versions of Kafka required a separate service called Zookeeper to coordinate the cluster. Kafka 3.7 can manage itself using a built-in protocol called KRaft (Kafka Raft). This means one less container to run and manage.
- **`KAFKA_AUTO_CREATE_TOPICS_ENABLE: "false"`**: Kafka organizes messages into **topics** (like channels or mailboxes). By default, Kafka auto-creates a topic the first time someone sends a message to it, but it creates it with only 1 partition (limiting parallelism). We disable this and instead create topics explicitly at application startup with the correct settings (6 partitions).
- **`CLUSTER_ID`**: A fixed cluster ID prevents the broker from reinitializing its storage on restart, which would lose all topic data. Think of it like a serial number for the database files.
- **Replication factor = 1**: In production, each message would be copied to 3+ Kafka nodes for redundancy. For local development with a single node, replication of 1 is fine.
#### Elasticsearch 8.13 — Search and Analytics
```yaml
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:8.13.0
environment:
- discovery.type=single-node
- xpack.security.enabled=false
- ES_JAVA_OPTS=-Xms512m -Xmx512m
- cluster.name=vigilcare-dev
ports:
- "9200:9200"
volumes:
- es_data:/usr/share/elasticsearch/data
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:9200/_cluster/health"]
interval: 10s
timeout: 5s
retries: 10
networks:
- vigilcare_net
```
- **`xpack.security.enabled=false`**: Disables TLS encryption and authentication for development. This makes it simpler to connect (no certificates or passwords needed). Production would enable this.
- **`ES_JAVA_OPTS=-Xms512m -Xmx512m`**: Elasticsearch runs on the Java Virtual Machine (JVM). By default it can consume a lot of memory. This caps the JVM heap (working memory) at 512MB to prevent Elasticsearch from consuming all your system memory during development.
- **Health check**: A health check is a command Docker runs periodically to verify the service is actually working, not just running. Here it hits the Elasticsearch health endpoint every 10 seconds. Other services that depend on Elasticsearch can use `condition: service_healthy` to wait until this check passes before starting.
#### RabbitMQ 3.13 — Notification Queuing
```yaml
rabbitmq:
image: rabbitmq:3.13-management-alpine
container_name: vigilcare_rabbitmq
ports:
- "5674:5672" # AMQP protocol
- "15674:15672" # Management UI
environment:
RABBITMQ_DEFAULT_USER: guest
RABBITMQ_DEFAULT_PASS: guest
healthcheck:
test: ["CMD", "rabbitmq-diagnostics", "ping"]
interval: 10s
timeout: 5s
retries: 5
networks:
- vigilcare_net
```
RabbitMQ is a message broker — it accepts, stores, and forwards messages between parts of your application (explained in detail in Guide 7). The `management-alpine` image variant includes a web-based management UI at `http://localhost:15674` where you can see queues, messages, and connections. AMQP (Advanced Message Queuing Protocol) on port 5674 is the wire protocol applications use to talk to RabbitMQ. The health check uses `rabbitmq-diagnostics ping` rather than a simple TCP port check — this validates that the broker process is actually ready to accept connections, not just that the port is open.
#### MinIO — S3-Compatible Object Storage
```yaml
minio:
image: minio/minio:RELEASE.2024-07-04T14-25-45Z
container_name: vigilcare_minio
command: server /data --console-address ":9001"
ports:
- "9005:9000" # S3 API
- "9006:9001" # Web console
environment:
MINIO_ROOT_USER: minioadmin
MINIO_ROOT_PASSWORD: minioadmin
volumes:
- minio_data:/data
networks:
- vigilcare_net
```
MinIO is an open-source object storage server that speaks the same API as Amazon S3. This means you can develop locally against MinIO and deploy to AWS S3 later without changing your code. The `command` override tells MinIO to expose its web console on a separate port. Two ports: 9005 for the S3 API (your application code reads and writes files here), 9006 for the web console where you can browse stored files in your browser.
#### Seq — Log Aggregation
```yaml
seq:
image: datalust/seq:latest
environment:
ACCEPT_EULA: "Y"
SEQ_FIRSTRUN_ADMINPASSWORD: "admin"
ports:
- "5345:80"
volumes:
- seq_data:/data
networks:
- vigilcare_net
```
Seq provides structured log search and filtering. The API runs on its default port 80 inside the container, mapped to 5345 externally. Serilog ships logs to `http://localhost:5345`.
### Observability Services
#### Prometheus — Metrics Collection
```yaml
prometheus:
image: prom/prometheus:v2.52.0
container_name: vigilcare_prometheus
command:
- "--config.file=/etc/prometheus/prometheus.yml"
- "--storage.tsdb.retention.time=7d"
ports:
- "9101:9090"
volumes:
- ./infra/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
- prometheus_data:/prometheus
extra_hosts:
- "host.docker.internal:host-gateway"
networks:
- vigilcare_net
```
- **`extra_hosts`**: Containers live in their own private network. `host.docker.internal:host-gateway` adds a DNS entry so the Prometheus container can reach your laptop (the "host machine") where the .NET API runs. On macOS and Windows this mapping exists by default, but Linux needs it explicitly. The prometheus.yml config then uses `host.docker.internal:5270` as the target to scrape metrics from.
- **`--storage.tsdb.retention.time=7d`**: Keeps 7 days of metrics data (TSDB stands for Time-Series DataBase — Prometheus's internal storage format). This prevents disk usage from growing indefinitely during development.
- **Bind mount vs named volume**: The `:ro` suffix means "read-only." A **bind mount** (`./infra/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml`) maps a specific file on your laptop directly into the container. Unlike a named volume (which Docker manages), you edit the file on your laptop and the container sees the changes immediately. The `:ro` flag means the container can read the file but cannot write to it.
#### Grafana — Dashboard Visualization
```yaml
grafana:
image: grafana/grafana:10.4.3
container_name: vigilcare_grafana
ports:
- "3101:3000"
environment:
GF_SECURITY_ADMIN_USER: admin
GF_SECURITY_ADMIN_PASSWORD: admin
GF_USERS_ALLOW_SIGN_UP: "false"
volumes:
- ./infra/grafana/provisioning:/etc/grafana/provisioning:ro
- ./infra/grafana/dashboards:/var/lib/grafana/dashboards:ro
- grafana_data:/var/lib/grafana
depends_on:
- prometheus
networks:
- vigilcare_net
```
- **`depends_on: prometheus`**: Grafana starts after Prometheus. Note that `depends_on` without a `condition` only controls **start order**, not readiness — Docker starts Grafana after starting (not after Prometheus is fully ready). Grafana won't crash if Prometheus takes a few seconds to start up, but the datasource won't return data until Prometheus is actually accepting connections.
- **Provisioning volumes**: Two read-only bind mounts auto-configure Grafana on first start:
- `provisioning/` — datasource configuration pointing Grafana to `http://prometheus:9090`
- `dashboards/` — pre-built dashboard JSON files loaded automatically
- **`grafana_data` volume**: Persists Grafana's internal database (saved queries, user preferences) across restarts.
---
## Profiles: Optional Service Groups
The ward gateway services use Docker Compose **profiles** to avoid starting them during normal development:
```yaml
ward-gateway-db:
image: postgres:16
profiles: ["ward-gateway", "full"]
# ...
ward-gateway-redis:
image: redis:7-alpine
profiles: ["ward-gateway", "full"]
# ...
ward-gateway-rabbitmq:
image: rabbitmq:3.13-management-alpine
profiles: ["ward-gateway", "full"]
# ...
ward-gateway-api:
profiles: ["ward-gateway", "full"]
build:
context: .
dockerfile: VigilCare.WardGateway/Dockerfile
# ...
```
Services without a `profiles` key start by default. Services with profiles only start when you explicitly request that profile:
```bash
# Start only the core stack (no gateway services)
docker compose up -d
# Start core + ward gateway
docker compose --profile ward-gateway up -d
# Start everything
docker compose --profile full up -d
```
This keeps the default development experience lightweight — 9 services instead of 13.
---
## The Ward Gateway Service
The gateway is the only application service that runs inside Docker (the main API runs on the host via `dotnet run`). It uses a **multi-stage Dockerfile**:
```dockerfile
FROM mcr.microsoft.com/dotnet/sdk:8.0 AS build
WORKDIR /src
COPY VigilCareClinical.sln ./
COPY VigilCare.ClinicalContracts/ VigilCare.ClinicalContracts/
COPY VigilCare.WardGateway/ VigilCare.WardGateway/
RUN dotnet restore VigilCare.WardGateway/VigilCare.WardGateway.csproj
RUN dotnet publish VigilCare.WardGateway/VigilCare.WardGateway.csproj \
-c Release -o /app/publish --no-restore
FROM mcr.microsoft.com/dotnet/aspnet:8.0 AS runtime
WORKDIR /app
COPY --from=build /app/publish .
ENV ASPNETCORE_URLS=http://+:8080
EXPOSE 8080
ENTRYPOINT ["dotnet", "VigilCare.WardGateway.dll"]
```
**What is a multi-stage build?** A Dockerfile can have multiple `FROM` lines, each starting a new "stage." The first stage (`AS build`) uses the full .NET SDK image (~900MB) to compile the code. The second stage (`AS runtime`) starts fresh from the smaller ASP.NET runtime image (~220MB) and copies only the compiled output. The final image doesn't contain the SDK, source code, or NuGet package cache — it's lean and production-ready. This is like using a full workshop to build furniture, then shipping only the finished table to the customer.
**Build context**: Set to `.` (project root) because the gateway depends on `VigilCare.ClinicalContracts` — a shared library. The Dockerfile needs to COPY both projects into the build stage.
### Gateway Environment Variables
The gateway container receives all its configuration through environment variables, overriding `appsettings.json`:
```yaml
environment:
ASPNETCORE_ENVIRONMENT: Development
ConnectionStrings__GatewayDb: "Host=ward-gateway-db;Port=5432;..."
Redis__ConnectionString: "ward-gateway-redis:6379"
RabbitMq__Host: ward-gateway-rabbitmq
CentralApi__BaseUrl: "http://host.docker.internal:5270"
Gateway__GatewayId: "22222222-2222-2222-2222-222222222222"
GATEWAY_SYNC_JWT: "${GATEWAY_SYNC_JWT:-}"
```
Key patterns:
- **Double underscore** (`__`) maps to nested JSON keys. ASP.NET Core reads environment variables and translates `__` into the `:` separator used in JSON. So `ConnectionStrings__GatewayDb` overrides `{"ConnectionStrings": {"GatewayDb": "..."}}` in `appsettings.json`.
- **`host.docker.internal`** in `CentralApi__BaseUrl`: The gateway runs inside Docker, but the main API runs on your laptop. This special hostname lets the container reach back out to your laptop.
- **`${GATEWAY_SYNC_JWT:-}`**: This is shell variable substitution. Docker Compose reads the value of `GATEWAY_SYNC_JWT` from a `.env` file in the project root or from your host environment. The `:-` means "use an empty string as the default if the variable isn't set."
### Gateway Dependency Conditions
```yaml
depends_on:
ward-gateway-db:
condition: service_healthy
ward-gateway-redis:
condition: service_started
ward-gateway-rabbitmq:
condition: service_healthy
```
- **`service_healthy`** waits for the health check to pass. Used for PostgreSQL and RabbitMQ which need time to initialize.
- **`service_started`** only waits for the container process to start. Used for Redis which is ready almost immediately.
---
## Health Checks
Three services have explicit health checks:
| Service | Check Command | Interval | Retries |
|---------|--------------|----------|---------|
| Elasticsearch | `curl -f http://localhost:9200/_cluster/health` | 10s | 10 |
| RabbitMQ | `rabbitmq-diagnostics ping` | 10s | 5 |
| ward-gateway-db | `pg_isready -U postgres` | 10s | 5 |
Health checks serve two purposes:
1. **Dependency ordering**: `depends_on` with `condition: service_healthy` blocks until the check passes.
2. **Visibility**: `docker compose ps` shows health status per container.
Services without health checks (Kafka, Redis, MinIO) start quickly enough that ordering isn't critical, or the application handles connection retries itself (e.g., `ThresholdCacheLoader` retries Redis 3 times with exponential backoff).
---
## The `extra_hosts` Pattern
```yaml
extra_hosts:
- "host.docker.internal:host-gateway"
```
This appears on two services: **Prometheus** and **ward-gateway-api**. On Linux, Docker doesn't provide `host.docker.internal` by default (it does on macOS and Windows). The `host-gateway` magic value resolves to the host machine's IP on the Docker bridge network.
This is needed because the .NET API runs on the host (via `dotnet run`), not inside Docker. Prometheus needs to scrape `host.docker.internal:5270/metrics`, and the ward gateway needs to call `host.docker.internal:5270` as the central API.
---
## Port Mapping Summary
| Service | Host Port | Container Port | Purpose |
|---------|-----------|---------------|---------|
| PostgreSQL | 5436 | 5432 | Primary database |
| Redis | 6382 | 6379 | Cache/state |
| Kafka | 9092 | 9092 | Event streaming |
| Elasticsearch | 9200 | 9200 | Search |
| RabbitMQ | 5674 | 5672 | AMQP |
| RabbitMQ Management | 15674 | 15672 | Web UI |
| MinIO S3 | 9005 | 9000 | Object storage API |
| MinIO Console | 9006 | 9001 | Web UI |
| Prometheus | 9101 | 9090 | Metrics |
| Grafana | 3101 | 3000 | Dashboards |
| Seq | 5345 | 80 | Log aggregation |
| Ward Gateway DB | 5437 | 5432 | Gateway database |
| Ward Gateway Redis | 6383 | 6379 | Gateway cache |
| Ward Gateway RabbitMQ | 5675 | 5672 | Gateway queues |
| Ward Gateway API | 5081 | 8080 | Gateway HTTP |
All host ports are intentionally non-standard to avoid collisions with locally installed services.
---
## Common Commands
```bash
# Start the core stack (background)
docker compose up -d
# Start with ward gateway
docker compose --profile ward-gateway up -d
# View running containers and health status
docker compose ps
# View logs for a specific service
docker compose logs -f kafka
# Restart a single service
docker compose restart elasticsearch
# Stop everything but keep volumes (data preserved)
docker compose down
# Stop everything and delete volumes (clean slate)
docker compose down -v
# Rebuild the ward gateway image after code changes
docker compose --profile ward-gateway build ward-gateway-api
docker compose --profile ward-gateway up -d ward-gateway-api
```
---
## How the API Connects
The API runs on the host machine (`dotnet run`), not in Docker. It connects to containerized services using the **host ports** defined in the compose file. These are configured in `appsettings.json`:
```json
{
"ConnectionStrings": {
"DefaultConnection": "Host=localhost;Port=5436;Database=vigilcare;..."
},
"Redis": { "ConnectionString": "localhost:6382" },
"Kafka": { "BootstrapServers": "localhost:9092" },
"Elasticsearch": { "Uri": "http://localhost:9200" },
"RabbitMq": { "Host": "localhost", "Port": 5674 },
"Minio": { "Endpoint": "localhost:9005" },
"Seq": { "ServerUrl": "http://localhost:5345" }
}
```
This split — infrastructure in Docker, application on the host — gives the best development experience: fast iteration on application code (no rebuild/restart cycle) with production-like infrastructure.
docs/guides/02-prometheus-grafana-monitoring.md
+561
View File
@@ -0,0 +1,561 @@
# Guide 2: Prometheus + Grafana Monitoring Stack
## What Are Prometheus and Grafana?
**Prometheus** is a time-series database designed for monitoring. Instead of your application pushing data to Prometheus, Prometheus **pulls** (or "scrapes") data from your application on a schedule — typically every 15 seconds. Your application exposes a `/metrics` HTTP endpoint with the current values of all metrics, and Prometheus stores a timestamped history of those values.
A **time-series** is just a series of numbers recorded over time — like "at 14:00 there were 3 pending events, at 14:15 there were 5, at 14:30 there were 0." Prometheus stores millions of these series efficiently and lets you query them with its built-in query language, PromQL.
**Grafana** is a visualization tool that connects to Prometheus (and other data sources) and renders interactive dashboards — line graphs, gauges, stat panels, and alerts. Prometheus stores the numbers; Grafana makes them visual.
**Why use them together?** Logging tells you _what happened_ ("observation ingested for encounter X"). Metrics tell you _how the system is performing right now_ ("we're ingesting 50 observations per second and the p99 latency is 23ms"). When something goes wrong, metrics tell you instantly — often before anyone notices a problem.
---
## Why Prometheus and Grafana in This Project?
VigilCareClinical is a patient safety system. A growing outbox means delayed alerts. High Kafka consumer lag means scores aren't being computed. An unacknowledged critical alert means a patient needs attention. Prometheus collects these metrics every 15 seconds, and Grafana visualizes them on dashboards that clinicians and engineers can watch in real time.
---
## Architecture Overview
```
.NET API (host) Docker
┌──────────────────┐ scrape ┌─────────────┐ query ┌─────────────┐
│ /metrics │ ◄──────────── │ Prometheus │ ◄───────── │ Grafana │
│ (prometheus-net)│ every 15s │ :9101 │ │ :3101 │
└──────────────────┘ └─────────────┘ └─────────────┘
▲
│ expose
┌───────┴──────────┐
│ ClinicalMetrics │ (singleton)
│ 4 Collectors │ (BackgroundServices)
└──────────────────┘
```
1. **prometheus-net** (a .NET library) exposes a `/metrics` endpoint on the API that outputs all metric values in Prometheus's text format
2. **Prometheus** (running in Docker) scrapes that endpoint every 15 seconds and stores the values with timestamps
3. **Grafana** (also in Docker) queries Prometheus and renders the data as charts and dashboards
---
## Step 1: The NuGet Package
```xml
<!-- VigilCareClinicalAPI.csproj -->
<PackageReference Include="prometheus-net.AspNetCore" Version="8.2.1" />
```
This package provides:
- The `Metrics` factory for creating counters, histograms, and gauges
- The `MapMetrics()` extension method to expose the `/metrics` HTTP endpoint
- ASP.NET Core middleware integration
---
## Step 2: Registering the Metrics Endpoint
In `Program.cs`:
```csharp
// Register ClinicalMetrics as a singleton so all services share the same instances
builder.Services.AddSingleton<ClinicalMetrics>();
// ... later, after building the app:
// Expose /metrics for Prometheus to scrape
app.MapMetrics("/metrics");
```
That single `MapMetrics("/metrics")` call serves the Prometheus text exposition format at `http://localhost:5270/metrics`.
---
## Step 3: Defining Custom Metrics — ClinicalMetrics
All custom metrics are defined in a single class, registered as a singleton. This ensures every service that injects `ClinicalMetrics` increments the same counter instances.
### Counters
Prometheus has three core metric types. The first is a **counter**.
A counter only goes up — it never decreases. Think of it like the odometer on a car: it tracks the total distance driven since the car was built. A counter tracks things like "total observations ingested since the app started." The raw number isn't that useful on its own (who cares that we've ingested 50,000 total?), but Prometheus's `rate()` function converts it to "observations per second over the last minute" — that's actionable.
```csharp
public sealed class ClinicalMetrics
{
// Labeled by observation_code and source
public readonly Counter ObservationsIngestedTotal = Metrics.CreateCounter(
"observations_ingested_total",
"Total observations ingested, labeled by observation code and source.",
labelNames: new[] { "observation_code", "source" });
// Labeled by alert_type and severity
public readonly Counter ClinicalAlertsTotal = Metrics.CreateCounter(
"clinical_alerts_total",
"Total clinical alerts generated, labeled by type and severity.",
labelNames: new[] { "alert_type", "severity" });
public readonly Counter News2ScoresTotal = Metrics.CreateCounter(
"news2_scores_total",
"Total NEWS2 scores computed, labeled by risk level.",
labelNames: new[] { "risk_level" });
public readonly Counter EscalationsTotal = Metrics.CreateCounter(
"escalations_total",
"Total alert escalations processed through the DLQ escalation path.");
public readonly Counter TrendAlertsTotal = Metrics.CreateCounter(
"trend_alerts_total",
"Total RAPID_DETERIORATION alerts generated.",
labelNames: new[] { "observation_code" });
public readonly Counter AlertSuppressionsTotal = Metrics.CreateCounter(
"alert_suppressions_total",
"Total alert suppression windows set after acknowledgment.",
labelNames: new[] { "alert_type" });
public readonly Counter QsofaDetectionsTotal = Metrics.CreateCounter(
"qsofa_detections_total",
"Total QSOFA_SCREEN alerts generated.");
public readonly Counter SepsisBundleComplianceTotal = Metrics.CreateCounter(
"sepsis_bundle_compliance_total",
"Sepsis bundle compliance outcomes.",
labelNames: new[] { "status" });
public readonly Counter GcsScoresTotal = Metrics.CreateCounter(
"gcs_scores_total",
"GCS scores computed, labeled by classification.",
labelNames: new[] { "classification" });
public readonly Counter SofaScoresTotal = Metrics.CreateCounter(
"sofa_scores_total",
"SOFA scores computed.",
labelNames: new[] { "has_delta_alert" });
public readonly Counter FhirIngestTotal = Metrics.CreateCounter(
"fhir_ingest_total",
"FHIR resource ingest operations.",
labelNames: new[] { "resource_type", "outcome" });
public readonly Counter FhirReadTotal = Metrics.CreateCounter(
"fhir_read_total",
"FHIR resource read/search operations.",
labelNames: new[] { "resource_type", "interaction", "outcome" });
public readonly Counter AuthorizationFailuresTotal = Metrics.CreateCounter(
"authorization_failures_total",
"Authorization failures by permission and role.",
labelNames: new[] { "permission", "role" });
public readonly Counter FhirMappingErrorsTotal = Metrics.CreateCounter(
"fhir_mapping_errors_total",
"FHIR mapping failures.",
labelNames: new[] { "reason" });
public readonly Counter PhiAccessLogsTotal = Metrics.CreateCounter(
"phi_access_logs_total",
"PHI access log entries written.",
labelNames: new[] { "access_type" });
public readonly Counter ClinicalSyncBatchesTotal = Metrics.CreateCounter(
"clinical_sync_batches_total",
"Sync batches processed.",
labelNames: new[] { "status" });
}
```
**What are labels?** Labels are key-value tags attached to a metric that let you slice and filter the data. Instead of creating separate counters like `critical_heart_rate_alerts_total` and `warning_spo2_alerts_total`, you create one counter `clinical_alerts_total` with labels `alert_type` and `severity`. This single counter then lets you query:
- Total alerts: `clinical_alerts_total`
- Only critical threshold breaches: `clinical_alerts_total{severity="Critical"}`
- Only SOFA sepsis alerts: `clinical_alerts_total{alert_type="SOFA_SEPSIS"}`
### Histograms
A **histogram** tracks the _distribution_ of values, not just a total count. For example, "how long does it take to ingest an observation?" — a counter can tell you how many were ingested, but a histogram tells you that 50% completed in under 10ms, 95% in under 50ms, and the slowest 1% took over 200ms.
Histograms work by defining **buckets** — thresholds like 5ms, 10ms, 25ms, 50ms, etc. Each observation is counted into all buckets it falls below. Prometheus then stores `_bucket` (how many observations fell into each bucket), `_sum` (total time across all observations), and `_count` (total number of observations). From these, you can calculate percentiles using the `histogram_quantile()` function in PromQL.
```csharp
public readonly Histogram ObservationIngestDuration = Metrics.CreateHistogram(
"observation_ingest_duration_seconds",
"Ingest transaction duration from request receipt to COMMIT.",
new HistogramConfiguration
{
Buckets = new[] { 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0 }
});
public readonly Histogram News2ScoringDuration = Metrics.CreateHistogram(
"news2_scoring_duration_seconds",
"Time to compute a NEWS2 score from Redis state.",
new HistogramConfiguration
{
Buckets = new[] { 0.001, 0.005, 0.01, 0.025, 0.05, 0.1 }
});
public readonly Histogram TrendAnalysisDuration = Metrics.CreateHistogram(
"trend_analysis_duration_seconds",
"Time to evaluate trend for one observation.",
new HistogramConfiguration
{
Buckets = new[] { 0.001, 0.005, 0.01, 0.025, 0.05, 0.1 }
});
public readonly Histogram SofaScoringDuration = Metrics.CreateHistogram(
"sofa_scoring_duration_seconds",
"SOFA scoring computation time.",
new HistogramConfiguration
{
Buckets = new[] { 0.001, 0.005, 0.01, 0.025, 0.05, 0.1 }
});
public readonly Histogram ClinicalSyncBatchDuration = Metrics.CreateHistogram(
"clinical_sync_batch_duration_seconds",
"Batch processing duration.",
new HistogramConfiguration
{
Buckets = new[] { 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0 }
});
```
**Bucket selection matters**: The ingest histogram uses fine-grained buckets (5ms to 1s) because sub-second latency is critical for patient safety. The sync batch histogram uses coarser buckets (100ms to 10s) because batch operations are inherently slower.
### Gauges
A **gauge** is a value that can go up or down — like a thermometer or a fuel gauge. It represents the _current state_ of something: "right now there are 3 unacknowledged alerts" or "the outbox has 47 pending events." Unlike counters (which only increase), gauges are set to an absolute value by background services that periodically check the current state.
```csharp
public readonly Gauge AlertsUnacknowledgedGauge = Metrics.CreateGauge(
"alerts_unacknowledged_gauge",
"Count of open CRITICAL alerts older than 5 minutes with no acknowledgment.");
public readonly Gauge KafkaConsumerLag = Metrics.CreateGauge(
"kafka_consumer_lag",
"Approximate consumer group lag in messages.",
labelNames: new[] { "consumer_group" });
public readonly Gauge OutboxPendingEvents = Metrics.CreateGauge(
"outbox_pending_events",
"Count of outbox events not yet relayed to Kafka.");
public readonly Gauge WardGatewaysOffline = Metrics.CreateGauge(
"ward_gateways_offline_gauge",
"Ward gateways with status OFFLINE or DEGRADED.",
labelNames: new[] { "site_code" });
public readonly Gauge WardGatewayBufferDepth = Metrics.CreateGauge(
"ward_gateway_buffer_depth",
"Reported unsynced event count per gateway.",
labelNames: new[] { "gateway_code", "department" });
public readonly Gauge AlertAcknowledgementRate = Metrics.CreateGauge(
"vigilcare_alert_acknowledgement_rate",
"Alert acknowledgement rate by type.",
labelNames: new[] { "alert_type" });
public readonly Gauge AlertFalsePositiveRate = Metrics.CreateGauge(
"vigilcare_alert_false_positive_rate",
"Clinician-reported false positive rate by type.",
labelNames: new[] { "alert_type" });
public readonly Gauge AlertUsefulRate = Metrics.CreateGauge(
"vigilcare_alert_useful_rate",
"Clinician-reported useful rate by type.",
labelNames: new[] { "alert_type" });
public readonly Gauge AlertAvgAckSeconds = Metrics.CreateGauge(
"vigilcare_alert_avg_ack_seconds",
"Average seconds from trigger to acknowledgement by type.",
labelNames: new[] { "alert_type" });
```
---
## Step 4: Using Metrics in Application Code
### Inline Counters and Timers
In `ObservationService`, the ingest path both times the transaction and increments counters:
```csharp
using var timer = _metrics.ObservationIngestDuration.NewTimer();
// ... perform the ingest transaction ...
_metrics.ObservationsIngestedTotal
.WithLabels(req.ObservationCode, req.Source.ToDbString())
.Inc();
if (alert is not null)
{
_metrics.ClinicalAlertsTotal
.WithLabels(alert.AlertType.ToDbString(), alert.Severity.ToDbString())
.Inc();
}
```
**How does `NewTimer()` work?** In C#, a `using` block runs some cleanup code when the block exits. `NewTimer()` starts a stopwatch and the `using` block ensures it records the elapsed time into the histogram when the code leaves the block — even if an exception is thrown. This means you don't need to manually calculate timing.
**What does `WithLabels()` do?** It selects which specific "bucket" of the counter to increment. A counter with labels is really a _family_ of counters — one for each unique label combination. `.WithLabels("HEART_RATE", "DEVICE")` increments the counter for heart rate observations from devices specifically.
---
## Step 5: Background Collectors
Gauges can't be updated inline in request handlers because they represent _current state_ ("how many right now?"), not events ("one more just happened"). You need a background process that periodically checks the current state and updates the gauge.
In .NET, a `BackgroundService` is a class that runs continuously in the background for the lifetime of the application. Four of these poll data sources on a timer and update gauge values.
### AlertsUnacknowledgedCollector (every 30s)
The most clinically significant metric. Queries PostgreSQL for CRITICAL alerts that have been open more than 5 minutes with no acknowledgment:
```csharp
protected override async Task ExecuteAsync(CancellationToken ct)
{
using var timer = new PeriodicTimer(TimeSpan.FromSeconds(30));
while (await timer.WaitForNextTickAsync(ct))
await CollectAsync(ct);
}
private async Task CollectAsync(CancellationToken ct)
{
await using var scope = _scopes.CreateAsyncScope();
var db = scope.ServiceProvider.GetRequiredService<AppDbContext>();
var cutoff = DateTimeOffset.UtcNow - TimeSpan.FromMinutes(5);
var count = await db.ClinicalAlerts
.CountAsync(a => a.Severity == AlertSeverity.Critical
&& a.Status == AlertStatus.Open
&& a.TriggeredAt < cutoff, ct);
_metrics.AlertsUnacknowledgedGauge.Set(count);
if (count > 0)
_logger.LogWarning(
"[PATIENT-SAFETY] alerts_unacknowledged_gauge={Count}", count);
}
```
It also logs a warning when the count is non-zero — this shows up in Seq as a structured log entry tagged `[PATIENT-SAFETY]`.
### OutboxPendingCollector (every 30s)
Counts outbox events not yet relayed to Kafka. A growing number means the relay is falling behind or Kafka is unreachable:
```csharp
var count = await db.OutboxEvents
.CountAsync(e => e.ProcessedAt == null, ct);
_metrics.OutboxPendingEvents.Set(count);
```
### KafkaConsumerLagCollector (every 30s)
Measures how far behind each Kafka consumer group is. **Consumer lag** is the number of messages that have been published but not yet processed. Think of it like a queue at a bank — lag is how many people are still waiting. This collector uses Kafka's admin API to check the lag for each consumer group without actually consuming any messages:
```csharp
private static readonly string[] Groups =
{
"es-indexer",
"sepsis-engine",
"notification-publisher",
"data-lake-writer",
};
private async Task CollectGroupLagAsync(string groupId, CancellationToken ct)
{
using var admin = new AdminClientBuilder(adminConfig).Build();
// Get committed offsets for this consumer group
var result = await admin.ListConsumerGroupOffsetsAsync(...);
// Query high watermarks with a temporary consumer
using var tempConsumer = new ConsumerBuilder<Ignore, Ignore>(config).Build();
long totalLag = 0;
foreach (var tpo in partitions)
{
var watermarks = tempConsumer.QueryWatermarkOffsets(
tpo.TopicPartition, TimeSpan.FromSeconds(5));
var lag = watermarks.High.Value - tpo.Offset.Value;
totalLag += Math.Max(0L, lag);
}
_metrics.KafkaConsumerLag.WithLabels(groupId).Set(totalLag);
}
```
The lag is per consumer group. If `sepsis-engine` has high lag but `es-indexer` doesn't, the Grafana panel makes it immediately visible which pipeline is falling behind.
### WardGatewayMetricsCollector (every 60s)
Queries PostgreSQL for gateway status and buffer depth:
```csharp
var offlineBySite = await db.WardGateways
.Include(g => g.Site)
.Where(g => g.Status != GatewayStatus.Online)
.GroupBy(g => g.Site.SiteCode)
.Select(g => new { SiteCode = g.Key, Count = g.Count() })
.ToListAsync(ct);
foreach (var row in offlineBySite)
_metrics.WardGatewaysOffline.WithLabels(row.SiteCode).Set(row.Count);
var allGateways = await db.WardGateways.AsNoTracking().ToListAsync(ct);
foreach (var g in allGateways)
_metrics.WardGatewayBufferDepth
.WithLabels(g.GatewayCode, g.Department)
.Set(g.ReportedBufferDepth);
```
All four collectors follow the same pattern:
1. **Extend `BackgroundService`** — .NET's base class for long-running background work
2. **Use `PeriodicTimer` for the poll loop** — fires a callback at a fixed interval (30s or 60s)
3. **Create a DI scope per tick** — in .NET's dependency injection (DI), database contexts are "scoped" (one per request/operation). Background services are singletons (one for the whole app), so they need to create a new scope each tick to get a fresh database context
4. **Catch and log errors without crashing the collector** — a transient database timeout shouldn't kill the metrics collection permanently
---
## Step 6: Prometheus Configuration
File: `infra/prometheus/prometheus.yml`
```yaml
global:
scrape_interval: 15s
evaluation_interval: 15s
scrape_configs:
- job_name: vigilcare_api
static_configs:
- targets: ["host.docker.internal:5270"]
metrics_path: /metrics
```
- **`scrape_interval: 15s`**: Prometheus pulls metrics every 15 seconds. This determines the resolution (granularity) of your data — you'll have one data point every 15 seconds. A shorter interval gives more detail but uses more storage and CPU.
- **`host.docker.internal:5270`**: The API runs on the host machine on port 5270. The `extra_hosts` entry in `docker-compose.yml` makes this hostname resolvable from inside the Prometheus container.
- **`metrics_path: /metrics`**: Matches the `app.MapMetrics("/metrics")` endpoint in Program.cs.
---
## Step 7: Grafana Provisioning
Grafana is configured entirely through file provisioning — no manual setup needed after `docker compose up`.
### Datasource: `infra/grafana/provisioning/datasources/prometheus.yml`
```yaml
apiVersion: 1
datasources:
- name: Prometheus
type: prometheus
url: http://prometheus:9090
isDefault: true
access: proxy
```
Uses the Docker service name `prometheus` (not `localhost`) because Grafana runs inside the Docker network.
### Dashboard Provider: `infra/grafana/provisioning/datasources/dashboards/config.yml`
```yaml
apiVersion: 1
providers:
- name: VigilCare
type: file
options:
path: /var/lib/grafana/dashboards
```
Tells Grafana to load all JSON files from `/var/lib/grafana/dashboards`, which is bind-mounted from `./infra/grafana/dashboards/`.
---
## Step 8: Grafana Dashboards
### Clinical Operations Dashboard (`vigilcare.json`)
| Panel | Type | Query | Purpose |
|-------|------|-------|---------|
| Unacknowledged Critical Alerts | Stat | `alerts_unacknowledged_gauge` | Primary safety indicator |
| Observation Ingest Rate | Timeseries | `rate(observations_ingested_total[1m])` | Data flow health |
| Clinical Alerts by Type (5m) | Bar gauge | `increase(clinical_alerts_total[5m])` | Alert volume breakdown |
| Outbox Pending Events | Stat | `outbox_pending_events` | Outbox relay health |
| Kafka Consumer Lag by Group | Timeseries | `kafka_consumer_lag` | Per-pipeline lag |
| Ingest Latency p50 / p99 | Timeseries | `histogram_quantile(0.50, rate(...[5m]))` / `histogram_quantile(0.99, rate(...[5m]))` | Performance SLA |
| Escalations Total | Stat | `escalations_total` | Missed alert tracking |
| Ward Gateways Offline | Stat | `sum(ward_gateways_offline_gauge)` | Edge connectivity |
| Gateway Buffer Depth | Bar gauge | `ward_gateway_buffer_depth` | Per-gateway backlog |
| Clinical Sync Batches (5m rate) | Timeseries | `rate(clinical_sync_batches_total[5m])` | Sync throughput |
| Total Sync Backlog | Stat | `sum(ward_gateway_buffer_depth)` | Fleet-wide backlog |
### Alert Quality Dashboard (`alert-quality-dashboard.json`)
| Panel | Type | Query | Purpose |
|-------|------|-------|---------|
| Acknowledgement Rate by Type | Timeseries | `vigilcare_alert_acknowledgement_rate` | Are alerts being seen? |
| False Positive Rate by Type | Timeseries | `vigilcare_alert_false_positive_rate` | Alert fatigue tracking |
| Useful Rate by Type | Timeseries | `vigilcare_alert_useful_rate` | Clinical value |
| Avg Ack Seconds by Type | Timeseries | `vigilcare_alert_avg_ack_seconds` | Response time SLA |
---
## Metrics Reference
### All Counters
| Metric | Labels | What It Tracks |
|--------|--------|---------------|
| `observations_ingested_total` | `observation_code`, `source` | Vital sign ingest volume |
| `clinical_alerts_total` | `alert_type`, `severity` | Alert generation rate |
| `news2_scores_total` | `risk_level` | NEWS2 scoring frequency |
| `escalations_total` | — | Unacknowledged alert escalations |
| `trend_alerts_total` | `observation_code` | Rapid deterioration detections |
| `alert_suppressions_total` | `alert_type` | Suppression window activations |
| `qsofa_detections_total` | — | qSOFA screen triggers |
| `sepsis_bundle_compliance_total` | `status` | Bundle compliance outcomes |
| `gcs_scores_total` | `classification` | GCS scoring by severity |
| `sofa_scores_total` | `has_delta_alert` | SOFA scoring with/without alerts |
| `fhir_ingest_total` | `resource_type`, `outcome` | FHIR inbound operations |
| `fhir_read_total` | `resource_type`, `interaction`, `outcome` | FHIR read/search operations |
| `authorization_failures_total` | `permission`, `role` | Failed auth attempts |
| `fhir_mapping_errors_total` | `reason` | FHIR mapping failures |
| `phi_access_logs_total` | `access_type` | PHI access audit entries |
| `clinical_sync_batches_total` | `status` | Gateway sync batch outcomes |
### All Histograms
| Metric | Buckets | What It Tracks |
|--------|---------|---------------|
| `observation_ingest_duration_seconds` | 5ms–1s | Full ingest transaction time |
| `news2_scoring_duration_seconds` | 1ms–100ms | NEWS2 computation time |
| `trend_analysis_duration_seconds` | 1ms–100ms | Trend evaluation time |
| `sofa_scoring_duration_seconds` | 1ms–100ms | SOFA computation time |
| `clinical_sync_batch_duration_seconds` | 100ms–10s | Batch processing time |
### All Gauges
| Metric | Labels | Collector | Interval |
|--------|--------|-----------|----------|
| `alerts_unacknowledged_gauge` | — | `AlertsUnacknowledgedCollector` | 30s |
| `kafka_consumer_lag` | `consumer_group` | `KafkaConsumerLagCollector` | 30s |
| `outbox_pending_events` | — | `OutboxPendingCollector` | 30s |
| `ward_gateways_offline_gauge` | `site_code` | `WardGatewayMetricsCollector` | 60s |
| `ward_gateway_buffer_depth` | `gateway_code`, `department` | `WardGatewayMetricsCollector` | 60s |
| `vigilcare_alert_acknowledgement_rate` | `alert_type` | `AlertQualityAggregatorService` | 60min |
| `vigilcare_alert_false_positive_rate` | `alert_type` | `AlertQualityAggregatorService` | 60min |
| `vigilcare_alert_useful_rate` | `alert_type` | `AlertQualityAggregatorService` | 60min |
| `vigilcare_alert_avg_ack_seconds` | `alert_type` | `AlertQualityAggregatorService` | 60min |
---
## Accessing the Stack
| Tool | URL | Credentials |
|------|-----|-------------|
| Prometheus | http://localhost:9101 | None |
| Grafana | http://localhost:3101 | admin / admin |
| Raw metrics | http://localhost:5270/metrics | JWT or anonymous |
docs/guides/03-structured-logging-serilog-seq.md
+479
View File
@@ -0,0 +1,479 @@
# Guide 3: Structured Logging with Serilog + Seq
## What Are Serilog and Seq?
**Logging** is how your application records what it's doing — "user logged in," "observation saved," "database connection failed." Every application needs logging for debugging, auditing, and monitoring.
**Serilog** is a logging library for .NET that replaces the built-in `Microsoft.Extensions.Logging`. The key difference: Serilog captures log data as **structured events** (with named fields you can query) rather than flat text strings. It sends these events to one or more **sinks** — destinations like the console, a file, or a log server.
**Seq** is a log server with a web UI. It receives structured log events from Serilog over HTTP, stores them, and lets you search and filter them through a browser interface. Think of it as a specialized search engine for your application's logs.
---
## Why Structured Logging?
Traditional text logs look like this:
```
[2026-06-24 14:23:01] WARNING: Critical threshold breach for encounter 3fa85f64-5717-4562-b3fc-2c963f66afa6
```
You can grep for the encounter ID, but you can't query "show me all critical breaches in the last hour" without parsing free-form text. Structured logging captures each piece of information as a named property:
```json
{
"Timestamp": "2026-06-24T14:23:01Z",
"Level": "Warning",
"MessageTemplate": "Critical threshold breach. Code={Code} Value={Value} AlertId={AlertId}",
"Properties": {
"Code": "HEART_RATE",
"Value": 182.0,
"AlertId": "9b2a1c3d-...",
"EncounterId": "3fa85f64-...",
"PatientId": "7e4b2a1f-...",
"CorrelationId": "abc-123-def",
"MachineName": "dev-laptop",
"ThreadId": 14
}
}
```
Now you can filter by `Code = "HEART_RATE"`, group by `EncounterId`, or correlate across services using `CorrelationId`.
---
## Architecture
```
Application Code
│
│ _logger.LogInformation("...", ...)
▼
Serilog Pipeline
│
├── Enrichers (automatically add extra properties to every event)
│
├──► Console Sink (prints to your terminal during development)
│
└──► Seq Sink ──────► Seq Server (http://localhost:5345)
│
└── Web UI: search, filter, dashboards
```
**What is a sink?** A sink is a destination where Serilog sends log events. Think of it like plumbing — log events flow from your code through the pipeline and out to one or more sinks. You can have multiple sinks active simultaneously (the same event goes to the console AND to Seq).
**What is an enricher?** An enricher automatically attaches extra properties to every log event as it flows through the pipeline. For example, the `MachineName` enricher adds the computer's hostname to every event without you writing any extra code.
---
## Step 1: NuGet Packages
```xml
<!-- VigilCareClinicalAPI.csproj -->
<PackageReference Include="Serilog.AspNetCore" Version="10.0.0" />
<PackageReference Include="Serilog.Enrichers.Environment" Version="2.3.0" />
<PackageReference Include="Serilog.Enrichers.Thread" Version="3.1.0" />
<PackageReference Include="Serilog.Sinks.Console" Version="6.1.1" />
<PackageReference Include="Serilog.Sinks.Seq" Version="9.1.0" />
```
| Package | Purpose |
|---------|---------|
| `Serilog.AspNetCore` | Integrates Serilog with the ASP.NET Core host, replaces the default Microsoft logger |
| `Serilog.Enrichers.Environment` | Adds `MachineName` to every log event |
| `Serilog.Enrichers.Thread` | Adds `ThreadId` to every log event |
| `Serilog.Sinks.Console` | Writes to stdout (visible in the terminal during `dotnet run`) |
| `Serilog.Sinks.Seq` | Ships structured events to the Seq server over HTTP |
---
## Step 2: Configuration
Serilog is configured in two places: `appsettings.json` (declarative) and `Program.cs` (code).
### appsettings.json
```json
{
"Serilog": {
"Using": ["Serilog.Sinks.Console", "Serilog.Sinks.Seq"],
"MinimumLevel": {
"Default": "Information",
"Override": {
"Microsoft.AspNetCore": "Warning",
"Microsoft.EntityFrameworkCore.Database.Command": "Information"
}
},
"WriteTo": [
{ "Name": "Console" },
{
"Name": "Seq",
"Args": {
"serverUrl": "http://localhost:5345"
}
}
],
"Enrich": ["FromLogContext", "WithMachineName", "WithThreadId"]
}
}
```
Key settings:
- **`MinimumLevel.Default: Information`**: Log levels from lowest to highest are: Debug < Information < Warning < Error < Fatal. This setting means "log Information and above, but drop Debug-level events." Debug events are very chatty and usually only turned on temporarily when investigating a specific issue.
- **`Override: Microsoft.AspNetCore: Warning`**: The ASP.NET Core framework generates its own logs ("Request starting HTTP/1.1 GET /api/...", "Request finished ..."). At the Information level, this creates a flood of framework noise that drowns out your application's logs. Setting it to Warning means you only see framework logs when something goes wrong.
- **`Override: Microsoft.EntityFrameworkCore.Database.Command: Information`**: An exception to the rule above — this keeps EF Core SQL command logging visible. During development, it's useful to see the actual SQL queries being generated by your LINQ code.
- **`WriteTo`**: Two sinks run simultaneously. Every log event goes to both Console (your terminal) and Seq (the log server). This is one of Serilog's superpowers — the same event, multiple destinations, with no extra code.
- **`Enrich`**: Three enrichers automatically attach properties to every event: `FromLogContext` (reads any properties pushed by your code), `WithMachineName` (computer hostname), `WithThreadId` (which thread is running).
### Program.cs
```csharp
if (!builder.Environment.IsEnvironment("Testing"))
{
builder.Host.UseSerilog((ctx, services, config) =>
config.ReadFrom.Configuration(ctx.Configuration)
.ReadFrom.Services(services)
.Enrich.FromLogContext()
.Enrich.WithMachineName()
.Enrich.WithThreadId());
}
```
The `if (!Testing)` guard is needed because Serilog's setup is a one-time operation per process. Integration tests can create multiple fake servers within a single test run, and initializing Serilog a second time would throw an error.
`ReadFrom.Configuration` reads the `Serilog` section from appsettings.json. This means you can change logging settings (add sinks, change levels) by editing config files without recompiling your code.
---
## Step 3: Request Logging
Every time someone calls your API (e.g., `POST /api/encounters/123/observations`), ASP.NET Core's default logger writes multiple log events: "request starting," "reading headers," "writing response," "request finished." That's 4+ events per request — noisy and hard to read. Serilog replaces all of those with a single, compact summary event:
```csharp
app.UseSerilogRequestLogging(options =>
{
options.MessageTemplate =
"HTTP {RequestMethod} {RequestPath} responded {StatusCode} in {Elapsed:0.000}ms";
});
```
This produces one structured event per request with properties:
- `RequestMethod` (GET, POST, PATCH)
- `RequestPath` (/api/encounters/123/observations)
- `StatusCode` (200, 400, 500)
- `Elapsed` (milliseconds)
The `{Elapsed:0.000}` format shows 3 decimal places (microsecond precision).
---
## Step 4: Correlation IDs
**What is a correlation ID?** When a single user action (like recording a vital sign) triggers work across multiple services — the API, Kafka consumers, the notification system — each service writes its own logs. A correlation ID is a unique identifier that ties all of these log entries together. By searching for one correlation ID in Seq, you can see every log event from every service that was involved in processing that one request. Without it, you'd have no way to connect "observation ingested" in the API to "qSOFA evaluated" in the sepsis engine to "page sent" in the notification service.
**What is middleware?** In ASP.NET Core, middleware is code that runs on every HTTP request, in a pipeline. Each middleware component can inspect the request, do some work, and pass it to the next middleware in the chain. Think of it like a series of checkpoints at an airport — each checkpoint does one thing (check ID, scan bags, stamp passport).
The `CorrelationIdMiddleware` adds a `CorrelationId` property to every log event within a request:
```csharp
public sealed class CorrelationIdMiddleware
{
private const string Header = "X-Correlation-Id";
private readonly RequestDelegate _next;
public CorrelationIdMiddleware(RequestDelegate next) => _next = next;
public async Task InvokeAsync(HttpContext ctx)
{
var correlationId = ctx.Request.Headers[Header].FirstOrDefault()
?? Guid.NewGuid().ToString();
ctx.Response.Headers[Header] = correlationId;
using (LogContext.PushProperty("CorrelationId", correlationId))
{
await _next(ctx);
}
}
}
```
How it works:
1. Check if the incoming HTTP request has an `X-Correlation-Id` header (callers like the ward gateway can set this so their ID propagates)
2. If no header is present, generate a new GUID (a random unique identifier)
3. Echo the ID back in the response header so the caller can use it for their own logging
4. Push it onto Serilog's `LogContext` — this is the key part. `LogContext.PushProperty` makes the correlation ID appear automatically on every `_logger.Log*()` call within this request, without passing it explicitly as a parameter
The `using` block ensures the property is removed when the request completes. Without this, the property could "leak" into the next request handled on the same thread, causing logs to have the wrong correlation ID.
Registered in the middleware pipeline in `Program.cs`:
```csharp
app.UseMiddleware<CorrelationIdMiddleware>();
app.UseMiddleware<FhirApiKeyOrJwtMiddleware>();
app.UseMiddleware<ExceptionHandlerMiddleware>();
```
Order matters — `CorrelationIdMiddleware` runs first so that all subsequent middleware and handlers have the correlation ID in their log context.
---
## Step 5: Adding Context to Logs with LogContext.PushProperty
The correlation ID middleware adds context to the entire request. But sometimes you want to add context for just part of the request — for example, when processing a specific encounter. `LogContext.PushProperty` lets you push additional named properties that automatically appear on all log events within a `using` block:
```csharp
// In ObservationService.IngestAsync
using (LogContext.PushProperty("EncounterId", encounterId))
using (LogContext.PushProperty("PatientId", encounter.PatientId))
{
// Every log call within this block automatically includes
// EncounterId and PatientId as structured properties
_logger.LogInformation(
"Observation {ObservationId} ingested for encounter {EncounterId}. AlertCreated={AlertCreated}",
observation.Id, encounterId, alert is not null);
}
```
The `{EncounterId}` in the message template creates a property from the parameter. The `LogContext.PushProperty("EncounterId", encounterId)` adds it as ambient context — even if a called method doesn't pass it explicitly, it appears on the log event.
---
## Step 6: Exception Handling Middleware
The `ExceptionHandlerMiddleware` catches all unhandled exceptions and logs them at the appropriate level:
```csharp
public async Task InvokeAsync(HttpContext context)
{
try
{
await _next(context);
}
catch (NotFoundException ex)
{
_logger.LogWarning("{Message}", ex.Message);
await WriteAsync(context, 404,
ApiResponse<object>.Fail(404, ex.Message, ex.ErrorCode));
}
catch (BadRequestException ex)
{
_logger.LogWarning("{Message}", ex.Message);
await WriteAsync(context, 400,
ApiResponse<object>.Fail(400, ex.Message, ex.ErrorCode));
}
catch (ValidationException ex)
{
_logger.LogWarning("{Message}", ex.Message);
await WriteAsync(context, 422,
ApiResponse<object>.Fail(422, ex.Message, ex.ErrorCode));
}
catch (ConflictException ex)
{
_logger.LogWarning("{Message}", ex.Message);
await WriteAsync(context, 409,
ApiResponse<object>.Fail(409, ex.Message, ex.ErrorCode));
}
catch (Exception ex)
{
_logger.LogError(ex, "Unhandled exception");
await WriteAsync(context, 500,
ApiResponse<object>.Fail(500, "An unexpected error occurred", "INTERNAL_ERROR"));
}
}
```
Design decisions:
- **Client errors (4xx)** are logged as `Warning` — they're expected and don't indicate system problems
- **Server errors (5xx)** are logged as `Error` with the full exception — these need investigation
- The exception object is passed as the first argument to `LogError(ex, ...)`, which captures the stack trace as a structured property in Seq
- The response body never leaks exception details to the client — it returns a generic "An unexpected error occurred" message
---
## Step 7: Logging Patterns in Background Services
Background services log lifecycle events and errors with consistent patterns:
### Startup Announcement
```csharp
// OutboxRelayService
_logger.LogInformation("Outbox relay started. PollInterval={Interval}ms",
_options.OutboxPollIntervalMs);
// TrendAnalyzerService
_logger.LogInformation("TrendAnalyzerService started — consumer group: trend-analyzer");
```
### Error Recovery
```csharp
// OutboxRelayService - logs and continues to next poll cycle
catch (Exception ex) when (ex is not OperationCanceledException)
{
_logger.LogError(ex, "Outbox relay error — will retry on next poll cycle");
}
```
The `when (ex is not OperationCanceledException)` filter is a C# exception filter. `OperationCanceledException` is thrown when the application is shutting down — it's expected and normal, not an error. Without this filter, every graceful shutdown would log a scary-looking error message.
### Retry with Backoff
```csharp
// ThresholdCacheLoader - retries Redis 3 times with exponential backoff
catch (RedisException ex)
{
_logger.LogWarning(ex,
"Redis unavailable during threshold cache load — attempt {Attempt}/{Max}",
attempt + 1, maxAttempts);
}
// After all retries exhausted:
_logger.LogError(
"Failed to load thresholds into Redis after {Max} attempts — " +
"application will start without cache; observation ingest falls back to PostgreSQL",
maxAttempts);
```
The warning-then-error pattern: retryable failures are `Warning` (not alarming), final failure is `Error` (needs attention).
### Patient Safety Logs
```csharp
// AlertsUnacknowledgedCollector
if (count > 0)
_logger.LogWarning(
"[PATIENT-SAFETY] alerts_unacknowledged_gauge={Count} " +
"(CRITICAL alerts open > 5 min)", count);
```
The `[PATIENT-SAFETY]` prefix is a convention for logs that indicate clinical risk. In Seq, you can create a saved search for this prefix.
### Clinical Event Logs
```csharp
// ObservationService - critical threshold breach
_logger.LogWarning(
"Critical threshold breach. Code={Code} Value={Value} AlertId={AlertId}",
req.ObservationCode, req.Value, alert.Id);
// ObservationService - every observation ingest
_logger.LogInformation(
"Observation {ObservationId} ingested for encounter {EncounterId}. AlertCreated={AlertCreated}",
observation.Id, encounterId, alert is not null);
// Duplicate detection
_logger.LogInformation(
"Duplicate idempotency key {Key} for encounter {EncounterId} — returning original",
req.IdempotencyKey, encounterId);
```
---
## Step 8: Structured Property Naming Conventions
The project follows consistent naming patterns:
| Pattern | Example | When to use |
|---------|---------|-------------|
| `{EntityId}` | `{ObservationId}`, `{AlertId}` | Primary key of the entity being processed |
| `{EntityProperty}` | `{Code}`, `{Value}`, `{Status}` | Properties of the entity |
| `{Count}` | `{Count}` | Numeric counts |
| `{Attempt}/{Max}` | `{Attempt}/{Max}` | Retry tracking |
| `[TAG]` prefix | `[PATIENT-SAFETY]`, `[RECONCILIATION]` | Category markers for saved searches |
Always use **message templates** with named placeholders, never C# string interpolation:
```csharp
// CORRECT — creates structured properties
_logger.LogInformation("Observation {ObservationId} ingested", observation.Id);
// WRONG — creates a flat string, loses structured queryability
_logger.LogInformation($"Observation {observation.Id} ingested");
```
These look similar but behave very differently. The first form uses Serilog's **message template** syntax — the `{ObservationId}` placeholder creates a named property that Seq can index, filter, and group by. The second form uses C#'s `$""` string interpolation, which bakes the value directly into the message text before Serilog ever sees it. Seq can only do full-text search on it, not structured queries. This is the single most important rule to follow with structured logging.
---
## Seq: The Log Aggregation Server
### Docker Setup
```yaml
seq:
image: datalust/seq:latest
environment:
ACCEPT_EULA: "Y"
SEQ_FIRSTRUN_ADMINPASSWORD: "admin"
ports:
- "5345:80"
volumes:
- seq_data:/data
```
### Accessing Seq
Open `http://localhost:5345` in a browser. Login: admin / admin.
### What You Can Do in Seq
- **Search by property**: `CorrelationId = "abc-123"` shows every log event from a single request across all services
- **Filter by level**: Click "Warning" to see only warnings and errors
- **Filter by source**: `SourceContext like "OutboxRelay%"` shows only outbox relay logs
- **Time range**: Narrow to a specific time window when an incident occurred
- **Live tail**: Watch logs stream in real time during replay simulation
### Useful Seq Queries
```
# All critical threshold breaches
AlertId is not null and @Level = 'Warning' and @MessageTemplate like '%threshold breach%'
# All patient safety events
@Message like '[PATIENT-SAFETY]%'
# Trace a single request across the system
CorrelationId = 'your-correlation-id-here'
# All errors in the last hour
@Level = 'Error' and @Timestamp > Now() - 1h
# All observation ingests for a specific encounter
EncounterId = '3fa85f64-5717-4562-b3fc-2c963f66afa6'
# Outbox relay problems
SourceContext like 'OutboxRelay%' and @Level in ['Warning', 'Error']
```
---
## Log Level Guidelines
| Level | When to Use | Example |
|-------|-------------|---------|
| `Debug` | Detailed diagnostic info, noisy, off by default | Kafka lag collection failures, ES debug upserts |
| `Information` | Normal operations worth recording | Service started, observation ingested, batch processed |
| `Warning` | Unexpected but handled situations | Duplicate idempotency key, Redis retry, threshold breach, client errors (4xx) |
| `Error` | Failures that need investigation | Unhandled exceptions, max retries exhausted, service crashes |
---
## Enrichers Summary
Every log event automatically includes these properties:
| Property | Source | Example Value |
|----------|--------|---------------|
| `CorrelationId` | `CorrelationIdMiddleware` | `"a1b2c3d4-e5f6-..."` |
| `MachineName` | `Serilog.Enrichers.Environment` | `"dev-laptop"` |
| `ThreadId` | `Serilog.Enrichers.Thread` | `14` |
| `SourceContext` | Serilog (automatic from `ILogger<T>`) | `"ObservationService"` |
| `RequestMethod` | `UseSerilogRequestLogging` | `"POST"` |
| `RequestPath` | `UseSerilogRequestLogging` | `"/api/encounters/123/observations"` |
| `StatusCode` | `UseSerilogRequestLogging` | `201` |
Plus any ambient properties pushed via `LogContext.PushProperty` (like `EncounterId`, `PatientId`).
docs/guides/04-postgresql-entity-framework-core.md
+525
View File
@@ -0,0 +1,525 @@
# Guide 4: PostgreSQL with Entity Framework Core
## What Are PostgreSQL and Entity Framework Core?
**PostgreSQL** (often called "Postgres") is an open-source relational database — it stores data in tables with rows and columns, and you query it using SQL. If you've used MySQL or SQL Server, PostgreSQL works similarly but offers advanced features like JSONB columns (storing JSON data that you can query), partial indexes (indexes that only cover some rows), and robust support for concurrent transactions.
**Entity Framework Core** (EF Core) is an **ORM** — an Object-Relational Mapper. Without an ORM, you'd write raw SQL strings in your C# code, manually map database columns to C# properties, and handle connection management yourself. An ORM lets you work with database rows as if they were regular C# objects:
```csharp
// Without ORM — raw SQL, manual mapping
var sql = "SELECT id, encounter_id, value FROM observations WHERE encounter_id = @id";
// ... execute, read columns, create objects manually
// With EF Core — C# objects, LINQ queries
var observations = await db.Observations
.Where(o => o.EncounterId == encounterId)
.ToListAsync();
```
EF Core translates your LINQ queries into SQL, maps the results back to C# objects, tracks changes, and generates database migrations (versioned schema changes).
---
## Why PostgreSQL + EF Core in This Project?
PostgreSQL is the primary relational database for all clinical data — patients, encounters, observations, alerts, scores, audit logs, and the transactional outbox. EF Core provides the ORM layer that maps C# entities to PostgreSQL tables, handles migrations, and generates SQL while allowing raw SQL when needed (like `FOR UPDATE SKIP LOCKED` for the outbox relay).
---
## Architecture Overview
```
Application Code
│
│ _db.Observations.Add(...)
│ _db.SaveChangesAsync()
▼
AppDbContext (EF Core)
│
├── Entity Configurations (IEntityTypeConfiguration<T>)
│ ├── Column mappings (snake_case)
│ ├── Check constraints
│ ├── Indexes (unique, partial, composite)
│ └── Value conversions (enum ↔ string, PHI encryption)
│
├── Migrations (40+ tracked schema changes)
│
└──► PostgreSQL 16
├── JSONB columns (audit logs, staleness flags, alert explanations)
├── Partial unique indexes (idempotency, active encounters)
├── Sequences (MRN generation)
└── FOR UPDATE SKIP LOCKED (outbox concurrency)
```
---
## The DbContext
**What is a DbContext?** The `DbContext` is the main class you interact with in EF Core. It represents a session with the database — you use it to query data, add new records, and save changes. Think of it as a "database connection wrapper" that knows about your tables and how your C# classes map to them.
Each `DbSet<T>` property represents one table. `DbSet<Patient>` means "the `patients` table, where each row maps to a `Patient` C# object."
`AppDbContext` is this project's DbContext. It declares all 24 `DbSet<T>` properties and loads entity configurations from the assembly:
```csharp
public class AppDbContext : DbContext
{
private readonly IPhiEncryptionService? _phiCrypto;
public AppDbContext(
DbContextOptions<AppDbContext> options,
IPhiEncryptionService? phiCrypto = null) : base(options)
{
_phiCrypto = phiCrypto;
}
public DbSet<Patient> Patients => Set<Patient>();
public DbSet<Encounter> Encounters => Set<Encounter>();
public DbSet<Observation> Observations => Set<Observation>();
public DbSet<ClinicalAlert> ClinicalAlerts => Set<ClinicalAlert>();
public DbSet<OutboxEvent> OutboxEvents => Set<OutboxEvent>();
public DbSet<Order> Orders => Set<Order>();
public DbSet<News2Score> News2Scores => Set<News2Score>();
public DbSet<GcsScore> GcsScores => Set<GcsScore>();
public DbSet<SofaScore> SofaScores => Set<SofaScore>();
public DbSet<QsofaEvaluation> QsofaEvaluations => Set<QsofaEvaluation>();
public DbSet<SepsisBundle> SepsisBundles => Set<SepsisBundle>();
public DbSet<SepsisBundleElement> SepsisBundleElements => Set<SepsisBundleElement>();
public DbSet<MedicationAdministration> MedicationAdministrations => Set<MedicationAdministration>();
public DbSet<AlertThreshold> AlertThresholds => Set<AlertThreshold>();
public DbSet<ClinicalAuditLog> ClinicalAuditLogs => Set<ClinicalAuditLog>();
public DbSet<ClinicalUser> ClinicalUsers => Set<ClinicalUser>();
public DbSet<ExternalResourceIdentifier> ExternalResourceIdentifiers => Set<ExternalResourceIdentifier>();
public DbSet<PhiAccessLog> PhiAccessLogs => Set<PhiAccessLog>();
public DbSet<ClinicalSite> ClinicalSites => Set<ClinicalSite>();
public DbSet<WardGateway> WardGateways => Set<WardGateway>();
public DbSet<ClinicalSyncBatch> ClinicalSyncBatches => Set<ClinicalSyncBatch>();
public DbSet<ClinicalSyncConflict> ClinicalSyncConflicts => Set<ClinicalSyncConflict>();
public DbSet<AlertFeedback> AlertFeedbacks => Set<AlertFeedback>();
public DbSet<AlertQualityMetric> AlertQualityMetrics => Set<AlertQualityMetric>();
public DbSet<ReconciliationAlert> ReconciliationAlerts => Set<ReconciliationAlert>();
protected override void OnModelCreating(ModelBuilder modelBuilder)
{
modelBuilder.ApplyConfigurationsFromAssembly(typeof(AppDbContext).Assembly);
if (_phiCrypto is not null)
modelBuilder.ConfigurePhiConverters(_phiCrypto);
}
}
```
Key design decisions:
- **`ApplyConfigurationsFromAssembly`** automatically finds all entity configuration classes in the project (explained below) instead of registering them one by one
- **Optional `IPhiEncryptionService`** — PHI stands for Protected Health Information (patient names, dates of birth, etc.). When this service is provided, EF Core automatically encrypts PHI columns when writing to the database and decrypts when reading. Integration tests pass `null` to skip encryption for simpler test setup.
- **Expression-body DbSets** (`=> Set<T>()`) — a concise C# syntax. This is functionally the same as a property with a getter
### Registration in Program.cs
```csharp
builder.Services.AddDbContext<AppDbContext>((sp, opts) =>
{
opts.UseNpgsql(builder.Configuration.GetConnectionString("DefaultConnection"));
});
```
Connection string from `appsettings.json`:
```json
{
"ConnectionStrings": {
"DefaultConnection": "Host=localhost;Port=5436;Database=vigilcare;Username=postgres;Password=password"
}
}
```
Port 5436 matches the Docker Compose mapping (host 5436 → container 5432).
---
## Entity Configurations
**What is an entity configuration?** EF Core needs to know how your C# classes map to database tables — which property maps to which column, what the column type is, which columns have indexes, etc. You configure this by creating a class that implements `IEntityTypeConfiguration<T>` for each entity.
Each entity has a dedicated configuration class in `Data/Configurations/`. This keeps the `DbContext` clean and puts all the schema details for one table in one file.
### Snake_Case Column Mapping
PostgreSQL convention is `snake_case` (like `encounter_id`). C# convention is `PascalCase` (like `EncounterId`). Every column is explicitly mapped to bridge the two naming styles:
```csharp
builder.Property(o => o.EncounterId).HasColumnName("encounter_id");
builder.Property(o => o.ObservationCode).HasColumnName("observation_code")
.HasMaxLength(50).IsRequired();
builder.Property(o => o.Value).HasColumnName("value")
.HasColumnType("decimal(10,3)").IsRequired();
builder.Property(o => o.CreatedAt).HasColumnName("created_at")
.HasDefaultValueSql("NOW()");
```
### UUID Primary Keys
All entities use `Guid` primary keys, generated by PostgreSQL:
```csharp
builder.Property(p => p.Id).HasColumnName("id")
.HasDefaultValueSql("gen_random_uuid()");
```
`gen_random_uuid()` generates UUIDs server-side, so the application doesn't need to generate them unless it wants to (which it does for outbox events and alerts, where the ID is needed before the INSERT).
### Enum-to-String Conversions
**What is a value conversion?** EF Core can automatically convert between your C# type and the database type when reading and writing. This is configured with `HasConversion()`.
In C#, enums are typically stored as integers internally (`Critical = 0`, `Warning = 1`). But if you store those integers in the database, the data is unreadable without the code ("what does status 2 mean?"), and renumbering the enum breaks everything. Instead, this project stores enums as human-readable strings:
```csharp
builder.Property(a => a.Severity)
.HasColumnName("severity")
.HasMaxLength(20)
.HasConversion(
v => v.ToDbString(), // AlertSeverity.Critical → "CRITICAL"
v => AlertSeverityExtensions.FromDbString(v)) // "CRITICAL" → AlertSeverity.Critical
.IsRequired();
```
The `ToDbString()` / `FromDbString()` pattern is used for all enums: `AlertStatus`, `AlertType`, `AlertSeverity`, `ObservationSource`, `AuditAction`, etc.
### JSONB Columns
**What is JSONB?** PostgreSQL can store JSON data in two column types: `json` (stored as text) and `jsonb` (stored as binary, pre-parsed JSON). JSONB is faster to query and can be indexed. It's useful for data that has a flexible or evolving structure — you don't need to define rigid columns for every possible field.
JSONB is a good fit for data that doesn't need relational querying (JOINs, foreign keys) but benefits from being stored alongside the row:
```csharp
// Audit log — before/after snapshots as JSON
builder.Property(a => a.PreviousValueJson).HasColumnName("previous_value_json")
.HasColumnType("jsonb");
builder.Property(a => a.NewValueJson).HasColumnName("new_value_json")
.HasColumnType("jsonb");
// SOFA score — staleness tracking per organ
builder.Property(s => s.StalenessFlags).HasColumnName("staleness_flags")
.HasColumnType("jsonb");
// Alert explanation — structured clinical reasoning
builder.Property(a => a.Explanation)
.HasColumnName("explanation")
.HasColumnType("jsonb")
.HasConversion(
v => v == null ? null : JsonSerializer.Serialize(v, JsonOptions),
v => string.IsNullOrEmpty(v) ? null : JsonSerializer.Deserialize<AlertExplanation>(v, JsonOptions)!);
```
JSONB is stored as binary JSON in PostgreSQL — it's indexed for fast access and supports `@>` containment queries, though this project reads it as opaque blobs in most cases.
---
## Check Constraints
**What is a check constraint?** A check constraint is a rule the database enforces on every INSERT and UPDATE. If the data violates the rule, the database rejects the operation with an error. This is a safety net — even if there's a bug in your application code, the database won't allow invalid data.
Think of it as a bouncer at the door: "you can only enter if your severity is 'WARNING' or 'CRITICAL' — anything else gets rejected."
```csharp
builder.ToTable("observations", t =>
{
t.HasCheckConstraint("chk_observations_source",
"source IN ('MANUAL', 'DEVICE', 'LAB')");
});
builder.ToTable("clinical_alerts", t =>
{
t.HasCheckConstraint("chk_clinical_alerts_severity",
"severity IN ('WARNING', 'CRITICAL')");
t.HasCheckConstraint("chk_clinical_alerts_status",
"status IN ('OPEN', 'ACKNOWLEDGED', 'RESOLVED', 'ESCALATED')");
t.HasCheckConstraint("chk_clinical_alerts_alert_type",
"alert_type IN ('SEPSIS_WARNING', 'CRITICAL_HEART_RATE', ...)");
});
builder.ToTable("sepsis_bundles", t =>
{
t.HasCheckConstraint("chk_sepsis_bundles_compliance_status",
"compliance_status IN ('IN_PROGRESS', 'COMPLIANT', 'NON_COMPLIANT')");
});
```
These constraints are created by EF Core migrations and enforced by PostgreSQL on every INSERT and UPDATE.
---
## Indexing Strategies
**What is a database index?** An index is a data structure that speeds up lookups, like the index at the back of a textbook. Without an index, the database has to scan every row in the table to find what you're looking for (a "full table scan"). With an index, it can jump directly to the matching rows. The tradeoff: indexes use extra disk space and slow down writes slightly (the index needs updating too).
**What is a unique index?** A unique index does two things: it speeds up lookups AND it prevents duplicate values. If you try to insert a row that would create a duplicate, the database rejects it with an error.
### Unique Indexes
Simple uniqueness constraints:
```csharp
// One MRN per patient
builder.HasIndex(p => p.Mrn).IsUnique();
// One external identifier per resource type + system + value
builder.HasIndex(e => new { e.ResourceType, e.System, e.Value }).IsUnique();
```
### Partial Unique Indexes
**What is a partial index?** A standard index covers every row in the table. A partial index only covers rows that match a filter condition (the `HasFilter()` clause). This is a PostgreSQL-specific feature (not available in all databases) and is useful when you only need uniqueness or fast lookups on a subset of rows:
```csharp
// Only non-null idempotency keys are deduplicated.
// Devices that don't send a key are not subject to deduplication.
builder.HasIndex(o => o.IdempotencyKey)
.IsUnique()
.HasFilter("idempotency_key IS NOT NULL");
// Only one gateway-synced alert per client_alert_id
builder.HasIndex(a => a.ClientAlertId)
.IsUnique()
.HasFilter("client_alert_id IS NOT NULL");
// Efficient baseline lookup for SOFA scoring
builder.HasIndex(s => s.EncounterId)
.HasFilter("is_baseline = true")
.HasDatabaseName("idx_sofa_scores_baseline");
```
The observation idempotency index is critical for safety — it prevents duplicate observations from device retries without requiring every observation to have an idempotency key.
### Composite Indexes
A **composite index** covers multiple columns together. This is useful when your queries always filter on a combination of columns. The index on `(EncounterId, ObservationCode, RecordedAt)` speeds up queries like "get all heart rate observations for encounter X, sorted by time":
```csharp
// Observations queried by encounter + code + time
builder.HasIndex(o => new { o.EncounterId, o.ObservationCode, o.RecordedAt });
// Alerts queried by encounter + time, patient + time
builder.HasIndex(a => new { a.EncounterId, a.TriggeredAt });
builder.HasIndex(a => new { a.PatientId, a.TriggeredAt });
// Open alerts by severity (for the unacknowledged collector)
builder.HasIndex(a => new { a.Severity, a.TriggeredAt })
.HasFilter("status = 'OPEN'");
// Active alerts by encounter + type (for duplicate prevention)
builder.HasIndex(a => new { a.EncounterId, a.AlertType, a.ObservationCode })
.HasFilter("status IN ('OPEN', 'ESCALATED')");
```
### Append-Only Tables
Audit logs are never updated or deleted — only inserted:
```csharp
// clinical_audit_logs — indexes for querying, no update patterns
builder.HasIndex(a => a.EntityType);
builder.HasIndex(a => a.EntityId);
builder.HasIndex(a => a.UserId);
builder.HasIndex(a => a.CreatedAt);
```
---
## PHI Encryption with Value Converters
**What is PHI?** Protected Health Information — any data that could identify a patient (names, dates of birth, medical records). Healthcare regulations require PHI to be encrypted "at rest" (when stored on disk).
EF Core value converters can automatically encrypt data when writing to the database and decrypt when reading. Your application code works with plaintext strings as usual — the encryption is invisible:
```csharp
public static class PatientPhiConverterConfigurator
{
public static void ConfigurePhiConverters(
this ModelBuilder modelBuilder, IPhiEncryptionService crypto)
{
var entity = modelBuilder.Entity<Patient>();
entity.Property(p => p.FirstName)
.HasColumnType("text")
.HasConversion(
v => crypto.Encrypt(v), // encrypt on write
v => crypto.Decrypt(v)); // decrypt on read
entity.Property(p => p.LastName)
.HasColumnType("text")
.HasConversion(
v => crypto.Encrypt(v),
v => crypto.Decrypt(v));
// DateOfBirth stored as encrypted ISO string
entity.Property(p => p.DateOfBirth)
.HasConversion(
v => crypto.Encrypt(v.ToString("yyyy-MM-dd")),
v => DateOnly.Parse(crypto.Decrypt(v)));
}
}
```
This is transparent to application code — `patient.FirstName` always returns the plaintext value. The database stores ciphertext. The `NameSearchToken` column (HMAC-based) enables searching encrypted names without decrypting every row.
---
## Relationship Configuration
All relationships use `DeleteBehavior.Restrict` to prevent accidental cascade deletes in a patient safety system:
```csharp
builder.HasOne(o => o.Encounter)
.WithMany(e => e.Observations)
.HasForeignKey(o => o.EncounterId)
.OnDelete(DeleteBehavior.Restrict);
builder.HasOne(b => b.Encounter)
.WithMany()
.HasForeignKey(b => b.EncounterId)
.OnDelete(DeleteBehavior.Restrict);
builder.HasOne(b => b.TriggeringAlert)
.WithMany()
.HasForeignKey(b => b.TriggeringAlertId)
.OnDelete(DeleteBehavior.Restrict);
```
---
## Transactions and Atomicity
**What is a transaction?** A database transaction groups multiple operations into one atomic unit — either ALL of them succeed, or NONE of them do. If anything fails midway (power outage, constraint violation, application crash), the database rolls back all changes as if nothing happened. This is critical when you need to ensure consistency — for example, you never want an alert to be created without its corresponding observation.
### Outbox Pattern Transaction
The observation ingest writes the observation, the outbox event, and optionally a critical alert in one transaction:
```csharp
await using var tx = await _db.Database.BeginTransactionAsync();
_db.Observations.Add(observation);
_db.OutboxEvents.Add(BuildOutboxEvent("observation.recorded", ...));
if (IsCriticalBreach(req.Value, threshold))
{
_db.ClinicalAlerts.Add(alert);
_db.OutboxEvents.Add(BuildOutboxEvent("alert.generated", ...));
}
await _db.SaveChangesAsync();
await tx.CommitAsync();
```
If anything fails, the entire batch rolls back — no orphaned alerts or lost observations.
### FOR UPDATE SKIP LOCKED
**What is row locking?** When two processes try to update the same database row simultaneously, you get a race condition. PostgreSQL's `FOR UPDATE` clause locks the selected rows so no other transaction can modify them until you're done. `SKIP LOCKED` is an additional modifier that says "if another transaction already locked some rows, skip those instead of waiting." This lets multiple relay instances run in parallel without blocking each other.
The outbox relay uses this raw SQL for concurrent-safe row locking:
```csharp
var events = await db.OutboxEvents
.FromSqlRaw("""
SELECT id, topic, payload, partition_key, created_at, processed_at,
retry_count, last_error, failed_at
FROM outbox_events
WHERE processed_at IS NULL AND failed_at IS NULL
ORDER BY created_at ASC
LIMIT {0}
FOR UPDATE SKIP LOCKED
""", _options.OutboxBatchSize)
.ToListAsync(ct);
```
`FOR UPDATE SKIP LOCKED` means: lock these rows, but if another relay instance already locked some of them, skip those instead of blocking. Both instances make progress without duplicating work.
### Conditional INSERT with NOT EXISTS
Sometimes you need to insert a row only if a certain condition is true. A naive approach (check first, then insert) has a race condition — between your check and your insert, another process might insert the same row. The `INSERT ... SELECT ... WHERE NOT EXISTS` pattern does both in one atomic SQL statement:
Alert creation uses this pattern to prevent duplicate alerts:
```csharp
var affected = await db.Database.ExecuteSqlInterpolatedAsync($"""
INSERT INTO clinical_alerts
(id, encounter_id, patient_id, alert_type, severity, details, status, triggered_at)
SELECT {alertId}, {encounterId}, {patientId},
'QSOFA_SCREEN', 'WARNING', {details}, 'OPEN', {triggeredAt}
WHERE NOT EXISTS (
SELECT 1 FROM clinical_alerts
WHERE encounter_id = {encounterId}
AND alert_type = 'QSOFA_SCREEN'
AND status IN ('OPEN', 'ESCALATED')
)
""", ct);
```
This prevents duplicate alerts without application-level locking — PostgreSQL guarantees atomicity of the INSERT...SELECT.
---
## Migrations
**What is a migration?** A migration is a versioned change to your database schema — like adding a column, creating a table, or adding an index. EF Core compares your current C# entity configurations to the last known database state and generates a migration file with the differences (the "up" to apply the change and the "down" to reverse it). Migrations are committed to version control so every developer and deployment environment applies the same schema changes in the same order.
The project has 40+ migrations tracked chronologically. Key commands:
```bash
# Create a new migration
dotnet ef migrations add AddNewFeature \
--project VigilCareClinicalAPI \
--startup-project VigilCareClinicalAPI
# Apply pending migrations
dotnet ef database update \
--project VigilCareClinicalAPI \
--startup-project VigilCareClinicalAPI
# Generate SQL script (for production deployments)
dotnet ef migrations script \
--project VigilCareClinicalAPI \
--startup-project VigilCareClinicalAPI
```
Migrations are applied automatically on startup during development (via `db.Database.Migrate()` in the seeder path).
---
## Default Values and Sentinels
PostgreSQL can generate default values for columns when you don't provide one in the INSERT. But EF Core needs to know when you _intentionally_ left a property unset vs when you set it to a value that happens to match the default.
```csharp
builder.Property(a => a.Status)
.HasDefaultValueSql("'OPEN'")
.HasSentinel((AlertStatus)(-1));
builder.Property(o => o.Source)
.HasDefaultValueSql("'MANUAL'")
.HasSentinel((ObservationSource)(-1));
```
**What is a sentinel value?** A sentinel is a special "marker" value that means "not set." `HasSentinel((AlertStatus)(-1))` tells EF Core: "if the C# property equals -1, that means it was never explicitly set, so let PostgreSQL provide the default ('OPEN')." This is needed because the C# enum default (0) might be a valid, meaningful enum value that you actually want to store — without a sentinel, EF Core couldn't distinguish between "I set this to 0" and "I forgot to set this."
---
## Entity Summary
| Entity | Table | Key Features |
|--------|-------|-------------|
| Patient | `patients` | PHI encryption, MRN unique index, name search token |
| Encounter | `encounters` | Status machine, active encounter uniqueness |
| Observation | `observations` | Partial unique index on idempotency_key, append-only |
| ClinicalAlert | `clinical_alerts` | Check constraints, 4 partial indexes, JSONB explanation |
| OutboxEvent | `outbox_events` | FOR UPDATE SKIP LOCKED, retry tracking |
| ClinicalAuditLog | `clinical_audit_logs` | Append-only, JSONB before/after snapshots |
| News2Score | `news2_scores` | 7 parameter scores, risk level |
| GcsScore | `gcs_scores` | 3 components, classification |
| SofaScore | `sofa_scores` | 6 organ scores, baseline flag, partial index |
| QsofaEvaluation | `qsofa_evaluations` | 3 criteria values, alert-fired flag |
| SepsisBundle | `sepsis_bundles` | Compliance status check constraint, deadline tracking |
| ExternalResourceIdentifier | `external_resource_identifiers` | Composite unique index for FHIR mapping |
docs/guides/05-redis-clinical-state-store.md
+569
View File
@@ -0,0 +1,569 @@
# Guide 5: Redis as Clinical State Store
## What is Redis?
**Redis** is an in-memory key-value store. Unlike a traditional database (PostgreSQL) that stores data on disk, Redis keeps everything in RAM (your computer's working memory). This makes it extremely fast — reads and writes typically complete in under 1 millisecond.
The simplest way to think of Redis is as a giant dictionary (or hash map): you store values under string keys, and you can retrieve them by key almost instantly. For example, `SET "threshold:HEART_RATE" "{...json...}"` stores a value, and `GET "threshold:HEART_RATE"` retrieves it.
**TTL (Time-To-Live)** is one of Redis's most powerful features. When you store a value, you can say "automatically delete this after 30 minutes." The value disappears on its own — no cleanup code needed. This makes Redis ideal for temporary state that should expire naturally.
**What Redis is NOT**: Redis is not a replacement for PostgreSQL. It doesn't support complex queries, JOINs, or transactions across multiple keys (in the way SQL databases do). Data in Redis can be lost if the server restarts (though it can be configured to persist). Use PostgreSQL as your source of truth; use Redis for fast temporary state and caching.
---
## Why Redis in This Project?
Clinical scoring engines (NEWS2, GCS, qSOFA, SOFA, trend detection) need to aggregate multiple observations arriving at different times. A patient's respiratory rate arrives at 14:01, their heart rate at 14:03, blood pressure at 14:05. To compute a NEWS2 score, you need all 7 parameters within a time window. Redis holds this temporary state in memory with automatic TTL expiration, so stale parameters don't produce false scores.
Redis also caches alert thresholds (avoiding a database query on every observation ingest) and manages alert suppression windows (preventing repeated warnings after acknowledgment).
---
## Architecture Overview
```
Observation arrives
│
┌────────────────┼────────────────┐
▼ ▼ ▼
┌─────────────┐ ┌──────────┐ ┌──────────────┐
│ Threshold │ │ qSOFA │ │ NEWS2 │
│ Cache │ │ Criteria│ │ Parameters │
│ │ │ │ │ │
│ threshold: │ │ qsofa: │ │ news2: │
│ {code} │ │ {enc}: │ │ {enc}: │
│ │ │ {code} │ │ {code} │
│ No TTL │ │ 30min TTL│ │ 4hr TTL │
└─────────────┘ └──────────┘ └──────────────┘
┌────────────────┼────────────────┐
▼ ▼ ▼
┌─────────────┐ ┌──────────┐ ┌──────────────┐
│ GCS │ │ SOFA │ │ Trend │
│ Components │ │ Lab │ │ History │
│ │ │ Cache │ │ │
│ gcs: │ │ sofa: │ │ trend: │
│ {enc}: │ │ {enc}: │ │ {enc}: │
│ {comp} │ │ {code} │ │ {code} │
│ No TTL │ │ 24hr TTL │ │ 2hr TTL │
└─────────────┘ └──────────┘ └──────────────┘
┌──────────────┐
│ Alert │
│ Suppression │
│ │
│ suppress: │
│ {enc}: │
│ {type} │
│ 30min TTL │
└──────────────┘
```
---
## Connection Setup
### Program.cs Registration
```csharp
builder.Services.AddSingleton<IConnectionMultiplexer>(sp =>
ConnectionMultiplexer.Connect(
sp.GetRequiredService<IConfiguration>()["Redis:ConnectionString"]!));
```
**What is `IConnectionMultiplexer`?** This is the StackExchange.Redis library's connection class. "Multiplexer" means it manages multiple connections to Redis internally and can handle many concurrent operations over a small number of physical TCP connections.
It's registered as a **singleton** — meaning one instance is created and shared across the entire application. This is the correct approach because `ConnectionMultiplexer` is thread-safe and designed to be reused. Creating a new connection for every operation would be wasteful and slow.
### Configuration
```json
{
"Redis": {
"ConnectionString": "localhost:6382"
}
}
```
### Health Check
```csharp
public sealed class RedisHealthCheck : IHealthCheck
{
private readonly IConnectionMultiplexer _redis;
public async Task<HealthCheckResult> CheckHealthAsync(
HealthCheckContext context, CancellationToken cancellationToken = default)
{
var db = _redis.GetDatabase();
var latency = await db.PingAsync();
var data = new Dictionary<string, object> { ["ping_ms"] = latency.TotalMilliseconds };
return HealthCheckResult.Healthy(data: data);
}
}
```
Used by the `/health/ready` endpoint to verify Redis is reachable.
---
## Key Naming Convention
All Redis keys follow the pattern: `{domain}:{encounterId}:{discriminator}`
| Domain | Key Pattern | Example |
|--------|-------------|---------|
| Threshold cache | `threshold:{observationCode}` | `threshold:HEART_RATE` |
| qSOFA criteria | `qsofa:{encounterId}:{code}` | `qsofa:3fa85f64-...:RESP_RATE` |
| NEWS2 parameters | `news2:{encounterId}:{code}` | `news2:3fa85f64-...:SPO2` |
| GCS components | `gcs:{encounterId}:{component}` | `gcs:3fa85f64-...:GCS_EYE` |
| SOFA lab values | `sofa:{encounterId}:{code}` | `sofa:3fa85f64-...:PLATELET_K_UL` |
| Trend history | `trend:{encounterId}:{code}` | `trend:3fa85f64-...:HEART_RATE` |
| Alert suppression | `suppress:{encounterId}:{alertType}` | `suppress:3fa85f64-...:WARNING_HEART_RATE` |
Encounter-scoped keys ensure different patients' data never collides — two patients can both have a `qsofa:...:RESP_RATE` key without interfering because the encounter IDs in the middle are different. The `:` separator is a Redis convention for logical namespacing (like folders in a file path). It has no special meaning to Redis itself, but tools like Redis Commander display colon-separated keys as a tree structure.
---
## Usage Pattern 1: Threshold Cache (Pre-Loading)
**What is caching?** Caching means storing a copy of frequently-accessed data in a faster location. Instead of querying PostgreSQL every time an observation arrives (which involves network round-trips and disk I/O), we load threshold values into Redis once at startup. Redis serves the data from memory in under 1ms, compared to 5-10ms for a PostgreSQL query. When you're processing hundreds of observations per second, this difference adds up.
Alert thresholds are loaded from PostgreSQL into Redis on application startup, so observation ingest doesn't need a database query for every threshold lookup.
### ThresholdCacheLoader — Startup Pre-Load
```csharp
public class ThresholdCacheLoader : IHostedService
{
public async Task StartAsync(CancellationToken cancellationToken)
{
var thresholds = await db.AlertThresholds.ToListAsync(cancellationToken);
const int maxAttempts = 3;
int[] backoffMs = [2000, 4000, 8000];
for (var attempt = 0; attempt < maxAttempts; attempt++)
{
try
{
var cache = _redis.GetDatabase();
var batch = cache.CreateBatch();
foreach (var t in thresholds)
{
var json = JsonSerializer.Serialize(new
{
t.ObservationCode,
t.CriticalLow, t.WarningLow,
t.WarningHigh, t.CriticalHigh
});
_ = batch.StringSetAsync($"threshold:{t.ObservationCode}", json);
}
batch.Execute();
_logger.LogInformation("Loaded {Count} alert thresholds into Redis cache",
thresholds.Count);
return;
}
catch (RedisException ex)
{
_logger.LogWarning(ex,
"Redis unavailable during threshold cache load — attempt {Attempt}/{Max}",
attempt + 1, maxAttempts);
if (attempt < maxAttempts - 1)
await Task.Delay(backoffMs[attempt], cancellationToken);
}
}
_logger.LogError(
"Failed to load thresholds into Redis after {Max} attempts — " +
"application will start without cache; observation ingest falls back to PostgreSQL",
maxAttempts);
}
}
```
Key patterns:
- **`CreateBatch()`** groups all the SET commands and sends them to Redis in one network round trip instead of 15+ separate calls. This is called **pipelining** — it dramatically reduces latency when you need to do many operations at once.
- **Retry with exponential backoff** (2s, 4s, 8s) — Redis might still be starting up in Docker when the application starts. Instead of failing immediately, we retry with increasing delays (2 seconds, then 4, then 8). This pattern is called "exponential backoff" and is a standard approach for handling transient failures.
- **Graceful degradation** — if Redis is unreachable after 3 attempts, the app starts anyway and falls back to PostgreSQL for threshold lookups. The app works, just slightly slower.
- **No TTL** — thresholds rarely change, so they stay in cache indefinitely. The cache is updated when an admin modifies thresholds via the API.
---
## Usage Pattern 2: qSOFA Criteria (Sliding Window)
**What is a sliding window?** A sliding window is a time-based boundary that moves forward continuously. Imagine a 30-minute window — at 2:00 PM it covers 1:30–2:00, at 2:05 it covers 1:35–2:05, at 2:10 it covers 1:40–2:10. Only data within the current window counts. Redis TTLs implement this naturally: when you store a value with a 30-minute TTL, it automatically disappears after 30 minutes. If a new observation arrives, you overwrite the key with a fresh 30-minute TTL, effectively "sliding" the window forward.
qSOFA evaluates 3 criteria (respiratory rate >= 22, systolic BP <= 100, altered mentation). Each criterion has a 30-minute TTL — if no new observation arrives within 30 minutes, the criterion expires automatically.
### QsofaCalculator — Key Generation
```csharp
public static class QsofaCalculator
{
public static readonly IReadOnlyList<string> QsofaCodes = new[]
{ "RESP_RATE", "SYSTOLIC_BP", "AVPU" };
public static string CriterionKey(Guid encounterId, string code) =>
$"qsofa:{encounterId}:{code}";
public static RedisKey[] AllCriterionKeys(Guid encounterId) =>
QsofaCodes.Select(c => (RedisKey)CriterionKey(encounterId, c)).ToArray();
public static bool MeetsCriterion(string observationCode, decimal value) =>
observationCode switch
{
"RESP_RATE" => value >= 22m,
"SYSTOLIC_BP" => value <= 100m,
"AVPU" => value >= 1m,
_ => false
};
}
```
### QsofaDetector — Set/Delete Pattern
```csharp
public async Task<QsofaResult> ProcessObservationAsync(
Guid encounterId, Guid patientId,
string observationCode, decimal value, CancellationToken ct)
{
var cache = _redis.GetDatabase();
var key = QsofaCalculator.CriterionKey(encounterId, observationCode);
if (QsofaCalculator.MeetsCriterion(observationCode, value))
{
// Criterion met — set with 30-minute TTL
await cache.StringSetAsync(
key, value.ToString(), TimeSpan.FromSeconds(1800));
}
else
{
// Criterion not met — delete immediately
await cache.KeyDeleteAsync(key);
}
return await EvaluateAndMaybeAlertAsync(encounterId, patientId, ct);
}
```
### Evaluation — MGET for Batch Read
**What is MGET?** MGET ("multi-get") is a Redis command that retrieves multiple keys in a single network round trip. Instead of calling `GET key1`, then `GET key2`, then `GET key3` (3 round trips), `MGET key1 key2 key3` returns all three values at once. In the StackExchange.Redis library, calling `StringGetAsync` with an array of keys automatically uses MGET under the hood.
```csharp
private async Task<QsofaResult> EvaluateAndMaybeAlertAsync(
Guid encounterId, Guid patientId, CancellationToken ct)
{
var cache = _redis.GetDatabase();
var allKeys = QsofaCalculator.AllCriterionKeys(encounterId);
// MGET reads all 3 criterion values in one round trip
var values = await cache.StringGetAsync(allKeys);
var activeCount = QsofaCalculator.CountActiveCriteria(values);
if (activeCount >= 2)
await TryCreateScreenAlertAsync(encounterId, patientId, activeCount, values, ct);
// ...
}
```
The TTL-based sliding window means: if a patient's respiratory rate was >= 22 at 14:00 but no new reading arrives by 14:30, the criterion expires and the qSOFA score drops — even without an explicit "normal" reading.
---
## Usage Pattern 3: NEWS2 Parameters (7-Parameter Aggregation)
NEWS2 needs all 7 parameters to compute a score. Each parameter is cached independently with a 4-hour TTL:
```csharp
// Store a parameter value with its individual score
var paramData = JsonSerializer.Serialize(new
{
value,
score = individualScore,
recordedAt = DateTimeOffset.UtcNow
});
await cache.StringSetAsync(
News2Calculator.ParameterKey(encounterId, observationCode),
paramData,
TimeSpan.FromSeconds(14400)); // 4 hours
```
Evaluation reads all 7 with a single MGET:
```csharp
var allKeys = News2Calculator.AllParameterKeys(encounterId);
var allValues = await cache.StringGetAsync(allKeys);
// Check if all 7 parameters are present
for (int i = 0; i < 7; i++)
{
if (!allValues[i].HasValue)
{
return News2Result.IncompleteParameters(allValues.Count(v => v.HasValue));
}
var cached = JsonSerializer.Deserialize<News2CachedParam>(allValues[i]!);
scores[i] = cached?.Score;
}
var totalScore = scores.Select(s => s!.Value).Sum();
```
The 4-hour TTL is much longer than qSOFA's 30 minutes because NEWS2 parameters (like temperature) may only be measured every few hours.
### Consciousness Resolution: GCS-First, AVPU-Fallback
The NEWS2 consciousness parameter prefers GCS over AVPU:
```csharp
private async Task<int?> ResolveConsciousnessScoreAsync(Guid encounterId)
{
var cache = _redis.GetDatabase();
// Try GCS first — read all 3 component values
var gcsValues = await cache.StringGetAsync(GcsCalculator.AllComponentKeys(encounterId));
if (gcsValues.All(v => v.HasValue))
{
var eye = decimal.Parse(gcsValues[0]!);
var verbal = decimal.Parse(gcsValues[1]!);
var motor = decimal.Parse(gcsValues[2]!);
var total = GcsCalculator.ComputeTotal(eye, verbal, motor)!.Value;
return News2Calculator.ScoreConsciousnessFromGcs(total);
}
// Fallback to AVPU
var avpuVal = await cache.StringGetAsync(
News2Calculator.ParameterKey(encounterId, "AVPU"));
if (!avpuVal.HasValue) return null;
var cached = JsonSerializer.Deserialize<News2CachedParam>(avpuVal!);
return cached?.Score;
}
```
---
## Usage Pattern 4: GCS Components (Temporary Assembly)
GCS has 3 components (Eye, Verbal, Motor) that may arrive as separate observations. Redis holds each component until all 3 are present:
```csharp
public static class GcsCalculator
{
public static readonly IReadOnlyList<string> ComponentCodes = new[]
{ "GCS_EYE", "GCS_VERBAL", "GCS_MOTOR" };
public static string ComponentKey(Guid encounterId, string code) =>
$"gcs:{encounterId}:{code}";
public static RedisKey[] AllComponentKeys(Guid encounterId) =>
ComponentCodes
.Select(code => (RedisKey)$"gcs:{encounterId}:{code}")
.ToArray();
public static int? ComputeTotal(decimal? eye, decimal? verbal, decimal? motor)
{
if (eye is null || verbal is null || motor is null) return null;
return (int)(eye.Value + verbal.Value + motor.Value);
}
}
```
GCS component keys have no explicit TTL — they persist until replaced by a new assessment. When all 3 components are present, the total is computed, persisted to PostgreSQL, and published to the `gcs.scored` Kafka topic for downstream consumers (SOFA CNS, NEWS2 consciousness, qSOFA altered mentation).
---
## Usage Pattern 5: SOFA Lab Cache (Staleness Tracking)
SOFA scoring depends on lab values (platelets, bilirubin, creatinine) that arrive infrequently. The cache stores both the value and when it was recorded:
```csharp
public class SofaLabCache
{
public async Task StoreAsync(
Guid encounterId, string code, decimal value, DateTimeOffset recordedAt)
{
var json = JsonSerializer.Serialize(new SofaCachedValue(value, recordedAt));
var key = SofaCalculator.CacheKey(encounterId, code);
await _redis.GetDatabase().StringSetAsync(
key, json, TimeSpan.FromHours(_options.LabStalenessHours)); // 24 hours
}
public SofaValueStatus Classify(SofaCachedValue? value)
{
if (value is null) return SofaValueStatus.Expired;
var age = DateTimeOffset.UtcNow - value.RecordedAt;
if (age.TotalHours > _options.LabStalenessHours) return SofaValueStatus.Expired; // 24h
if (age.TotalHours > _options.LabWarningHours) return SofaValueStatus.Stale; // 12h
return SofaValueStatus.Current;
}
public async Task<Dictionary<string, SofaCachedValue>> GetAllAsync(Guid encounterId)
{
var cache = _redis.GetDatabase();
var keys = SofaCalculator.SofaObservationCodes
.Select(c => (RedisKey)SofaCalculator.CacheKey(encounterId, c))
.ToArray();
var values = await cache.StringGetAsync(keys); // MGET for all codes at once
// ... deserialize non-null values into dictionary
}
}
```
Three-tier staleness classification:
- **Current** (< 12 hours): value used as-is
- **Stale** (12–24 hours): value carried forward but flagged in SOFA score `staleness_flags` JSONB
- **Expired** (> 24 hours): organ score omitted from calculation
---
## Usage Pattern 6: Trend History (Sliding Window with JSON Lists)
Trend detection stores a history list of recent values in a single Redis key, serialized as JSON:
```csharp
public async Task<TrendResult> ProcessObservationAsync(
Guid encounterId, Guid patientId,
string observationCode, decimal value, DateTimeOffset recordedAt,
CancellationToken ct)
{
var cache = _redis.GetDatabase();
var key = TrendCalculator.HistoryKey(encounterId, observationCode);
// Read existing history
var historyJson = await cache.StringGetAsync(key);
var history = historyJson.HasValue
? JsonSerializer.Deserialize<List<TrendHistoryEntry>>(historyJson!) ?? new()
: new List<TrendHistoryEntry>();
// Append new entry
history.Add(new TrendHistoryEntry(value, recordedAt));
// Trim: remove entries outside window, keep max N entries
var cutoff = recordedAt.AddMinutes(-_options.WindowMinutes);
history = history
.Where(e => e.RecordedAt >= cutoff)
.TakeLast(_options.MaxHistoryEntries)
.ToList();
// Write back with TTL
await cache.StringSetAsync(
key,
JsonSerializer.Serialize(history),
TimeSpan.FromSeconds(_options.HistoryTtlSeconds)); // 2 hours
// Compute rate of change
var rate = TrendCalculator.ComputeRatePerMinute(history, _options.WindowMinutes);
// ...
}
```
Configuration from `appsettings.json`:
```json
{
"TrendDetection": {
"WindowMinutes": 30,
"MaxHistoryEntries": 10,
"HistoryTtlSeconds": 7200,
"RateThresholdsPerMinute": {
"HEART_RATE": 0.5,
"RESP_RATE": 0.3,
"SYSTOLIC_BP": 1.0,
"TEMP_C": 0.05,
"SPO2": 0.2
}
}
}
```
---
## Usage Pattern 7: Alert Suppression (TTL-Based Windows)
When a clinician acknowledges a WARNING-level alert, a suppression key prevents the same alert type from firing again for a configurable period:
```csharp
public class AlertSuppressionService : IAlertSuppressionService
{
public static string SuppressionKey(Guid encounterId, AlertType alertType) =>
$"suppress:{encounterId}:{alertType.ToDbString()}";
public async Task SetSuppressionAsync(
Guid encounterId, AlertType alertType, TimeSpan ttl, CancellationToken ct)
{
var cache = _redis.GetDatabase();
await cache.StringSetAsync(SuppressionKey(encounterId, alertType), "1", ttl);
_metrics.AlertSuppressionsTotal.WithLabels(alertType.ToDbString()).Inc();
}
public async Task<bool> IsSuppressedAsync(
Guid encounterId, AlertType alertType, CancellationToken ct)
{
var cache = _redis.GetDatabase();
return await cache.KeyExistsAsync(SuppressionKey(encounterId, alertType));
}
}
```
Scoring engines check suppression before creating WARNING alerts:
```csharp
if (alertType == AlertType.News2Warning)
{
var suppression = _services.GetRequiredService<IAlertSuppressionService>();
if (await suppression.IsSuppressedAsync(encounterId, alertType, ct))
{
_logger.LogDebug("NEWS2_WARNING suppressed for encounter {Id}", encounterId);
return false;
}
}
```
CRITICAL alerts are never suppressed — they always fire regardless of any suppression keys.
---
## Redis Operations Summary
| Operation | Redis Command | When Used |
|-----------|--------------|-----------|
| `StringSetAsync(key, value, ttl)` | `SET key value EX ttl` | Store parameter/criterion with expiry |
| `StringGetAsync(key)` | `GET key` | Read a single value |
| `StringGetAsync(keys[])` | `MGET key1 key2 ...` | Batch read all criteria/parameters |
| `KeyDeleteAsync(key)` | `DEL key` | Remove criterion when no longer met |
| `KeyExistsAsync(key)` | `EXISTS key` | Check alert suppression |
| `CreateBatch()` + `Execute()` | Pipeline | Bulk threshold cache loading |
| `PingAsync()` | `PING` | Health check |
---
## TTL Summary
| Domain | TTL | Reason |
|--------|-----|--------|
| Thresholds | None | Long-lived, invalidated on API update |
| qSOFA criteria | 30 min | Bedside screen — criteria expire without refresh |
| NEWS2 parameters | 4 hours | Vitals measured every few hours |
| GCS components | None | Persist until replaced by new assessment |
| SOFA labs | 24 hours | Labs can be infrequent; carry forward with staleness tracking |
| Trend history | 2 hours | Only recent velocity matters |
| Alert suppression | 30 min (configurable) | Prevents alert fatigue after acknowledgment |
---
## Key Design Principle: Eventual Consistency Is Acceptable
**What is eventual consistency?** In a distributed system, "consistency" means all components see the same data at the same time. "Eventual consistency" relaxes this — components might temporarily see stale or missing data, but will converge to the correct state eventually. This is a deliberate tradeoff: you accept slightly stale data in exchange for much faster performance.
Redis state is not the source of truth — PostgreSQL is. If Redis loses data (restart, eviction), the worst case is:
- A score computation waits for the next observation to refill the cache
- A suppression window ends early (alert fires sooner than expected)
- Threshold cache falls back to PostgreSQL lookup
No clinical data is lost. Redis is an optimization layer, not a durability layer.
docs/guides/06-apache-kafka-event-streaming.md
+549
View File
@@ -0,0 +1,549 @@
# Guide 6: Apache Kafka Event Streaming
## What is Kafka?
**Apache Kafka** is a distributed event streaming platform. At its simplest, it's a highly reliable message bus: producers send messages to Kafka, and consumers read them. But unlike a simple message queue, Kafka stores messages durably (on disk) and lets multiple independent consumers each read the same messages at their own pace.
Key concepts:
- **Topic**: A named channel for messages, like a mailbox or category. For example, `observation.recorded` is a topic for observation events. Producers publish to topics; consumers subscribe to topics.
- **Partition**: Each topic is divided into partitions (like lanes on a highway). Partitions allow parallelism — multiple consumers can read from different partitions simultaneously. Messages with the same **partition key** (like an encounter ID) always land on the same partition, which guarantees ordering for that key.
- **Consumer Group**: A named group of consumers that share the work of reading a topic. Kafka assigns each partition to exactly one consumer in the group, so messages are processed once per group. Different groups process messages independently — if both the "scoring engine" group and the "search indexer" group subscribe to the same topic, each group gets every message.
- **Offset**: A sequential number that identifies each message's position within a partition. Consumers track their offset (how far they've read). If a consumer crashes and restarts, it picks up where it left off.
- **Producer**: Code that sends messages to Kafka topics.
- **Consumer**: Code that reads messages from Kafka topics.
**How is Kafka different from a regular queue (like RabbitMQ)?** In a traditional queue, once a message is consumed, it's gone. In Kafka, messages persist (for days or longer), and multiple consumer groups can independently read the same messages. This makes Kafka ideal for event-driven architectures where one event needs to trigger many independent downstream processes.
---
## Why Kafka in This Project?
When a nurse records a vital sign, the system must simultaneously: evaluate threshold breaches, compute NEWS2/qSOFA/SOFA scores, detect trends, index the observation in Elasticsearch, write it to the data lake, and potentially page a physician. Doing all of this synchronously in the HTTP request would take too long and couple unrelated systems.
Kafka decouples the write path from the processing path. The API writes an event to the outbox, the outbox relay publishes it to Kafka, and 9 independent consumer groups each process it at their own pace. If the trend analyzer is slow, the qSOFA evaluator is unaffected.
---
## Architecture Overview
```
HTTP Request
│
▼
┌──────────────┐ outbox ┌──────────────┐
│ Observation │───────────────│ OutboxRelay │
│ Service │ (PostgreSQL) │ Service │
└──────────────┘ └──────┬───────┘
│ Kafka Producer
▼
┌───────────────────────┐
│ Kafka Broker (KRaft) │
│ │
│ observation.recorded │ ← 6 partitions
│ alert.generated │ ← 6 partitions
│ encounter.status.* │ ← 6 partitions
│ gcs.scored │ ← 6 partitions
│ sepsis.bundle.* │ ← 6 partitions
└───────┬───────────────┘
│
┌───────────────────┼───────────────────┐
│ │ │
┌─────────┴──────┐ ┌────────┴───────┐ ┌───────┴────────┐
│ es-indexer │ │ sepsis-engine │ │ news2-scoring │
│ (CQRS proj.) │ │ (qSOFA eval) │ │ (7-param agg) │
├─────────────────┤ ├────────────────┤ ├────────────────┤
│ warning-eval. │ │ gcs-scoring │ │ sofa-scoring │
├─────────────────┤ ├────────────────┤ ├────────────────┤
│ trend-analyzer │ │ notification- │ │ data-lake- │
│ (rate-of-change)│ │ publisher │ │ writer │
└─────────────────┘ └────────────────┘ └────────────────┘
```
---
## Kafka Configuration
### Docker Compose — KRaft Mode (No Zookeeper)
```yaml
kafka:
image: apache/kafka:3.7.0
environment:
KAFKA_NODE_ID: 1
KAFKA_PROCESS_ROLES: broker,controller
KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
KAFKA_AUTO_CREATE_TOPICS_ENABLE: "false"
CLUSTER_ID: "MkU3OEVBNTcwNTJENDM2Qk"
```
Key decisions:
- **KRaft mode** (`KAFKA_PROCESS_ROLES: broker,controller`): Kafka 3.7 runs its own metadata consensus without Zookeeper, eliminating an entire service
- **Auto-creation disabled**: Topics are provisioned explicitly with 6 partitions. Auto-creation would silently create single-partition topics if the relay publishes before the provisioner runs
- **Fixed CLUSTER_ID**: Prevents storage reinitialization on container restart
### Application Configuration
```csharp
public class KafkaOptions
{
public const string Section = "Kafka";
public string BootstrapServers { get; set; } = null!;
public KafkaTopicOptions Topics { get; set; } = null!;
public int NumPartitions { get; set; } = 6;
public short ReplicationFactor { get; set; } = 3;
public int OutboxBatchSize { get; set; } = 100;
public int OutboxPollIntervalMs { get; set; } = 500;
public int OutboxMaxRetries { get; set; } = 10;
public int MaxPoisonRetries { get; set; } = 5;
}
public class KafkaTopicOptions
{
public string ObservationRecorded { get; set; } = "observation.recorded";
public string AlertGenerated { get; set; } = "alert.generated";
public string AlertAcknowledged { get; set; } = "alert.acknowledged";
public string EncounterStatusChanged { get; set; } = "encounter.status.changed";
public string SepsisBundleCreated { get; set; } = "sepsis.bundle.created";
public string SepsisBundleUpdated { get; set; } = "sepsis.bundle.updated";
public string GcsScored { get; set; } = "gcs.scored";
}
```
```json
{
"Kafka": {
"BootstrapServers": "localhost:9092",
"Topics": {
"ObservationRecorded": "observation.recorded",
"AlertGenerated": "alert.generated",
"EncounterStatusChanged": "encounter.status.changed",
"GcsScored": "gcs.scored"
},
"NumPartitions": 6,
"OutboxBatchSize": 100,
"OutboxPollIntervalMs": 1000
}
}
```
---
## Topic Provisioning
Topics are created on application startup by `KafkaTopicProvisioner`:
```csharp
public class KafkaTopicProvisioner : IHostedService
{
public async Task StartAsync(CancellationToken cancellationToken)
{
using var admin = new AdminClientBuilder(new AdminClientConfig
{
BootstrapServers = _options.BootstrapServers
}).Build();
var topicNames = new[]
{
_options.Topics.ObservationRecorded,
_options.Topics.AlertGenerated,
_options.Topics.AlertAcknowledged,
_options.Topics.EncounterStatusChanged,
_options.Topics.SepsisBundleCreated,
_options.Topics.SepsisBundleUpdated,
_options.Topics.GcsScored
};
var specs = topicNames.Select(name => new TopicSpecification
{
Name = name,
NumPartitions = _options.NumPartitions, // 6
ReplicationFactor = _options.ReplicationFactor
}).ToList();
try
{
await admin.CreateTopicsAsync(specs);
_logger.LogInformation("Kafka topics provisioned: {Topics}",
string.Join(", ", topicNames));
}
catch (CreateTopicsException ex)
{
// TopicAlreadyExists is not an error — idempotent startup
var errors = ex.Results
.Where(r => r.Error.Code is not (ErrorCode.NoError or ErrorCode.TopicAlreadyExists))
.ToList();
if (errors.Count > 0)
throw new InvalidOperationException(
$"Failed to create Kafka topics: {string.Join(", ", errors.Select(e => e.Error.Reason))}");
_logger.LogInformation("Kafka topics already exist — skipping creation");
}
}
}
```
7 topics, all with 6 partitions. The `TopicAlreadyExists` error is swallowed — the provisioner is idempotent across restarts.
---
## The Outbox Relay — Producing Messages
Messages are never published directly to Kafka from request handlers. Why? Because you'd face an impossible consistency problem: if you save to the database AND publish to Kafka in the same request, one might succeed and the other might fail, leaving your system in an inconsistent state. Instead, this project uses the **outbox pattern** (covered in detail in Guide 10): an `OutboxEvent` row is written to PostgreSQL in the same transaction as the domain entity (guaranteed atomic), and a background service called `OutboxRelayService` reads those rows and publishes them to Kafka separately.
### Idempotent Producer
**What does "idempotent" mean?** An operation is idempotent if doing it multiple times produces the same result as doing it once. An idempotent Kafka producer ensures that if a message is accidentally sent twice (due to a network timeout and retry), Kafka stores it only once:
```csharp
_producer = new ProducerBuilder<string, string>(new ProducerConfig
{
BootstrapServers = _options.BootstrapServers,
Acks = Acks.All,
EnableIdempotence = true,
MessageSendMaxRetries = 3,
RetryBackoffMs = 100
}).Build();
```
- **`Acks.All`**: After the producer sends a message, it waits for confirmation. `All` means the Kafka broker confirms only after all replica copies have stored the message. This is the highest durability guarantee — your message won't be lost even if a broker crashes.
- **`EnableIdempotence = true`**: The broker assigns each producer an internal ID and sequence number. If a network timeout causes the client to retry a message the broker already accepted, the broker recognizes the duplicate by its sequence number and silently drops it.
### Publishing Logic
```csharp
private async Task ProcessBatchAsync(CancellationToken ct)
{
var db = scope.ServiceProvider.GetRequiredService<AppDbContext>();
await using var tx = await db.Database.BeginTransactionAsync(ct);
// Lock rows, skip any already locked by another relay instance
var events = await db.OutboxEvents
.FromSqlRaw("""
SELECT ... FROM outbox_events
WHERE processed_at IS NULL AND failed_at IS NULL
ORDER BY created_at ASC
LIMIT {0}
FOR UPDATE SKIP LOCKED
""", _options.OutboxBatchSize)
.ToListAsync(ct);
foreach (var ev in events)
{
try
{
var result = await _producer!.ProduceAsync(
ev.Topic,
new Message<string, string>
{
Key = ev.PartitionKey ?? string.Empty,
Value = ev.Payload
}, ct);
ev.ProcessedAt = DateTimeOffset.UtcNow;
}
catch (ProduceException<string, string> ex)
{
ev.RetryCount++;
ev.LastError = ex.Error.Reason;
if (ev.RetryCount >= _options.OutboxMaxRetries)
{
ev.FailedAt = DateTimeOffset.UtcNow;
_logger.LogError(ex,
"Outbox event {Id} permanently failed after {Retries} retries",
ev.Id, ev.RetryCount);
}
break; // Stop processing batch on first failure
}
}
await db.SaveChangesAsync(ct);
await tx.CommitAsync(ct);
}
```
Key patterns:
- **Partition key** is the `encounterId` — all events for one encounter land on the same partition, guaranteeing ordering per encounter
- **Retry tracking** — each failed event increments `RetryCount` and records `LastError`. After `OutboxMaxRetries` (10), the event is marked as permanently failed with `FailedAt`
- **Break on failure** — if one event fails to publish, the batch stops. This prevents out-of-order delivery within an encounter.
---
## Consumer Patterns
All consumers follow the same structural pattern with minor variations. Understanding this pattern is important because you'll see it repeated across 9 different services.
### Pattern 1: Simple Consumer (SepsisEngineService)
The simplest pattern — subscribe to one topic, read messages in a loop, process each one, and commit the offset (tell Kafka "I've finished processing this message"):
```csharp
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
var config = new ConsumerConfig
{
BootstrapServers = _kafkaOptions.BootstrapServers,
GroupId = "sepsis-engine",
AutoOffsetReset = AutoOffsetReset.Earliest,
EnableAutoCommit = false
};
using var consumer = new ConsumerBuilder<string, string>(config).Build();
consumer.Subscribe(_kafkaOptions.Topics.ObservationRecorded);
var guard = new PoisonPillGuard("sepsis-engine", _kafkaOptions.MaxPoisonRetries, _logger);
while (!stoppingToken.IsCancellationRequested)
{
ConsumeResult<string, string>? result = null;
try
{
result = consumer.Consume(stoppingToken);
var evt = JsonSerializer.Deserialize<SepsisObservationEvent>(
result.Message.Value)!;
using var scope = _services.CreateScope();
var qsofaDetector = scope.ServiceProvider.GetRequiredService<QsofaDetector>();
await qsofaDetector.ProcessObservationAsync(
evt.EncounterId, evt.PatientId, evt.ObservationCode, evt.Value, stoppingToken);
consumer.Commit(result);
guard.OnSuccess();
}
catch (OperationCanceledException) { break; }
catch (Exception ex)
{
if (result is not null && guard.ShouldSkip(result, ex))
{
consumer.Commit(result);
continue;
}
_logger.LogError(ex, "SepsisEngine failed — will retry");
await Task.Delay(2000, stoppingToken);
}
}
consumer.Close();
}
```
Common configuration across all consumers:
- **`AutoOffsetReset = Earliest`**: When a consumer starts for the first time (no previously committed offset), should it read from the beginning of the topic (`Earliest`) or only new messages (`Latest`)? We use `Earliest` to ensure no events are missed — even if the consumer starts hours after the topic was created.
- **`EnableAutoCommit = false`**: By default, Kafka consumers automatically commit their offset every 5 seconds (telling Kafka "I've processed everything up to here"). But what if the consumer crashes between the auto-commit and actually finishing the work? The message would be marked as processed but never actually handled. Manual commit means we only tell Kafka "done" after we've confirmed the work succeeded. This gives us **at-least-once delivery** — a message might be processed twice (if the consumer crashes after processing but before committing), but it will never be lost.
- **`consumer.Close()`** in the `finally` block: Tells the Kafka broker "I'm leaving the group." This triggers an immediate rebalance so other consumers pick up this consumer's partitions right away, instead of waiting for a session timeout (typically 30 seconds).
### Pattern 2: Multi-Topic Consumer (NotificationPublisherService)
Subscribes to multiple topics and routes based on the topic name:
```csharp
consumer.Subscribe(new[]
{
_kafkaOptions.Topics.AlertGenerated,
_kafkaOptions.Topics.EncounterStatusChanged,
});
// In the consume loop:
if (result.Topic == _kafkaOptions.Topics.AlertGenerated)
await HandleAlertGeneratedAsync(chan, props, result.Message.Value, stoppingToken);
else if (result.Topic == _kafkaOptions.Topics.EncounterStatusChanged)
await HandleEncounterStatusChangedAsync(chan, props, result.Message.Value, stoppingToken);
```
This consumer bridges Kafka → RabbitMQ: critical alerts go to the paging queue, discharged encounters go to the discharge summary queue.
### Pattern 3: Buffered Consumer (DataLakeWriterService)
Buffers events in memory and flushes to MinIO in batches:
```csharp
consumer.Subscribe(new[]
{
_kafkaOptions.Topics.ObservationRecorded,
_kafkaOptions.Topics.AlertGenerated,
_kafkaOptions.Topics.EncounterStatusChanged,
});
// Non-blocking consume with timeout
result = consumer.Consume(TimeSpan.FromMilliseconds(500));
if (result is not null)
AddToBuffer(result);
// Flush when buffer is full or timer expires
var shouldFlushCount = totalBuffered >= _opts.FlushCount; // 1000 events
var shouldFlushTime = elapsed >= TimeSpan.FromSeconds(_opts.FlushIntervalSeconds); // 300s
```
The data lake writer tracks per-partition high watermarks and only commits offsets for partitions where all MinIO uploads succeeded — partial-commit safety.
---
## The Poison Pill Guard
**What is a poison pill?** In messaging systems, a "poison pill" is a message that a consumer cannot process — maybe the JSON is malformed, a required field is missing, or the data violates a business rule. Without protection, the consumer reads the message, fails to process it, doesn't commit the offset, and reads the same message again on the next loop iteration — forever. The consumer is stuck in an infinite retry loop, and all subsequent messages on that partition are blocked behind it.
The `PoisonPillGuard` detects this situation and skips unprocessable messages:
```csharp
public sealed class PoisonPillGuard
{
private static readonly Counter PoisonPillsSkipped = Metrics.CreateCounter(
"kafka_poison_pills_skipped_total",
"Messages skipped as poison pills.",
labelNames: new[] { "consumer_group", "topic" });
public bool ShouldSkip(ConsumeResult<string, string> result, Exception ex)
{
// Permanent errors — skip immediately
if (IsPermanent(ex))
{
LogSkip(result, ex, "permanent");
return true;
}
// Transient errors — retry up to maxRetries
var key = (result.Topic, result.Partition.Value, result.Offset.Value);
if (_lastFailedKey == key)
_retryCount++;
else
{
_lastFailedKey = key;
_retryCount = 1;
}
if (_retryCount >= _maxRetries)
{
LogSkip(result, ex, $"transient after {_retryCount} retries");
return true;
}
return false;
}
public void OnSuccess()
{
_lastFailedKey = null;
_retryCount = 0;
}
private static bool IsPermanent(Exception ex) =>
GetRoot(ex) is JsonException or FormatException or ArgumentNullException;
}
```
Classification:
- **Permanent errors** (JsonException, FormatException, ArgumentNullException): The message payload is malformed — retrying won't help. Skip immediately.
- **Transient errors** (database timeout, Redis unavailable): Retry up to `MaxPoisonRetries` (5). If still failing, skip and move on.
Skipped messages are:
1. Logged at `Critical` level with the full payload (truncated to 2000 chars)
2. Counted in the `kafka_poison_pills_skipped_total` Prometheus metric
3. Committed (offset advances past the poison pill)
---
## Consumer Group Summary
| Group ID | Topics | Purpose | Service |
|----------|--------|---------|---------|
| `es-indexer` | observation.recorded, alert.generated, encounter.status.changed, sepsis.bundle.*, gcs.scored | CQRS projection to Elasticsearch | `EsIndexerService` |
| `sepsis-engine` | observation.recorded | qSOFA screening | `SepsisEngineService` |
| `warning-evaluator` | observation.recorded | Warning-range threshold alerts | `WarningAlertService` |
| `news2-scoring` | observation.recorded | NEWS2 composite score | `News2ScoringService` |
| `gcs-scoring` | observation.recorded | GCS component aggregation | `GcsScoringService` |
| `sofa-scoring` | observation.recorded, gcs.scored | SOFA organ-dysfunction scoring | `SofaScoringService` |
| `trend-analyzer` | observation.recorded | Rate-of-change detection | `TrendAnalyzerService` |
| `notification-publisher` | alert.generated, encounter.status.changed | Kafka → RabbitMQ bridge | `NotificationPublisherService` |
| `data-lake-writer` | observation.recorded, alert.generated, encounter.status.changed | Parquet files to MinIO | `DataLakeWriterService` |
All 9 consumer groups process `observation.recorded` independently. Publishing one observation event triggers 7+ parallel processing paths.
---
## Topic Summary
| Topic | Partition Key | Producers | Consumers |
|-------|---------------|-----------|-----------|
| `observation.recorded` | encounterId | OutboxRelay | es-indexer, sepsis-engine, warning-evaluator, news2-scoring, gcs-scoring, sofa-scoring, trend-analyzer, data-lake-writer |
| `alert.generated` | encounterId | OutboxRelay | es-indexer, notification-publisher, data-lake-writer |
| `alert.acknowledged` | encounterId | OutboxRelay | es-indexer |
| `encounter.status.changed` | encounterId | OutboxRelay | es-indexer, notification-publisher, data-lake-writer |
| `gcs.scored` | encounterId | GcsScoringService (direct) | sofa-scoring, es-indexer |
| `sepsis.bundle.created` | encounterId | OutboxRelay | es-indexer |
| `sepsis.bundle.updated` | encounterId | OutboxRelay | es-indexer |
All topics use `encounterId` as the partition key. This guarantees that all events for a single encounter are processed in order within each consumer group, which is essential for clinical correctness (you can't evaluate qSOFA before the observation that triggered it).
---
## Health Check
```csharp
public sealed class KafkaHealthCheck : IHealthCheck
{
public async Task<HealthCheckResult> CheckHealthAsync(
HealthCheckContext context, CancellationToken cancellationToken)
{
using var admin = new AdminClientBuilder(new AdminClientConfig
{
BootstrapServers = _options.BootstrapServers
}).Build();
var metadata = await Task.Run(
() => admin.GetMetadata(TimeSpan.FromSeconds(5)), cancellationToken);
var data = new Dictionary<string, object> { ["brokers"] = metadata.Brokers.Count };
return HealthCheckResult.Healthy(data: data);
}
}
```
Queries the broker metadata to verify the cluster is reachable and has at least one broker.
---
## Monitoring
### Consumer Lag Collector
The `KafkaConsumerLagCollector` polls offset lag every 30 seconds for 4 key consumer groups:
```csharp
private static readonly string[] Groups =
{
"es-indexer",
"sepsis-engine",
"notification-publisher",
"data-lake-writer",
};
```
For each group:
1. Query committed offsets via the Admin API
2. Query high watermarks via a temporary consumer
3. Lag = high watermark − committed offset (summed across all partitions)
4. Set `kafka_consumer_lag` gauge with the group ID as a label
A rising lag means a consumer is falling behind the event stream — visible instantly on the Grafana dashboard.
---
## Key Design Decisions
### Why Outbox + Relay Instead of Direct Publish?
Imagine your code does this: (1) save observation to database, (2) publish event to Kafka. What happens if step 1 succeeds but step 2 fails (network blip)? You have data in your database that downstream consumers never learn about. What if step 2 succeeds but step 1 fails? Consumers process an event for data that doesn't exist.
The outbox pattern solves this by writing BOTH the domain entity and the outbox event in one PostgreSQL transaction — they either both succeed or both fail. The relay service reads the outbox table separately and publishes to Kafka. If the relay fails, it retries later (the outbox row is still there). The idempotent producer prevents duplicates from retries. This gives you guaranteed eventual delivery with no data loss.
### Why 6 Partitions?
In Kafka, the number of partitions determines the maximum parallelism for a consumer group — one consumer instance can read from one partition. With 6 partitions, you can run up to 6 consumer instances per group. For local development with a single instance, that instance reads all 6 partitions. In production, you could scale to 6 instances per consumer group for horizontal parallelism. The partition key (`encounterId`) distributes encounters evenly across partitions via a hash function.
### Why Manual Commit?
As explained in the consumer configuration section, auto-commit periodically tells Kafka "I've processed everything" — even if you haven't. This means messages can be lost if the consumer crashes at the wrong moment. Manual commit after successful processing guarantees **at-least-once delivery**: a message might be processed twice (rare, only on crash), but it will never be silently lost. For a patient safety system, losing a vital sign observation is unacceptable, so at-least-once is the right tradeoff.
docs/guides/07-rabbitmq-notification-queuing.md
+411
View File
@@ -0,0 +1,411 @@
# Guide 7: RabbitMQ for Notification Queuing
## What is RabbitMQ?
**RabbitMQ** is a message broker — a middleman that accepts, routes, and delivers messages between parts of your application. Think of it like a post office: producers drop off letters (messages), the post office sorts them into the right mailboxes (queues), and consumers pick them up.
Key concepts:
- **Queue**: A named buffer that stores messages until a consumer processes them. Unlike Kafka (where messages persist for all consumer groups), RabbitMQ queues are point-to-point by default — once a consumer acknowledges a message, it's removed from the queue.
- **Exchange**: A routing layer that sits in front of queues. Producers send messages to an exchange (not directly to a queue), and the exchange decides which queue(s) to route each message to based on routing rules.
- **Routing Key**: A string attached to each message that the exchange uses to decide routing. With a **direct exchange**, the routing key must exactly match the queue's binding key.
- **Binding**: A rule that connects an exchange to a queue with a specific routing key. "Route messages with key `alerts.paging` to the queue `alerts.paging.queue`."
- **ACK (Acknowledge)**: When a consumer successfully processes a message, it sends an ACK back to RabbitMQ, which removes the message from the queue.
- **NACK (Negative Acknowledge)**: When a consumer fails to process a message, it sends a NACK. The message can either be requeued (try again) or sent to a dead-letter queue.
- **Dead-Letter Queue (DLQ)**: A special queue where "rejected" messages go. Instead of losing failed messages, they're stored in the DLQ for later analysis or automatic retry.
**How is RabbitMQ different from Kafka?** Kafka is designed for high-throughput event streaming where many consumers read the same stream independently. RabbitMQ is designed for task distribution — each message is processed by exactly one consumer, and the broker manages acknowledgments and retries. In this project, Kafka handles the "fan-out" (one event → 9 consumers), while RabbitMQ handles point-to-point workflows (page a physician, generate a discharge summary).
---
## Why RabbitMQ in This Project?
When a critical alert fires, someone needs to be paged. If they don't respond, the alert must escalate to a backup. This requires a workflow with timeouts, acknowledgment tracking, and dead-letter routing — exactly what RabbitMQ excels at. Kafka handles the high-volume event streaming; RabbitMQ handles the delivery-guaranteed notification workflows.
---
## Architecture Overview
```
Kafka Consumer RabbitMQ
(NotificationPublisher) ┌─────────────────────────────────────┐
│ │ clinical.notifications.exchange │
│ BasicPublish │ (direct exchange) │
▼ └──────┬──────┬──────┬──────┬────────┘
│ │ │ │
routing key: alerts. alerts. notif. notif.
paging escal. disch. recon.
│ │ │ │
▼ ▼ ▼ ▼
┌────────┐ ┌──────┐ ┌─────┐ ┌──────┐
│paging │ │escal.│ │disch│ │recon.│
│.queue │ │.queue│ │.queue│ │.queue│
└───┬────┘ └──────┘ └─────┘ └──────┘
│ ▲
NACK │ │ after TTL expires
(timeout) ▼ │
┌────────┐ │
│paging │────┘
│.dlq │ (dead-letter re-routes
└────────┘ to escalation queue)
```
---
## Configuration
```csharp
public sealed class RabbitMqOptions
{
public const string Section = "RabbitMq";
public string Host { get; init; } = "localhost";
public int Port { get; init; } = 5674;
public string Username { get; init; } = "guest";
public string Password { get; init; } = "guest";
public string VirtualHost { get; init; } = "/";
public int PagingAckTimeoutMs { get; init; } = 300000; // 5 minutes
}
```
```json
{
"RabbitMq": {
"Host": "localhost",
"Port": 5674,
"Username": "guest",
"Password": "guest",
"PagingAckTimeoutMs": 300000
}
}
```
The `PagingAckTimeoutMs` drives two things: how long the paging worker waits for a physician to acknowledge, and the TTL on the dead-letter queue.
### Connection Factory
```csharp
public static class RabbitMqConnectionFactory
{
public static ConnectionFactory Create(RabbitMqOptions o, bool dispatchConsumersAsync = false)
=> new()
{
HostName = o.Host,
Port = o.Port,
UserName = o.Username,
Password = o.Password,
VirtualHost = o.VirtualHost,
DispatchConsumersAsync = dispatchConsumersAsync,
};
}
```
**What is `DispatchConsumersAsync`?** By default, the RabbitMQ .NET client delivers messages to consumers on a synchronous thread. Setting this to `true` lets consumers use `async/await` in their message handlers — necessary when the handler needs to call the database or other async services.
---
## Topology Provisioning
**What is topology?** In RabbitMQ, "topology" means the structure of exchanges, queues, and bindings. Before any message can flow, these must exist. The `RabbitMqTopologyProvisioner` creates them on application startup:
```csharp
public sealed class RabbitMqTopologyProvisioner : IHostedService
{
public const string Exchange = "clinical.notifications.exchange";
public const string PagingKey = "alerts.paging";
public const string EscalKey = "alerts.escalation";
public const string DischargeKey = "notifications.discharge";
public Task StartAsync(CancellationToken ct)
{
// Create the exchange
channel.ExchangeDeclare(Exchange, ExchangeType.Direct, durable: true);
// Create queues and bind them to the exchange with routing keys
channel.QueueDeclare("alerts.paging.queue", durable: true, ...);
channel.QueueBind("alerts.paging.queue", Exchange, PagingKey);
channel.QueueDeclare("alerts.escalation.queue", durable: true, ...);
channel.QueueBind("alerts.escalation.queue", Exchange, EscalKey);
channel.QueueDeclare("notifications.discharge.queue", durable: true, ...);
channel.QueueBind("notifications.discharge.queue", Exchange, DischargeKey);
// ... more queues
}
}
```
**`durable: true`** means the queue survives a RabbitMQ restart. Without this, restarting the broker would delete the queue and all its messages.
All declare operations are **idempotent** — if the queue already exists with the same settings, RabbitMQ does nothing. This makes startup safe to run multiple times.
### The Paging Queue and Dead-Letter Chain
The paging queue has special arguments that set up automatic escalation:
```csharp
// alerts.paging.queue — dead-letters unacknowledged messages to the DLQ
channel.QueueDeclare(
queue: "alerts.paging.queue",
durable: true,
arguments: new Dictionary<string, object>
{
["x-dead-letter-exchange"] = "", // default exchange
["x-dead-letter-routing-key"] = "alerts.paging.dlq", // DLQ queue name
});
// alerts.paging.dlq — messages expire after PagingAckTimeoutMs, then re-route to escalation
channel.QueueDeclare(
queue: "alerts.paging.dlq",
durable: true,
arguments: new Dictionary<string, object>
{
["x-message-ttl"] = opts.PagingAckTimeoutMs, // 5 minutes
["x-dead-letter-exchange"] = Exchange, // back to main exchange
["x-dead-letter-routing-key"] = EscalKey, // → alerts.escalation.queue
});
```
**What is `x-message-ttl`?** A queue-level TTL (time-to-live). Any message sitting in this queue for longer than this duration is automatically removed. Combined with `x-dead-letter-exchange`, removed messages are re-routed instead of deleted.
This creates a chain: paging queue → NACK → DLQ → wait 5 minutes → escalation queue. No application code manages the delay — RabbitMQ handles it automatically.
**Immutable TTL gotcha**: `x-message-ttl` cannot be changed after a queue is created. If you need to change the timeout (e.g., from 5 minutes to 5 seconds for testing), the provisioner must delete and recreate the DLQ. The code handles this with a try/catch:
```csharp
try
{
channel.QueueDeclare(dlq, durable: true, arguments: args);
}
catch (OperationInterruptedException ex)
{
// TTL mismatch — delete and recreate
cleanup.QueueDelete(dlq);
cleanup.QueueDeclare(dlq, durable: true, arguments: args);
}
```
---
## The Paging Worker
The paging worker is the core notification workflow. It consumes messages from `alerts.paging.queue`, pages the attending physician, then polls the database waiting for acknowledgment:
```csharp
public sealed class PagingWorkerService : BackgroundService
{
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
var factory = RabbitMqConnectionFactory.Create(o, dispatchConsumersAsync: true);
using var connection = factory.CreateConnection("paging-worker");
using var channel = connection.CreateModel();
// Process one message at a time
channel.BasicQos(prefetchSize: 0, prefetchCount: 1, global: false);
var consumer = new AsyncEventingBasicConsumer(channel);
consumer.Received += async (sender, ea) =>
{
try
{
await HandlePageAsync(channel, ea, stoppingToken);
}
catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested)
{
// App shutting down — requeue so restart doesn't false-escalate
channel.BasicNack(ea.DeliveryTag, multiple: false, requeue: true);
}
catch (Exception ex)
{
// Unexpected error — send to DLQ for escalation
channel.BasicNack(ea.DeliveryTag, multiple: false, requeue: false);
}
};
channel.BasicConsume("alerts.paging.queue", autoAck: false, consumer);
await Task.Delay(Timeout.Infinite, stoppingToken);
}
}
```
**What is `BasicQos` (prefetch)?** Prefetch controls how many unacknowledged messages RabbitMQ sends to the consumer at once. `prefetchCount: 1` means "send me one message, wait for my ACK before sending the next one." For paging, this ensures the worker handles one alert at a time — you don't want to be simultaneously waiting on acknowledgments for 10 different alerts.
**What is `autoAck: false`?** When true, RabbitMQ considers the message acknowledged the moment it's delivered to the consumer. When false (manual acknowledgment), the consumer must explicitly ACK or NACK. Manual mode is safer because if the consumer crashes before finishing, the message is re-delivered.
### The Paging and Acknowledgment Loop
```csharp
private async Task HandlePageAsync(IModel channel, BasicDeliverEventArgs ea, CancellationToken ct)
{
var alertId = Guid.Parse(doc.RootElement.GetProperty("alertId").GetString()!);
_logger.LogWarning(
"[PAGE] Paging attending physician '{Physician}' for encounter {EncounterId}",
physician, encounterId);
var deadline = DateTimeOffset.UtcNow.AddMilliseconds(o.PagingAckTimeoutMs);
// Poll the database every 2 seconds waiting for acknowledgment
while (DateTimeOffset.UtcNow < deadline && !ct.IsCancellationRequested)
{
await Task.Delay(2_000, ct);
var acknowledged = await IsAlertAcknowledgedAsync(alertId, ct);
if (acknowledged)
{
// Physician acknowledged — ACK the message (removes from queue)
channel.BasicAck(ea.DeliveryTag, multiple: false);
return;
}
}
// Timeout: NACK with requeue=false → message goes to DLQ
// After x-message-ttl expires on the DLQ, it re-routes to escalation queue
channel.BasicNack(ea.DeliveryTag, multiple: false, requeue: false);
}
```
The flow:
1. **Page sent** — log the page to the attending physician
2. **Poll loop** — check the database every 2 seconds for up to 5 minutes
3. **If acknowledged** — `BasicAck` removes the message. Done.
4. **If timeout** — `BasicNack(requeue: false)` sends the message to the DLQ. After `x-message-ttl` expires, RabbitMQ automatically routes it to the escalation queue.
### Graceful Shutdown
When the application is stopping, in-flight messages are requeued (`requeue: true`) instead of NACKed to the DLQ. This prevents false escalations caused by application restarts.
---
## The Escalation Worker
The escalation worker consumes from `alerts.escalation.queue` — messages that reached here because no one acknowledged the page within the timeout:
```csharp
private async Task HandleEscalationAsync(IModel channel, BasicDeliverEventArgs ea, CancellationToken ct)
{
var alertId = Guid.Parse(doc.RootElement.GetProperty("alertId").GetString()!);
_logger.LogCritical("[ESCALATION] Paging on-call backup for alert {AlertId}", alertId);
// Update the alert status to "Escalated" in the database
var escalated = await UpdateAlertStatusEscalatedAsync(alertId, ct);
if (escalated)
_metrics.EscalationsTotal.Inc();
channel.BasicAck(ea.DeliveryTag, multiple: false);
}
private async Task<bool> UpdateAlertStatusEscalatedAsync(Guid alertId, CancellationToken ct)
{
var alert = await db.ClinicalAlerts.FindAsync(new object[] { alertId }, ct);
if (alert is null) return false;
// Only escalate if still open — if acknowledged between NACK and TTL expiry, leave it
if (alert.Status != AlertStatus.Open) return false;
alert.Status = AlertStatus.Escalated;
await db.SaveChangesAsync(ct);
return true;
}
```
The `status != AlertStatus.Open` check handles a race condition: if the physician acknowledges the alert while the message sits in the DLQ, the escalation worker should not overwrite the acknowledgment.
---
## The Discharge Summary Worker
When a patient is discharged, a message arrives on `notifications.discharge.queue`. The worker generates a text summary and uploads it to MinIO:
```csharp
private async Task HandleDischargeSummaryAsync(IModel channel, BasicDeliverEventArgs ea, CancellationToken ct)
{
var encounterId = Guid.Parse(doc.RootElement.GetProperty("encounterId").GetString()!);
var summary = await BuildSummaryAsync(encounterId, ct);
await UploadToMinioAsync(encounterId, summary, ct);
channel.BasicAck(ea.DeliveryTag, multiple: false);
}
```
On success: ACK. On failure: NACK with `requeue: true` (retry later — maybe MinIO was temporarily unreachable).
---
## Queue Summary
| Queue | Routing Key | Consumer | Purpose |
|-------|-------------|----------|---------|
| `alerts.paging.queue` | `alerts.paging` | `PagingWorkerService` | Page attending physician, wait for ACK |
| `alerts.paging.dlq` | (dead-letter from paging queue) | (none — auto-routes after TTL) | Hold unacknowledged pages before escalation |
| `alerts.escalation.queue` | `alerts.escalation` | `EscalationWorkerService` | Page on-call backup, mark alert escalated |
| `notifications.discharge.queue` | `notifications.discharge` | `DischargeSummaryWorkerService` | Generate and upload discharge summary |
| `notifications.reconciliation.queue` | `notifications.reconciliation` | `ReconciliationScheduler` | Safety findings (unacked alerts, pending orders) |
| `clinical.sync.batch_received` | `sync.batch_received` | `ClinicalSyncBatchConsumer` | Process gateway sync batches |
---
## The Complete Escalation Timeline
```
t=0:00 Critical alert fires
→ OutboxRelay → Kafka (alert.generated)
→ NotificationPublisher → RabbitMQ (alerts.paging.queue)
t=0:00 PagingWorker picks up message
→ Logs "[PAGE] Paging attending physician..."
→ Starts polling database every 2 seconds
t=0:00 IF physician acknowledges within 5 minutes:
to → PagingWorker sees ACK in database
t=5:00 → BasicAck — message removed from queue
→ Flow complete ✓
t=5:00 IF no acknowledgment after 5 minutes:
→ PagingWorker BasicNack(requeue: false)
→ Message routes to alerts.paging.dlq
→ DLQ holds message for PagingAckTimeoutMs (another 5 min)
t=10:00 DLQ x-message-ttl expires
→ Message auto-routes to alerts.escalation.queue
t=10:00 EscalationWorker picks up message
→ Logs "[ESCALATION] Paging on-call backup..."
→ Sets alert status → Escalated
→ Increments escalations_total metric
→ BasicAck — flow complete
```
---
## Health Check
```csharp
public sealed class RabbitMqHealthCheck : IHealthCheck
{
public async Task<HealthCheckResult> CheckHealthAsync(
HealthCheckContext context, CancellationToken cancellationToken)
{
var factory = RabbitMqConnectionFactory.Create(_options);
using var connection = await Task.Run(
() => factory.CreateConnection(), cancellationToken);
return HealthCheckResult.Healthy(
data: new Dictionary<string, object>
{ ["endpoint"] = connection.Endpoint.ToString() });
}
}
```
Opens a connection to verify RabbitMQ is reachable. Used by the `/health/ready` endpoint.
---
## Key Takeaways
- **RabbitMQ is for workflows, Kafka is for streaming** — use RabbitMQ when you need acknowledgments, timeouts, and dead-letter routing. Use Kafka when you need one event consumed by many independent groups.
- **Dead-letter queues + TTL = delayed retry** — RabbitMQ's built-in features handle the escalation timer without any application-level scheduling code.
- **Manual acknowledgment is essential** — `autoAck: false` ensures messages are only removed after successful processing, preventing data loss on crashes.
- **Prefetch controls concurrency** — `prefetchCount: 1` on the paging worker ensures one alert is paged at a time. The escalation worker uses `prefetchCount: 5` because escalation handling is fast (just a DB update).
- **Topology provisioning is idempotent** — declare operations safely re-run on every application restart. The exception is immutable queue arguments like `x-message-ttl`, which require delete-and-recreate.
docs/guides/08-elasticsearch-cqrs-read-store.md
+286
View File
@@ -0,0 +1,286 @@
# Guide 8: Elasticsearch as a CQRS Read Store
## What is Elasticsearch?
**Elasticsearch** is a distributed search and analytics engine. While PostgreSQL excels at transactional operations (INSERT, UPDATE, JOIN), Elasticsearch excels at searching across large amounts of data — full-text search, filtering, aggregations (sums, averages, counts grouped by category), and fuzzy matching.
Elasticsearch stores data in **indexes** (similar to database tables). Each index contains **documents** (similar to rows) in JSON format. Unlike a relational database, Elasticsearch doesn't require you to define a rigid schema upfront — though you should define **mappings** (field types) for predictable behavior.
Key terminology:
- **Index**: A collection of documents (like a table). Example: `patient_encounters`, `observations`.
- **Document**: A single JSON record (like a row). Identified by an ID.
- **Mapping**: The schema definition for an index — which fields exist and what type they are (`keyword` for exact match, `text` for full-text search, `date` for timestamps, etc.).
- **Keyword vs Text**: A `keyword` field stores the value as-is for exact matching and sorting ("ICU" must match exactly). A `text` field is analyzed (split into tokens, lowercased) for full-text search ("John Smith" matches a search for "john").
- **Aggregation**: A computation across documents — like SQL's `GROUP BY`, `COUNT`, `AVG`. Elasticsearch can aggregate millions of documents in milliseconds.
## What is CQRS?
**CQRS** stands for **Command Query Responsibility Segregation**. The core idea: use different data stores (or different models) for writes and reads.
- **Command side** (writes): Your application writes to PostgreSQL — it handles transactions, constraints, and data integrity.
- **Query side** (reads): Your application reads from Elasticsearch — it handles search, filtering, and analytics.
Why separate them? Because the ideal data structure for writing (normalized tables with foreign keys and constraints) is different from the ideal structure for reading (denormalized documents with all related data in one place). A single read from Elasticsearch can return a patient's name, encounter status, department, alert count, and latest NEWS2 score — without any JOINs.
The bridge between the two is an event stream. When data changes in PostgreSQL, an event is published to Kafka. A consumer reads that event and updates Elasticsearch. This means Elasticsearch is **eventually consistent** — there's a small delay (usually under a second) between a write to PostgreSQL and the data appearing in Elasticsearch.
---
## Why Elasticsearch in This Project?
The dashboard needs to search patients by name or MRN, filter encounters by department, see alert counts by severity, and display observation trends — all in near-real-time. These are analytics and search queries that Elasticsearch handles orders of magnitude faster than PostgreSQL with JOINs across large tables.
---
## Architecture Overview
```
Write Path (PostgreSQL) Read Path (Elasticsearch)
┌──────────────┐ ┌──────────────────────┐
│ API writes │ │ Dashboard queries │
│ observations,│ │ AnalyticsService │
│ alerts, │ │ SearchPatientsAsync │
│ encounters │ │ GetAlertSummaryAsync │
└──────┬───────┘ └──────────┬───────────┘
│ │
▼ ▼
PostgreSQL ──outbox──► Kafka ──es-indexer──► Elasticsearch
│
┌──────────┴──────────┐
│ patient_encounters │
│ observations │
│ clinical_alerts │
└─────────────────────┘
```
---
## Index Provisioning
Indexes and their mappings are created on application startup by `ElasticIndexProvisioner`:
```csharp
public class ElasticIndexProvisioner : IHostedService
{
public async Task StartAsync(CancellationToken cancellationToken)
{
await EnsureIndexAsync<PatientEncounterDocument>(
_options.Indices.PatientEncounters, BuildPatientEncountersMapping());
await EnsureIndexAsync<ObservationDocument>(
_options.Indices.Observations, BuildObservationsMapping());
await EnsureIndexAsync<ClinicalAlertDocument>(
_options.Indices.ClinicalAlerts, BuildClinicalAlertsMapping());
}
private async Task EnsureIndexAsync<T>(string indexName,
Action<CreateIndexRequestDescriptor<T>> configure) where T : class
{
var exists = await _elastic.Indices.ExistsAsync(indexName);
if (exists.Exists) return; // idempotent — skip if already exists
var resp = await _elastic.Indices.CreateAsync<T>(indexName, configure);
if (!resp.IsValidResponse)
throw new InvalidOperationException(
$"Failed to create index '{indexName}': {resp.DebugInformation}");
}
}
```
### The Patient Encounters Mapping
```csharp
private Action<CreateIndexRequestDescriptor<PatientEncounterDocument>>
BuildPatientEncountersMapping() =>
d => d.Mappings(m => m.Properties(p => p
.Keyword(k => k.EncounterId)
.Keyword(k => k.PatientId)
.Keyword(k => k.Mrn)
// text for full-text search + keyword sub-field for exact sort/filter
.Text(t => t.PatientName, tf => tf
.Fields(f => f.Keyword(k => k.PatientName)))
.Keyword(k => k.Department)
.Keyword(k => k.Status)
.IntegerNumber(i => i.News2Score!)
.Keyword(k => k.News2RiskLevel!)
.Date(d => d.AdmittedAt)
.IntegerNumber(i => i.OpenAlertCount)
.Date(d => d.LastObservationAt!)
));
```
The `PatientName` field has a **multi-field mapping**: it's stored as both `text` (for full-text search — searching "john" matches "John Smith") and `keyword` (for exact sorting and filtering). This is a common Elasticsearch pattern for fields that need both search and sort capabilities.
---
## The ES Indexer Service (Kafka → Elasticsearch)
`EsIndexerService` is the bridge between the write side (PostgreSQL/Kafka) and the read side (Elasticsearch). It's a Kafka consumer in the `es-indexer` consumer group that subscribes to 5 topics and routes each message to the appropriate handler:
```csharp
consumer.Subscribe(new[]
{
_kafkaOptions.Topics.ObservationRecorded,
_kafkaOptions.Topics.AlertGenerated,
_kafkaOptions.Topics.EncounterStatusChanged,
_kafkaOptions.Topics.SepsisBundleCreated,
_kafkaOptions.Topics.SepsisBundleUpdated
});
private Task DispatchAsync(string topic, string payload, CancellationToken ct)
=> topic switch
{
var t when t == _kafkaOptions.Topics.EncounterStatusChanged
=> HandleEncounterStatusChangedAsync(payload, ct),
var t when t == _kafkaOptions.Topics.ObservationRecorded
=> HandleObservationRecordedAsync(payload, ct),
var t when t == _kafkaOptions.Topics.AlertGenerated
=> HandleAlertGeneratedAsync(payload, ct),
// ... sepsis bundle topics
};
```
### Upsert for Encounters
**What is an upsert?** "Update or insert" — if the document exists, update it. If it doesn't exist, create it. This makes the operation **idempotent**: processing the same event twice produces the same result.
```csharp
var resp = await _elastic.UpdateAsync<PatientEncounterDocument, PatientEncounterDocument>(
_esOptions.Indices.PatientEncounters,
evt.EncounterId.ToString(),
u => u.Doc(doc).DocAsUpsert(true), // create if missing, replace if exists
ct);
```
### Append-Only for Observations
Observations are indexed by their unique `observationId`. Since observation IDs never repeat, this is naturally idempotent:
```csharp
await _elastic.IndexAsync(
doc,
i => i.Index(_esOptions.Indices.Observations).Id(doc.ObservationId),
ct);
```
The indexer also updates `lastObservationAt` on the parent encounter document using a **Painless script** (Elasticsearch's built-in scripting language). The script handles out-of-order delivery — an older observation reprocessed after a newer one won't overwrite the timestamp:
```painless
if (ctx._source.lastObservationAt == null ||
params.recordedAt > ctx._source.lastObservationAt) {
ctx._source.lastObservationAt = params.recordedAt;
}
```
### Denormalization for Alerts
When an alert fires, the indexer both creates the alert document AND updates the parent encounter document:
```csharp
// 1. Index the alert
await _elastic.IndexAsync(alertDoc, i => i.Index("clinical_alerts").Id(alertDoc.AlertId), ct);
// 2. Increment openAlertCount on the encounter + stamp NEWS2 score if present
await _elastic.UpdateAsync<PatientEncounterDocument, object>(
"patient_encounters", evt.EncounterId.ToString(),
u => u.Script(new Script(new InlineScript
{
Source = "ctx._source.openAlertCount += 1",
Language = ScriptLanguage.Painless
})).RetryOnConflict(3), ct);
```
**What is denormalization?** In a relational database, you'd JOIN alerts to encounters to get the count. In Elasticsearch, you store the count directly on the encounter document. This eliminates JOINs (which Elasticsearch doesn't support well) but means you must keep the denormalized data in sync through your event handlers.
**`RetryOnConflict(3)`** handles optimistic concurrency — if two events update the same encounter document simultaneously, Elasticsearch retries the update up to 3 times.
---
## Querying Elasticsearch — The Analytics Service
The `AnalyticsService` demonstrates common Elasticsearch query patterns.
### Patient Search (Full-Text + Filters)
```csharp
public async Task<object> SearchPatientsAsync(string? q, string? department,
string? status, int page, int pageSize)
{
var resp = await _elastic.SearchAsync<PatientEncounterDocument>(s => s
.Indices(_options.Indices.PatientEncounters)
.Query(q2 => q2.Bool(b =>
{
if (!string.IsNullOrWhiteSpace(q))
{
b.Should(
// MRN: exact keyword match — boosted x3 because MRN lookup
// is the most common search for clinicians
s2 => s2.Term(t => t.Field(p => p.Mrn).Value(q).Boost(3)),
// Patient name: full-text search
s2 => s2.Match(m => m.Field(p => p.PatientName).Query(q)),
// Department: exact match
s2 => s2.Term(t => t.Field(p => p.Department).Value(q))
);
b.MinimumShouldMatch(1);
}
if (filters.Count > 0)
b.Filter(filters.ToArray());
}))
.From((page - 1) * pageSize)
.Size(pageSize));
return new { total = resp.Total, page, pageSize, data = resp.Documents };
}
```
**What is boosting?** `Boost(3)` means an MRN match is weighted 3x higher than a name match in relevance scoring. If a clinician searches for "12345" and that matches both a patient's MRN and part of their phone number, the MRN match ranks higher.
### Alert Summary with Aggregations
```csharp
var resp = await _elastic.SearchAsync<ClinicalAlertDocument>(s => s
.Indices(_options.Indices.ClinicalAlerts)
.Query(q => q.Bool(b => b.Filter(filters.ToArray())))
.Aggregations(a => a
.Add("by_department", agg => agg.Terms(t => t
.Field(a => a.Department).Size(50)))
)
.Size(0)); // no raw documents — aggregation result only
```
`.Size(0)` tells Elasticsearch "I don't need the actual documents, just the aggregation results." This is much faster when you only need counts or statistics.
### Population Query with Cardinality
```csharp
var resp = await _elastic.SearchAsync<ObservationDocument>(s => s
.Indices(_options.Indices.Observations)
.Query(q => q.Bool(b => b.Filter(filters.ToArray())))
.Aggregations(a => a
.Add("unique_patients", agg => agg
.Cardinality(c => c.Field(o => o.PatientId)))
)
.Size(0));
```
**What is cardinality?** It counts distinct values — like SQL's `COUNT(DISTINCT patient_id)`. "How many unique patients had a heart rate above 120?" Two observations from the same patient count as one patient.
---
## Index Summary
| Index | Document ID | Write Pattern | Use Case |
|-------|-------------|---------------|----------|
| `patient_encounters` | `encounterId` | Upsert (create or replace) | Dashboard ward table, patient search |
| `observations` | `observationId` | Append (index by unique ID) | Observation trend queries, population analytics |
| `clinical_alerts` | `alertId` | Append (index by unique ID) | Alert volume by department/severity |
---
## Key Takeaways
- **CQRS separates concerns**: PostgreSQL handles writes with transactions and constraints. Elasticsearch handles reads with search and aggregations. Each store is optimized for its purpose.
- **Events are the bridge**: Kafka events flow from the write side to the read side. The ES indexer consumer keeps Elasticsearch in sync with PostgreSQL.
- **Idempotency matters**: Upserts and unique document IDs ensure that reprocessing the same event (after a crash or retry) doesn't corrupt the data.
- **Denormalization eliminates JOINs**: The encounter document contains patient name, alert count, NEWS2 score, and sepsis bundle status — all in one document, readable in one query.
- **Eventual consistency is the tradeoff**: There's a small delay between writing to PostgreSQL and the data appearing in Elasticsearch. For a dashboard that refreshes every 5-10 seconds, this is imperceptible.
docs/guides/09-minio-data-lake-parquet.md
+376
View File
@@ -0,0 +1,376 @@
# Guide 9: MinIO Data Lake with Parquet
## What Are MinIO, Parquet, and a Data Lake?
### MinIO — S3-Compatible Object Storage
**Object storage** is a way to store files (called "objects") in a flat structure identified by keys (like file paths). Unlike a filesystem with directories, object storage uses a single "bucket" (container) and key strings like `observations/2026/06/24/partition-0-offset-0000001000.parquet`.
**Amazon S3** is the most widely-used object storage service. **MinIO** is an open-source server that implements the same S3 API, so you can develop locally with MinIO and deploy to AWS S3 later without changing your code. The application uses the Minio .NET SDK, which speaks the S3 protocol.
### Parquet — Columnar File Format
**Parquet** is a file format designed for analytics. While JSON and CSV store data row-by-row, Parquet stores data column-by-column:
```
CSV/JSON (row-oriented): Parquet (column-oriented):
┌─────┬────────┬───────┐ ┌─────────────────────────┐
│ id │ code │ value │ │ id: [1, 2, 3, 4, ...] │
├─────┼────────┼───────┤ │ code: [HR, BP, HR, ...] │
│ 1 │ HR │ 80 │ │ value: [80, 120, 85, ...]│
│ 2 │ BP │ 120 │ └─────────────────────────┘
│ 3 │ HR │ 85 │
└─────┴────────┴───────┘
```
Why columnar? Analytics queries usually read a few columns from many rows ("give me all heart rate values"). Columnar storage lets the query engine read only the `code` and `value` columns, skipping everything else. Parquet also compresses data within each column (similar values compress well), so files are much smaller than JSON — often 10-50x.
Tools like Apache Spark, AWS Athena, Pandas, and DuckDB can read Parquet files natively and query them with SQL.
### Data Lake — Long-Term Analytics Storage
A **data lake** is a centralized storage repository where you dump raw data in its original form for later analysis. Unlike a database (which is optimized for real-time transactional queries), a data lake is optimized for batch analytics — "analyze all observations from the past 6 months" or "what percentage of critical alerts were acknowledged within 5 minutes across all departments?"
The typical pattern: application databases hold recent data for real-time operations, while the data lake holds historical data for research, compliance auditing, and machine learning.
---
## Why a Data Lake in This Project?
PostgreSQL and Elasticsearch serve real-time operational needs. But clinical data has long-term value — research on sepsis detection accuracy, compliance audits, training machine learning models. The data lake captures a complete, immutable history of all events in a format optimized for large-scale analytics, stored cheaply in object storage.
---
## Architecture Overview
```
Kafka Topics DataLakeWriterService MinIO (S3)
┌─────────────────┐ ┌──────────────────┐
│observation. │─┐ │ vigilcare/ │
│recorded │ │ ┌──────────────────────┐ │ │
├─────────────────┤ ├────►│ In-memory buffer │ │ observations/ │
│alert. │ │ │ (by topic + date + │ flush │ 2026/06/24/ │
│generated │ │ │ partition) │────────►│ part-0.pqt │
├─────────────────┤ │ │ │ │ │
│encounter.status. │─┘ │ Flush when: │ │ alerts/ │
│changed │ │ - 1000 events buffer │ │ 2026/06/24/ │
└─────────────────┘ │ - 5 min elapsed │ │ part-0.pqt │
└──────────────────────┘ │ │
│ encounters/ │
│ 2026/06/24/ │
│ part-0.pqt │
└──────────────────┘
```
---
## Configuration
### MinIO Client
```csharp
public static class MinioClientFactory
{
public static IMinioClient Build(MinioOptions opts)
{
var client = new MinioClient()
.WithEndpoint(opts.Endpoint)
.WithCredentials(opts.AccessKey, opts.SecretKey);
if (opts.UseSSL) client = client.WithSSL();
return client.Build();
}
}
```
```csharp
public sealed class MinioOptions
{
public string Endpoint { get; init; } = "localhost:9005";
public string AccessKey { get; init; } = "minioadmin";
public string SecretKey { get; init; } = "minioadmin";
public string BucketName { get; init; } = "vigilcare";
public bool UseSSL { get; init; } = false;
}
```
### Data Lake Options
```csharp
public sealed class DataLakeOptions
{
public int FlushCount { get; init; } = 1_000; // max events before forced flush
public int FlushIntervalSeconds { get; init; } = 300; // max time before forced flush (5 min)
public string BucketName { get; init; } = "vigilcare";
}
```
---
## The DataLakeWriterService
This is a Kafka consumer that buffers events in memory and flushes them to MinIO as Parquet files.
### Consumer Setup
```csharp
protected override async Task ExecuteAsync(CancellationToken ct)
{
var consumerConfig = new ConsumerConfig
{
BootstrapServers = _kafkaOptions.BootstrapServers,
GroupId = "data-lake-writer",
AutoOffsetReset = AutoOffsetReset.Earliest,
EnableAutoCommit = false,
};
using var consumer = new ConsumerBuilder<string, string>(consumerConfig).Build();
consumer.Subscribe(new[]
{
_kafkaOptions.Topics.ObservationRecorded,
_kafkaOptions.Topics.AlertGenerated,
_kafkaOptions.Topics.EncounterStatusChanged,
});
```
### Buffering Strategy
Instead of writing one Parquet file per event (too many small files), the service buffers events and flushes in batches:
```csharp
// Buffer key groups events into one Parquet file
private record BufferKey(string Topic, string DatePath, int Partition);
// Track the highest offset per partition for safe commits
private readonly Dictionary<TopicPartition, TopicPartitionOffset> _highWatermarks = new();
while (!ct.IsCancellationRequested)
{
var result = consumer.Consume(TimeSpan.FromMilliseconds(500));
if (result is not null)
AddToBuffer(result);
var totalBuffered = _buffer.Values.Sum(v => v.Count);
var shouldFlushCount = totalBuffered >= _opts.FlushCount; // 1000 events
var shouldFlushTime = elapsed >= TimeSpan.FromSeconds(300); // 5 minutes
if ((shouldFlushCount || shouldFlushTime) && totalBuffered > 0)
{
await FlushAsync(consumer, ct);
lastFlush = DateTimeOffset.UtcNow;
}
}
```
Events are grouped by **(topic, date, partition)**. All heart rate observations from June 24th on Kafka partition 0 end up in one Parquet file: `observations/2026/06/24/partition-0-offset-0000001000.parquet`.
### Date Partitioning
The date in the file path comes from the event's timestamp (e.g., `recordedAt` for observations), not the wall clock. This is important: analytics tools like AWS Athena and Apache Spark can "partition-prune" — when you query "observations from June 24th," they only read files from the `2026/06/24/` folder, skipping everything else:
```csharp
public static string ExtractDatePath(string topic, string payload, KafkaTopicOptions topics)
{
var ts = topic switch
{
var t when t == topics.ObservationRecorded => GetTimestamp(doc, "recordedAt"),
var t when t == topics.AlertGenerated => GetTimestamp(doc, "triggeredAt"),
var t when t == topics.EncounterStatusChanged => GetTimestamp(doc, "changedAt"),
_ => DateTimeOffset.UtcNow,
};
return $"{ts.Year:D4}/{ts.Month:D2}/{ts.Day:D2}";
}
```
### Partial-Commit Safety
**This is the most important safety feature of the data lake writer.** When flushing, some file uploads might succeed while others fail (e.g., MinIO is temporarily unreachable for one partition). The service only commits Kafka offsets for partitions where ALL uploads succeeded:
```csharp
private async Task FlushAsync(IConsumer<string, string> consumer, CancellationToken ct)
{
var failedPartitions = new HashSet<int>();
foreach (var (key, events) in _buffer)
{
try
{
var bytes = await BuildParquetAsync(key.Topic, events, key.Partition);
await UploadToMinioAsync(objectKey, bytes, ct);
flushedKeys.Add(key);
}
catch (Exception ex)
{
failedPartitions.Add(key.Partition);
}
}
// Only commit offsets for partitions with no failures
var safeOffsets = _highWatermarks
.Where(kv => !failedPartitions.Contains(kv.Key.Partition))
.Select(kv => kv.Value)
.ToList();
if (safeOffsets.Count > 0)
consumer.Commit(safeOffsets);
// Retain failed buffers for retry on next flush
foreach (var key in flushedKeys)
_buffer.Remove(key);
}
```
If partition 2 fails to upload but partitions 0, 1, 3, 4, 5 succeed: offsets for 0, 1, 3, 4, 5 are committed. Partition 2's buffer is retained and retried on the next flush cycle. If the service crashes before the next flush, Kafka re-delivers partition 2's events from the last committed offset — no data is lost.
### Shutdown Flush
On graceful shutdown, the service flushes any remaining buffered events:
```csharp
finally
{
if (_buffer.Values.Sum(v => v.Count) > 0)
{
try { await FlushAsync(consumer, CancellationToken.None); }
catch (Exception ex)
{
_logger.LogError(ex,
"DataLakeWriter shutdown flush failed — some events may be re-read on next start");
}
}
consumer.Close();
}
```
`CancellationToken.None` is used instead of the stopping token because the host shutdown might cancel the token before MinIO uploads finish.
---
## Building Parquet Files
The `ParquetFileBuilder` uses the `Parquet.Net` library to create Parquet files in memory:
```csharp
public static async Task<byte[]> BuildObservationsAsync(IReadOnlyList<ObservationRow> rows)
{
var schema = new ParquetSchema(
new DataField<string>("observation_id"),
new DataField<string>("encounter_id"),
new DataField<string>("patient_id"),
new DataField<string>("mrn"),
new DataField<string>("observation_code"),
new DataField<double>("value"),
new DataField<string>("unit"),
new DataField<string>("source"),
new DataField<string>("recorded_at"),
new DataField<int>("kafka_partition"),
new DataField<long>("kafka_offset")
);
using var ms = new MemoryStream();
using (var writer = await ParquetWriter.CreateAsync(schema, ms))
using (var rg = writer.CreateRowGroup())
{
var f = schema.DataFields;
await rg.WriteColumnAsync(new DataColumn(f[0], rows.Select(r => r.ObservationId).ToArray()));
await rg.WriteColumnAsync(new DataColumn(f[1], rows.Select(r => r.EncounterId).ToArray()));
// ... one WriteColumnAsync per field
}
return ms.ToArray();
}
```
Parquet is columnar, so you write one column at a time (all observation IDs, then all encounter IDs, etc.), not one row at a time.
The `kafka_partition` and `kafka_offset` fields provide **lineage** — given any Parquet row, you can trace it back to the exact Kafka message it came from. This is useful for debugging and auditing.
---
## Parsing Kafka Events
The `DataLakeEventParser` extracts fields from Kafka JSON payloads into typed row objects:
```csharp
public static ObservationRow ParseObservationRow(string payload, long offset, int partition)
{
var d = JsonDocument.Parse(payload).RootElement;
return new ObservationRow(
ObservationId : GetString(d, "observationId"),
EncounterId : GetString(d, "encounterId"),
PatientId : GetString(d, "patientId"),
ObservationCode : GetString(d, "observationCode", "code"), // fallback name
Value : GetDouble(d, "value"),
RecordedAt : GetTimestampString(d, "recordedAt"),
KafkaPartition : partition,
KafkaOffset : offset);
}
```
The parser is tolerant of field name variations (e.g., `"observationCode"` or `"code"`) for backwards compatibility with older event formats.
---
## Object Key Structure
```
vigilcare/ ← bucket
├── observations/ ← topic-derived folder
│ ├── 2026/06/23/ ← date partition from event timestamp
│ │ ├── partition-0-offset-0000001000.parquet
│ │ └── partition-3-offset-0000002500.parquet
│ └── 2026/06/24/
│ └── partition-0-offset-0000005000.parquet
├── alerts/
│ └── 2026/06/24/
│ └── partition-1-offset-0000000100.parquet
├── encounters/
│ └── 2026/06/24/
│ └── partition-2-offset-0000000050.parquet
└── discharge-summaries/ ← from DischargeSummaryWorkerService
└── {encounterId}/
└── summary.pdf
```
The key encodes: what type of data, when it happened, which Kafka partition, and the starting offset. This makes each file uniquely identifiable and traceable.
---
## Querying the Data Lake
Once data is in MinIO as Parquet files, you can query it using analytics tools:
**With DuckDB (local, fast):**
```sql
SELECT observation_code, AVG(value), COUNT(*)
FROM read_parquet('s3://vigilcare/observations/2026/06/*/partition-*.parquet')
WHERE observation_code = 'HEART_RATE'
GROUP BY observation_code;
```
**With Pandas (Python):**
```python
import pandas as pd
df = pd.read_parquet('s3://vigilcare/observations/2026/06/24/')
high_hr = df[df['observation_code'] == 'HEART_RATE'][df['value'] > 120]
```
**With AWS Athena (serverless SQL):**
```sql
SELECT COUNT(DISTINCT patient_id)
FROM vigilcare.observations
WHERE observation_code = 'SPO2' AND value < 92
AND recorded_at BETWEEN '2026-06-01' AND '2026-06-30';
```
The date-partitioned folder structure enables **partition pruning** — the query engine only reads files from the date range you're querying, making queries over specific time periods very fast even with years of historical data.
---
## Key Takeaways
- **Buffer-then-flush reduces file count**: One Parquet file per flush (up to 1000 events) instead of one file per event. Fewer, larger files are better for analytics tools.
- **Date partitioning enables fast queries**: Analytics tools skip irrelevant date folders entirely. Querying "last 7 days" reads 7 folders, not millions of files.
- **Partial-commit safety prevents data loss**: Only commit Kafka offsets for partitions where all uploads succeeded. Failed partitions are retried.
- **Kafka lineage enables tracing**: Every Parquet row includes the Kafka partition and offset it came from, creating an audit trail from source to storage.
- **Parquet + object storage is the industry standard**: This pattern (event stream → columnar files → S3-compatible storage) is how most data platforms work. The same Parquet files work with Spark, Athena, Pandas, DuckDB, and dozens of other tools.
- **MinIO is a local S3**: Develop against MinIO, deploy to AWS S3 with zero code changes. Only the endpoint and credentials change.
docs/guides/10-transactional-outbox-pattern.md
+403
View File
@@ -0,0 +1,403 @@
# Guide 10: Transactional Outbox Pattern
## What is the Outbox Pattern?
The outbox pattern solves a fundamental problem in distributed systems: **how do you reliably update a database AND send a message to a message broker (like Kafka) without losing data?**
### The Problem
Imagine this naive approach:
```csharp
// Step 1: Save the observation to the database
await db.Observations.Add(observation);
await db.SaveChangesAsync();
// Step 2: Publish an event to Kafka
await producer.ProduceAsync("observation.recorded", new Message { Value = payload });
```
What can go wrong?
- **Step 1 succeeds, step 2 fails** (Kafka is down): The observation is saved, but no downstream consumers (scoring engines, search indexer, data lake) ever learn about it. The patient's NEWS2 score is never updated.
- **Step 2 succeeds, step 1 fails** (database constraint violation): Consumers process an event for an observation that doesn't exist in the database.
- **Application crashes between steps**: Same problem — one succeeded, the other didn't.
You can't wrap both in a single transaction because PostgreSQL and Kafka are different systems — there's no "distributed transaction" that spans both atomically (and even if there were, it would be slow and fragile).
### The Solution
Instead of publishing to Kafka directly, write a **message record** (an "outbox event") to a table in the same database, in the same transaction as the domain data:
```
┌─────────────────────────────────────────────────────┐
│ Single PostgreSQL Transaction │
│ │
│ 1. INSERT INTO observations (...) VALUES (...) │
│ 2. INSERT INTO outbox_events (...) VALUES (...) │
│ 3. (if critical) INSERT INTO clinical_alerts (...) │
│ 4. INSERT INTO outbox_events (...) VALUES (...) │
│ │
│ COMMIT ← all or nothing │
└─────────────────────────────────────────────────────┘
│
│ A background service (the "relay") polls the outbox table
│ and publishes each event to Kafka separately
▼
┌─────────────────────────────────────────────────────┐
│ OutboxRelayService (background) │
│ │
│ 1. SELECT ... FROM outbox_events │
│ WHERE processed_at IS NULL FOR UPDATE SKIP LOCKED│
│ 2. ProduceAsync to Kafka │
│ 3. UPDATE outbox_events SET processed_at = NOW() │
│ 4. COMMIT │
└─────────────────────────────────────────────────────┘
```
Because both the observation and the outbox event are in the same PostgreSQL transaction, they either both commit or both roll back. The relay can safely retry — if it crashes or Kafka is temporarily down, the unprocessed outbox rows are still in the table, waiting to be picked up.
---
## The Outbox Event Entity
The `outbox_events` table stores messages waiting to be published:
```csharp
public class OutboxEventConfiguration : IEntityTypeConfiguration<OutboxEvent>
{
public void Configure(EntityTypeBuilder<OutboxEvent> builder)
{
builder.ToTable("outbox_events");
builder.HasKey(o => o.Id);
builder.Property(o => o.Id).HasColumnName("id")
.HasDefaultValueSql("gen_random_uuid()");
builder.Property(o => o.Topic).HasColumnName("topic")
.HasMaxLength(200).IsRequired();
builder.Property(o => o.Payload).HasColumnName("payload")
.HasColumnType("jsonb").IsRequired();
builder.Property(o => o.PartitionKey).HasColumnName("partition_key")
.HasMaxLength(36);
builder.Property(o => o.CreatedAt).HasColumnName("created_at")
.HasDefaultValueSql("NOW()");
builder.Property(o => o.ProcessedAt).HasColumnName("processed_at");
builder.Property(o => o.RetryCount).HasColumnName("retry_count")
.HasDefaultValue(0);
builder.Property(o => o.LastError).HasColumnName("last_error");
builder.Property(o => o.FailedAt).HasColumnName("failed_at");
// Partial index: only unprocessed, non-failed events need fast lookup
builder.HasIndex(o => o.CreatedAt)
.HasFilter("processed_at IS NULL AND failed_at IS NULL");
}
}
```
Each outbox event has:
| Column | Purpose |
|--------|---------|
| `id` | Unique identifier |
| `topic` | Which Kafka topic to publish to (e.g., `observation.recorded`) |
| `payload` | The JSON message body (stored as JSONB for compactness) |
| `partition_key` | Kafka partition key — typically the `encounterId`, ensuring all events for one encounter are ordered |
| `created_at` | When the event was written (used for ordering) |
| `processed_at` | Set to `NOW()` after successful Kafka publish — NULL means "not yet published" |
| `retry_count` | How many times the relay has tried and failed to publish this event |
| `last_error` | The error message from the most recent failed publish attempt |
| `failed_at` | Set when `retry_count` exceeds the maximum — marks the event as permanently failed |
The **partial index** on `created_at` only covers rows where `processed_at IS NULL AND failed_at IS NULL`. This keeps the index small and fast — once an event is processed, it's no longer in the index. The relay's query only touches unprocessed events.
---
## Writing Outbox Events (The Write Side)
Every time the application writes domain data that downstream consumers need to know about, it adds an outbox event in the same transaction. Here's the observation ingest path:
```csharp
// All of this happens inside a single PostgreSQL transaction
await using var tx = await _db.Database.BeginTransactionAsync();
// 1. Insert the observation
_db.Observations.Add(observation);
// 2. Insert an outbox event for the observation
_db.OutboxEvents.Add(BuildOutboxEvent("observation.recorded", new
{
observationId = observation.Id,
encounterId,
patientId = encounter.PatientId,
observationCode = req.ObservationCode,
value = req.Value,
unit = req.Unit,
recordedAt = req.RecordedAt,
partitionKey = encounterId.ToString()
}, encounterId.ToString()));
// 3. If critical threshold breach — also insert the alert + its outbox event
if (IsCriticalBreach(req.Value, threshold))
{
_db.ClinicalAlerts.Add(alert);
_db.OutboxEvents.Add(BuildOutboxEvent("alert.generated", new
{
alertId = alert.Id,
encounterId,
alertType = alert.AlertType.ToDbString(),
severity = alert.Severity.ToDbString(),
partitionKey = encounterId.ToString()
}, encounterId.ToString()));
}
// 4. COMMIT — observation, alert, and outbox events are all saved atomically
await _db.SaveChangesAsync();
await tx.CommitAsync();
```
If any step fails, the entire transaction rolls back — no orphaned events, no missing data.
---
## The Outbox Relay Service (The Read Side)
The `OutboxRelayService` is a background service that runs continuously, polling the `outbox_events` table for unprocessed events and publishing them to Kafka.
### The Idempotent Producer
```csharp
public override Task StartAsync(CancellationToken cancellationToken)
{
_producer = new ProducerBuilder<string, string>(new ProducerConfig
{
BootstrapServers = _options.BootstrapServers,
Acks = Acks.All,
EnableIdempotence = true,
MessageSendMaxRetries = 3,
RetryBackoffMs = 100
}).Build();
return base.StartAsync(cancellationToken);
}
```
**Why idempotent?** If the relay sends a message to Kafka but the acknowledgment is lost (network timeout), the relay retries. Without idempotency, Kafka would store the message twice. With `EnableIdempotence = true`, Kafka assigns the producer an internal ID and sequence number, recognizing and dropping duplicate deliveries.
### The Poll Loop
```csharp
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
while (!stoppingToken.IsCancellationRequested)
{
try
{
await ProcessBatchAsync(stoppingToken);
}
catch (Exception ex) when (ex is not OperationCanceledException)
{
_logger.LogError(ex, "Outbox relay error — will retry on next poll cycle");
}
await Task.Delay(_options.OutboxPollIntervalMs, stoppingToken); // 1000ms
}
}
```
The relay polls every second. If an error occurs (Kafka unreachable, database timeout), it logs the error and tries again on the next cycle. The `when (ex is not OperationCanceledException)` filter avoids logging expected shutdown cancellations as errors.
### Fetching and Locking Events
```csharp
private async Task ProcessBatchAsync(CancellationToken ct)
{
using var scope = _services.CreateScope();
var db = scope.ServiceProvider.GetRequiredService<AppDbContext>();
await using var tx = await db.Database.BeginTransactionAsync(ct);
var events = await db.OutboxEvents
.FromSqlRaw("""
SELECT id, topic, payload, partition_key, created_at, processed_at,
retry_count, last_error, failed_at
FROM outbox_events
WHERE processed_at IS NULL AND failed_at IS NULL
ORDER BY created_at ASC
LIMIT {0}
FOR UPDATE SKIP LOCKED
""", _options.OutboxBatchSize) // batch size: 100
.ToListAsync(ct);
if (events.Count == 0)
{
await tx.RollbackAsync(ct);
return;
}
```
**What is `FOR UPDATE SKIP LOCKED`?** This is a PostgreSQL feature for safe concurrent access:
- `FOR UPDATE` locks the selected rows so no other transaction can modify them until this transaction commits or rolls back
- `SKIP LOCKED` means "if another relay instance already locked some of these rows, skip them instead of waiting"
This lets you run multiple relay instances for higher throughput — each instance grabs a different batch of rows without blocking or duplicating work. Think of it like multiple cashiers at a grocery store, each serving the next available customer.
### Publishing Each Event
```csharp
foreach (var ev in events)
{
try
{
var result = await _producer!.ProduceAsync(
ev.Topic,
new Message<string, string>
{
Key = ev.PartitionKey ?? string.Empty,
Value = ev.Payload
}, ct);
ev.ProcessedAt = DateTimeOffset.UtcNow;
}
catch (ProduceException<string, string> ex)
{
ev.RetryCount++;
ev.LastError = ex.Error.Reason;
if (ev.RetryCount >= _options.OutboxMaxRetries) // 10 retries
{
ev.FailedAt = DateTimeOffset.UtcNow;
_logger.LogError(ex,
"Outbox event {Id} permanently failed after {Retries} retries",
ev.Id, ev.RetryCount);
}
break; // Stop batch on first failure
}
}
await db.SaveChangesAsync(ct);
await tx.CommitAsync(ct);
}
```
Key behaviors:
- **On success**: `ProcessedAt` is set. The event won't be picked up again (the WHERE clause filters it out).
- **On failure**: `RetryCount` is incremented and `LastError` is recorded. The event stays unprocessed and will be retried on the next poll cycle.
- **After max retries**: `FailedAt` is set, permanently marking the event as failed. This prevents an undeliverable message from blocking the entire queue forever. Failed events need manual investigation.
- **Break on first failure**: If one event fails to publish, the batch stops. This preserves ordering — events for the same encounter must be delivered in creation order.
### Some Events Route to RabbitMQ Instead
Not all outbox events go to Kafka. The relay checks the topic and routes accordingly:
```csharp
if (ev.Topic == ClinicalSyncOptions.BatchReceivedOutboxTopic)
{
_rabbitChannel!.BasicPublish(
exchange: _syncOpts.SyncExchange,
routingKey: _syncOpts.SyncBatchReceivedRoutingKey,
basicProperties: _rabbitProps,
body: Encoding.UTF8.GetBytes(ev.Payload));
}
else
{
await _producer!.ProduceAsync(ev.Topic, new Message<string, string>
{
Key = ev.PartitionKey ?? string.Empty,
Value = ev.Payload
}, ct);
}
```
This keeps the outbox pattern universal — any downstream message, whether Kafka or RabbitMQ, goes through the same transactional guarantee.
---
## Monitoring the Outbox
Two monitoring mechanisms track outbox health:
### OutboxPendingCollector (Prometheus gauge)
Every 30 seconds, counts how many events haven't been published yet:
```csharp
var count = await db.OutboxEvents
.CountAsync(e => e.ProcessedAt == null, ct);
_metrics.OutboxPendingEvents.Set(count);
```
A rising `outbox_pending_events` gauge on the Grafana dashboard means the relay is falling behind or Kafka is unreachable.
### Outbox Relay Logs
The relay logs each batch:
```csharp
_logger.LogInformation(
"Outbox relay published {Count} events. Failed={Failed}",
published.Count, hadFailure);
```
---
## The Complete Flow
```
1. HTTP Request arrives
POST /api/encounters/{id}/observations
2. ObservationService.IngestAsync()
┌─── PostgreSQL Transaction ───────────────────┐
│ INSERT INTO observations (...) │
│ INSERT INTO outbox_events (topic='obs.rec.') │
│ IF critical: │
│ INSERT INTO clinical_alerts (...) │
│ INSERT INTO outbox_events (topic='alert.') │
│ COMMIT │
└──────────────────────────────────────────────┘
3. HTTP Response returned to caller (201 Created)
The caller doesn't wait for Kafka — it's decoupled.
4. OutboxRelayService (1 second later)
┌─── Poll cycle ──────────────────────────────┐
│ SELECT FROM outbox_events FOR UPDATE SKIP.. │
│ ProduceAsync to Kafka (observation.recorded) │
│ ProduceAsync to Kafka (alert.generated) │
│ SET processed_at = NOW() │
│ COMMIT │
└──────────────────────────────────────────────┘
5. Kafka delivers to 9 consumer groups
es-indexer, sepsis-engine, news2-scoring, etc.
```
---
## Key Design Decisions
### Why Poll Instead of Change Data Capture?
Some implementations use PostgreSQL's logical replication or a CDC (Change Data Capture) tool like Debezium to stream outbox rows to Kafka. This project uses simple polling because:
- Polling is straightforward to implement and debug
- The `OutboxPollIntervalMs` (1 second) is fast enough for clinical use cases
- No additional infrastructure (Debezium connector, separate process) is needed
- The `FOR UPDATE SKIP LOCKED` pattern handles concurrency cleanly
### Why Break on First Failure?
If event A and event B are for the same encounter, they must arrive at Kafka in order (A before B). If event A fails to publish and we skip it to publish B, consumers would see B first — which could cause incorrect scoring or duplicate alerts. Breaking on first failure maintains ordering at the cost of potentially delaying later events.
### Why a Permanent Failure State?
After 10 retries, an event is marked `FailedAt` and excluded from future relay cycles. Without this, a single undeliverable event (e.g., payload too large for Kafka) would block the entire outbox forever. Failed events are visible in the database and Prometheus metrics for investigation.
---
## Key Takeaways
- **The outbox pattern guarantees atomicity** across database writes and message publishing by keeping both in the same PostgreSQL transaction.
- **The relay is eventually consistent** — there's a short delay (up to `OutboxPollIntervalMs`) between the database commit and Kafka delivery. For this project, 1 second is clinically acceptable.
- **Idempotent producers handle retry duplicates** — network timeouts that cause re-delivery are silently deduplicated by Kafka.
- **`FOR UPDATE SKIP LOCKED` enables horizontal scaling** — multiple relay instances can run without coordination.
- **Retry tracking prevents infinite loops** — events that persistently fail are marked and excluded after a configurable number of attempts.
- **The pattern applies to any message broker** — the same outbox row can be published to Kafka, RabbitMQ, or any other system. The transactional guarantee is with the database, not with a specific broker.
docs/guides/11-cqrs-command-query-separation.md
+219
View File
@@ -0,0 +1,219 @@
# Guide 11: CQRS (Command Query Responsibility Segregation)
## What is CQRS?
**CQRS** stands for **Command Query Responsibility Segregation**. It's an architectural pattern where you use **different models** (and often different databases) for writing data versus reading data.
In a traditional application, you have one database that handles everything:
```
┌─────────────┐ read + write ┌────────────┐
│ Application│ ◄──────────────────► │ PostgreSQL │
└─────────────┘ └────────────┘
```
With CQRS, you split the read and write paths:
```
┌─────────────┐ write ┌────────────┐
│ API writes │ ───────────────────► │ PostgreSQL │ (normalized, transactional)
└─────────────┘ └──────┬─────┘
│ events (via Kafka)
▼
┌─────────────┐ read ┌───────────────┐
│ Dashboard │ ◄────────────────── │ Elasticsearch │ (denormalized, searchable)
│ queries │ └───────────────┘
└─────────────┘
```
### Why Separate Reads and Writes?
The ideal data structure for writing is different from the ideal structure for reading:
**Writing needs**:
- Normalized tables (no duplicated data) to prevent inconsistencies
- Foreign keys and constraints to enforce business rules
- ACID transactions to guarantee atomicity
- Example: A patient's name is stored once in the `patients` table, referenced by ID everywhere else
**Reading needs**:
- Denormalized documents (all related data in one place) to avoid expensive JOINs
- Full-text search and fuzzy matching
- Fast aggregations (counts, averages by department)
- Example: A single document contains the patient name, encounter status, department, alert count, NEWS2 score — no JOINs needed
You can't optimize for both in one database. A normalized PostgreSQL schema is great for writes but requires multiple JOINs for complex reads. An Elasticsearch index is great for reads but has no foreign keys, no transactions, and no constraint enforcement.
### The Event Bridge
The write side and read side are connected by **events**. When the write side changes data, it publishes an event. A consumer on the read side processes that event and updates its own data store. This makes the read side **eventually consistent** — there's a short delay (usually under a second) between a write and the data appearing on the read side.
---
## How CQRS Works in This Project
```
WRITE PATH READ PATH
────────── ─────────
HTTP Request Dashboard / API
│ │
▼ ▼
┌──────────────┐ ┌────────────────┐
│ Controller │ │ Analytics │
│ │ │ Controller │
│ POST /obs │ │ GET /search │
│ POST /alerts │ │ GET /analytics │
└──────┬───────┘ └────────┬───────┘
│ │
▼ ▼
┌──────────────┐ ┌────────────────┐
│ Service │ │ Analytics │
│ Layer │ │ Service │
│ │ │ │
│ Validates, │ │ Queries ES │
│ writes to DB │ │ (no JOINs) │
└──────┬───────┘ └────────┬───────┘
│ │
▼ ▼
┌──────────────┐ outbox ┌─────┐ es-indexer ┌────────────────┐
│ PostgreSQL │──────────────►│Kafka│─────────────►│ Elasticsearch │
│ │ relay │ │ │ │
│ (source of │ └─────┘ │ patient_enc. │
│ truth) │ │ observations │
└──────────────┘ │ clinical_alerts│
└────────────────┘
```
### The Write Path (Commands)
"Commands" are operations that change state — creating patients, recording observations, acknowledging alerts. They go through the service layer and write to PostgreSQL:
1. **Validate** the request (FluentValidation)
2. **Apply business logic** (threshold checks, scoring)
3. **Write to PostgreSQL** (the source of truth) in a transaction
4. **Write outbox events** (in the same transaction) for downstream consumers
5. **Return the response** — the caller doesn't wait for Elasticsearch
### The Read Path (Queries)
"Queries" are operations that read state without changing it — searching patients, viewing analytics, getting alert summaries. They read from Elasticsearch:
1. **Build an Elasticsearch query** (filters, full-text search, aggregations)
2. **Execute against Elasticsearch** — one network call returns a denormalized document with all the data
3. **Return the results** — no JOINs, no multiple database calls
### The Bridge: EsIndexerService
The `EsIndexerService` Kafka consumer reads events from the write side and updates Elasticsearch:
| Kafka Topic | ES Operation | What Happens |
|-------------|-------------|-------------|
| `encounter.status.changed` | Upsert `patient_encounters` | Creates or updates the encounter document with patient name, department, status |
| `observation.recorded` | Index `observations` + update `patient_encounters` | Appends the observation and updates `lastObservationAt` on the encounter |
| `alert.generated` | Index `clinical_alerts` + update `patient_encounters` | Appends the alert and increments `openAlertCount` on the encounter |
| `sepsis.bundle.created` | Update `patient_encounters` | Stamps bundle status on the encounter document |
| `sepsis.bundle.updated` | Update `patient_encounters` | Updates bundle compliance status |
Each handler is **idempotent** — processing the same event twice produces the same result, so retries after failures don't corrupt data.
---
## Denormalization in Practice
In PostgreSQL (normalized), getting a ward overview requires joining 4+ tables:
```sql
-- Normalized: multiple JOINs
SELECT e.id, p.first_name, p.last_name, p.mrn,
e.department, e.status,
COUNT(a.id) as open_alerts,
MAX(o.recorded_at) as last_observation
FROM encounters e
JOIN patients p ON e.patient_id = p.id
LEFT JOIN clinical_alerts a ON a.encounter_id = e.id AND a.status = 'OPEN'
LEFT JOIN observations o ON o.encounter_id = e.id
WHERE e.status = 'Active'
GROUP BY e.id, p.first_name, p.last_name, p.mrn, e.department, e.status;
```
In Elasticsearch (denormalized), it's one query with no JOINs:
```json
{
"query": { "term": { "status": "Active" } },
"sort": [{ "openAlertCount": "desc" }]
}
```
Each `patient_encounters` document already contains:
```json
{
"encounterId": "...",
"patientName": "John Smith",
"mrn": "MRN-001",
"department": "ICU",
"status": "Active",
"openAlertCount": 3,
"lastObservationAt": "2026-06-24T14:23:00Z",
"news2Score": 7,
"news2RiskLevel": "HIGH",
"sepsisBundleStatus": "IN_PROGRESS"
}
```
The tradeoff: the ES indexer must keep these denormalized fields in sync whenever the source data changes. Every alert generated increments `openAlertCount`. Every observation updates `lastObservationAt`. Every NEWS2 score updates `news2Score` and `news2RiskLevel`. This is more work on the write side, but it makes the read side fast and simple.
---
## When Data Lives in Which Store
| Data Need | Store | Why |
|-----------|-------|-----|
| Record a new observation | PostgreSQL | Needs transactions, constraints, outbox |
| Acknowledge an alert | PostgreSQL | Needs atomic status update, audit log |
| Search patients by name/MRN | Elasticsearch | Full-text search, relevance scoring |
| Filter encounters by department | Elasticsearch | Fast keyword filtering on denormalized docs |
| Alert volume by department | Elasticsearch | Aggregations across millions of documents |
| Observation trend over time | Elasticsearch | Date histogram aggregations |
| Compute NEWS2/SOFA score | PostgreSQL (via Redis) | Needs transactional alert creation |
| Generate discharge summary | PostgreSQL | Needs authoritative patient/encounter data |
---
## Eventual Consistency
**What does "eventually consistent" mean in practice?** When a nurse records a vital sign:
1. **t=0ms**: PostgreSQL has the observation (write committed)
2. **t=0ms**: HTTP response returned to the nurse (she sees "saved")
3. **t=1000ms**: Outbox relay publishes to Kafka
4. **t=1100ms**: ES indexer consumer processes the event
5. **t=1100ms**: Elasticsearch has the observation (read store updated)
During the ~1 second between steps 2 and 5, the dashboard (reading from Elasticsearch) doesn't yet show the new observation. In practice, the dashboard polls every 5-10 seconds, so the delay is imperceptible.
**What if the read store is wrong?** Elasticsearch is never the source of truth. If it gets corrupted or out of sync, you can rebuild it entirely by resetting the `es-indexer` consumer group's Kafka offset to zero and replaying all events from the beginning.
---
## Where the Pattern Components Live
| Component | File | Role |
|-----------|------|------|
| Write side (domain logic) | `Services/ObservationService.cs` | Validates, writes to PostgreSQL + outbox |
| Outbox entity | `Data/Configurations/OutboxEventConfiguration.cs` | Defines the `outbox_events` table |
| Event relay | `BackgroundServices/OutboxRelayService.cs` | Polls outbox, publishes to Kafka |
| Event bridge | `BackgroundServices/ElasticsSearch/EsIndexerService.cs` | Kafka → Elasticsearch projections |
| Read side (queries) | `Services/AnalyticsService.cs` | Queries Elasticsearch for search/analytics |
| Index setup | `BackgroundServices/ElasticsSearch/ElasticIndexProvisioner.cs` | Creates ES indexes on startup |
---
## Key Takeaways
- **CQRS is about using the right tool for each job** — PostgreSQL for writes (transactions, constraints), Elasticsearch for reads (search, aggregations).
- **Events connect the two sides** — the outbox pattern guarantees events are published, and Kafka consumers update the read store.
- **Denormalization trades write complexity for read performance** — updating `openAlertCount` on every alert is extra work, but it eliminates JOINs on every dashboard refresh.
- **Eventual consistency is the tradeoff** — there's a short delay between a write and the data appearing in the read store. For dashboards that poll every few seconds, this is invisible.
- **The read store is rebuildable** — if Elasticsearch data gets corrupted, reset the consumer offset and replay all events. PostgreSQL is always the source of truth.
- **You don't need CQRS everywhere** — simple CRUD endpoints that don't need search or aggregations can read directly from PostgreSQL. CQRS adds complexity, so only use it where the read and write requirements genuinely differ.
docs/guides/12-event-driven-background-services.md
+451
View File
@@ -0,0 +1,451 @@
# Guide 12: Event-Driven Background Services in .NET
## What is a Background Service?
In a web API, most code runs in response to HTTP requests — a request arrives, your code processes it, a response goes back. But many tasks need to run continuously _without_ a request triggering them:
- Polling the outbox table every second to relay events to Kafka
- Consuming Kafka messages to compute clinical scores
- Checking every 30 seconds whether any critical alerts have gone unacknowledged
- Scanning every 5 minutes for overdue sepsis bundles
In .NET, these long-running tasks are called **background services**. They start when the application starts, run continuously in the background, and stop when the application shuts down.
.NET provides two base classes for this:
- **`IHostedService`**: Has `StartAsync` (called once when the app starts) and `StopAsync` (called once when the app stops). Good for one-time initialization tasks.
- **`BackgroundService`**: Extends `IHostedService` with an `ExecuteAsync` method that runs for the lifetime of the application. Good for continuous processing loops.
---
## Why Background Services in This Project?
VigilCareClinical has 25+ background services running inside the API process. They handle everything from event relay to clinical scoring to metrics collection. Without them, the API would only be able to do work when an HTTP request arrives — and most of the important work (computing NEWS2 scores, detecting sepsis, escalating alerts) happens asynchronously.
---
## Registration in Program.cs
Every background service is registered with dependency injection in `Program.cs`:
```csharp
// One-time initialization services
builder.Services.AddHostedService<ThresholdCacheLoader>();
builder.Services.AddHostedService<KafkaTopicProvisioner>();
builder.Services.AddHostedService<ElasticIndexProvisioner>();
builder.Services.AddHostedService<PatientPhiMigrationService>();
// Event relay
builder.Services.AddHostedService<OutboxRelayService>();
// Kafka consumers (clinical scoring engines)
builder.Services.AddHostedService<SepsisEngineService>();
builder.Services.AddHostedService<News2ScoringService>();
builder.Services.AddHostedService<GcsScoringService>();
builder.Services.AddHostedService<SofaScoringService>();
builder.Services.AddHostedService<TrendAnalyzerService>();
builder.Services.AddHostedService<WarningAlertService>();
// Kafka consumers (infrastructure)
builder.Services.AddHostedService<EsIndexerService>();
builder.Services.AddHostedService<DataLakeWriterService>();
builder.Services.AddHostedService<NotificationPublisherService>();
// RabbitMQ consumers
builder.Services.AddHostedService<PagingWorkerService>();
builder.Services.AddHostedService<EscalationWorkerService>();
builder.Services.AddHostedService<DischargeSummaryWorkerService>();
// Periodic scanners
builder.Services.AddHostedService<ReconciliationScheduler>();
builder.Services.AddHostedService<SepsisBundleMonitorService>();
builder.Services.AddHostedService<GatewayStaleDetectorService>();
builder.Services.AddHostedService<AlertQualityAggregatorService>();
// Metrics collectors
builder.Services.AddHostedService<AlertsUnacknowledgedCollector>();
builder.Services.AddHostedService<OutboxPendingCollector>();
builder.Services.AddHostedService<KafkaConsumerLagCollector>();
builder.Services.AddHostedService<WardGatewayMetricsCollector>();
```
`AddHostedService<T>()` tells .NET: "create an instance of this class and call its `StartAsync`/`ExecuteAsync` when the application starts." All hosted services run concurrently within the same process.
---
## The Four Patterns
Every background service in this project follows one of four patterns. Understanding these patterns makes it easy to read any service's code.
### Pattern 1: One-Time Initializer (IHostedService)
These services run once at startup and then stop. They prepare the system for operation.
```csharp
public class ThresholdCacheLoader : IHostedService
{
public async Task StartAsync(CancellationToken cancellationToken)
{
// Load all alert thresholds from PostgreSQL into Redis
var thresholds = await db.AlertThresholds.ToListAsync(cancellationToken);
for (var attempt = 0; attempt < 3; attempt++)
{
try
{
var batch = cache.CreateBatch();
foreach (var t in thresholds)
_ = batch.StringSetAsync($"threshold:{t.ObservationCode}", json);
batch.Execute();
return; // success — done
}
catch (RedisException ex)
{
_logger.LogWarning(ex, "Redis unavailable — attempt {Attempt}/3",
attempt + 1);
await Task.Delay(backoffMs[attempt], cancellationToken);
}
}
_logger.LogError("Failed to load thresholds — app will start without cache");
}
public Task StopAsync(CancellationToken cancellationToken) => Task.CompletedTask;
}
```
Other initializers:
- `KafkaTopicProvisioner` — creates Kafka topics with 6 partitions
- `ElasticIndexProvisioner` — creates Elasticsearch indexes with mappings
- `RabbitMqTopologyProvisioner` — declares exchanges, queues, and bindings
- `PatientPhiMigrationService` — one-time migration to add search tokens to existing patients
**Key characteristic**: `StartAsync` runs to completion, then the service does nothing until shutdown. The work is done during startup, before the application starts accepting HTTP requests.
### Pattern 2: Kafka Consumer (BackgroundService)
These services consume messages from Kafka topics and process them. They're the core of the event-driven architecture.
```csharp
public class SepsisEngineService : BackgroundService
{
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
// 1. Configure the consumer
var config = new ConsumerConfig
{
BootstrapServers = _kafkaOptions.BootstrapServers,
GroupId = "sepsis-engine",
AutoOffsetReset = AutoOffsetReset.Earliest,
EnableAutoCommit = false
};
using var consumer = new ConsumerBuilder<string, string>(config).Build();
consumer.Subscribe(_kafkaOptions.Topics.ObservationRecorded);
// 2. Create a poison pill guard
var guard = new PoisonPillGuard("sepsis-engine",
_kafkaOptions.MaxPoisonRetries, _logger);
try
{
// 3. Consume loop — runs for the lifetime of the application
while (!stoppingToken.IsCancellationRequested)
{
ConsumeResult<string, string>? result = null;
try
{
result = consumer.Consume(stoppingToken);
// 4. Process the message
var evt = JsonSerializer.Deserialize<SepsisObservationEvent>(
result.Message.Value)!;
using var scope = _services.CreateScope();
var detector = scope.ServiceProvider
.GetRequiredService<QsofaDetector>();
await detector.ProcessObservationAsync(
evt.EncounterId, evt.PatientId,
evt.ObservationCode, evt.Value, stoppingToken);
// 5. Commit offset (tell Kafka "I'm done with this message")
consumer.Commit(result);
guard.OnSuccess();
}
catch (OperationCanceledException) { break; }
catch (Exception ex)
{
// 6. Poison pill handling
if (result is not null && guard.ShouldSkip(result, ex))
{
consumer.Commit(result);
continue;
}
_logger.LogError(ex, "SepsisEngine failed — will retry");
await Task.Delay(2000, stoppingToken);
}
}
}
finally
{
consumer.Close(); // 7. Clean group leave on shutdown
}
}
}
```
Other Kafka consumers follow the same structure: `News2ScoringService`, `GcsScoringService`, `SofaScoringService`, `TrendAnalyzerService`, `WarningAlertService`, `EsIndexerService`, `DataLakeWriterService`, `NotificationPublisherService`.
**Key characteristics**:
- Infinite `while` loop broken only by `CancellationToken`
- Manual Kafka offset commit after successful processing
- `PoisonPillGuard` prevents stuck consumers
- DI scope created per message (database contexts are scoped, not singleton)
- `consumer.Close()` in `finally` for clean group leave
### Pattern 3: Periodic Timer (BackgroundService)
These services wake up on a fixed schedule, do some work, then go back to sleep. They monitor system state and collect metrics.
```csharp
public sealed class GatewayStaleDetectorService : BackgroundService
{
protected override async Task ExecuteAsync(CancellationToken ct)
{
// PeriodicTimer fires at a fixed interval
using var timer = new PeriodicTimer(
TimeSpan.FromMinutes(_opts.PollIntervalMinutes)); // 5 minutes
while (await timer.WaitForNextTickAsync(ct))
await DetectStaleAsync(ct);
}
private async Task DetectStaleAsync(CancellationToken ct)
{
// Create a DI scope for each tick
await using var scope = _scopes.CreateAsyncScope();
var db = scope.ServiceProvider.GetRequiredService<AppDbContext>();
var cutoff = DateTimeOffset.UtcNow.AddMinutes(-_opts.StaleThresholdMinutes);
var stale = await db.WardGateways
.Where(g => g.Status != GatewayStatus.Offline
&& g.LastHeartbeatAt < cutoff)
.ToListAsync(ct);
foreach (var gateway in stale)
{
gateway.MarkOffline();
_logger.LogWarning("Gateway {Code} marked OFFLINE", gateway.GatewayCode);
}
if (stale.Count > 0)
await db.SaveChangesAsync(ct);
}
}
```
Other periodic services:
- `AlertsUnacknowledgedCollector` — every 30 seconds, counts open critical alerts
- `OutboxPendingCollector` — every 30 seconds, counts pending outbox events
- `KafkaConsumerLagCollector` — every 30 seconds, checks Kafka consumer lag
- `WardGatewayMetricsCollector` — every 60 seconds, reports gateway status
- `ReconciliationScheduler` — every 30 minutes, runs safety checks
- `SepsisBundleMonitorService` — every 5 minutes, marks overdue bundles
- `AlertQualityAggregatorService` — every 60 minutes, computes alert quality metrics
**Key characteristics**:
- `PeriodicTimer` or `Task.Delay` for the interval
- DI scope created per tick (fresh database context each time)
- Errors are caught and logged, not propagated (the service continues to the next tick)
- No message consumption — these services query the database directly
**`PeriodicTimer` vs `Task.Delay`**: Both work for periodic execution. `PeriodicTimer` (introduced in .NET 6) is slightly more precise because it accounts for the time spent doing work — if your work takes 2 seconds and the interval is 30 seconds, the next tick fires 28 seconds after the work finishes, maintaining a true 30-second cadence. `Task.Delay` would wait 30 seconds _after_ the work finishes, making the actual interval 32 seconds.
### Pattern 4: RabbitMQ Consumer (BackgroundService)
These services consume messages from RabbitMQ queues using the event-driven consumer model (push-based, not pull-based like Kafka):
```csharp
public sealed class PagingWorkerService : BackgroundService
{
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
// 1. Wait for topology to be ready
await Task.Delay(TimeSpan.FromSeconds(5), stoppingToken);
// 2. Create connection and channel
var factory = RabbitMqConnectionFactory.Create(o, dispatchConsumersAsync: true);
using var connection = factory.CreateConnection("paging-worker");
using var channel = connection.CreateModel();
// 3. Set prefetch (how many unacked messages at once)
channel.BasicQos(prefetchSize: 0, prefetchCount: 1, global: false);
// 4. Register an async event handler
var consumer = new AsyncEventingBasicConsumer(channel);
consumer.Received += async (sender, ea) =>
{
await HandlePageAsync(channel, ea, stoppingToken);
};
// 5. Start consuming
channel.BasicConsume("alerts.paging.queue", autoAck: false, consumer);
// 6. Block until shutdown
await Task.Delay(Timeout.Infinite, stoppingToken);
}
}
```
**Why `Task.Delay(Timeout.Infinite)`?** Unlike Kafka consumers (which pull messages in a loop), RabbitMQ consumers are push-based — RabbitMQ delivers messages to the `Received` event handler. The `ExecuteAsync` method just needs to stay alive (not return) so the connection and event handler remain active. `Task.Delay(Timeout.Infinite, stoppingToken)` blocks forever until the cancellation token is triggered by application shutdown.
Other RabbitMQ consumers: `EscalationWorkerService`, `DischargeSummaryWorkerService`, `ClinicalSyncBatchConsumer`.
---
## The DI Scope Problem
**Why do background services need to create scopes?**
In .NET dependency injection, services have different lifetimes:
- **Singleton**: One instance for the entire application
- **Scoped**: One instance per "scope" (in a web app, one per HTTP request)
- **Transient**: A new instance every time it's requested
`AppDbContext` (the database context) is registered as **scoped** — each HTTP request gets its own instance to avoid thread-safety issues and stale data. But background services are **singletons** — they're created once and live forever.
If a background service tries to inject a scoped service directly, .NET throws an error. The solution: create a new scope for each unit of work:
```csharp
// Wrong — would fail because BackgroundService is a singleton
public class MyService : BackgroundService
{
private readonly AppDbContext _db; // scoped — can't inject into singleton
}
// Correct — create a scope per tick/message
public class MyService : BackgroundService
{
private readonly IServiceScopeFactory _scopes;
private async Task DoWorkAsync(CancellationToken ct)
{
await using var scope = _scopes.CreateAsyncScope();
var db = scope.ServiceProvider.GetRequiredService<AppDbContext>();
// use db within this scope...
} // scope is disposed, db is disposed
}
```
The scope acts like a mini HTTP request — it creates and disposes the database context cleanly.
---
## Startup Ordering
Background services start in the order they're registered. Services that provision infrastructure run first:
```csharp
// Phase 1: Infrastructure provisioning (these must complete first)
builder.Services.AddHostedService<ThresholdCacheLoader>(); // Redis cache
builder.Services.AddHostedService<KafkaTopicProvisioner>(); // Kafka topics
builder.Services.AddHostedService<ElasticIndexProvisioner>(); // ES indexes
// RabbitMqTopologyProvisioner runs as a singleton, registered separately
// Phase 2: Event processing (depends on Phase 1)
builder.Services.AddHostedService<OutboxRelayService>(); // needs Kafka topics
builder.Services.AddHostedService<EsIndexerService>(); // needs ES indexes
builder.Services.AddHostedService<SepsisEngineService>(); // needs Kafka topics
// Phase 3: Notification workers (depends on RabbitMQ topology)
builder.Services.AddHostedService<PagingWorkerService>();
builder.Services.AddHostedService<EscalationWorkerService>();
```
Some services add explicit delays to wait for infrastructure:
```csharp
// PagingWorkerService — wait 5 seconds for RabbitMQ topology provisioning
await Task.Delay(TimeSpan.FromSeconds(5), stoppingToken);
// KafkaConsumerLagCollector — wait 20 seconds for Kafka to be reachable
await Task.Delay(TimeSpan.FromSeconds(20), ct);
```
This is a pragmatic approach for a single-process application. In a microservices architecture, you'd use health checks and readiness probes instead.
---
## Graceful Shutdown
When the application shuts down (e.g., `Ctrl+C` or a deployment), .NET cancels the `CancellationToken` passed to each service. Well-behaved services respond to this:
**Kafka consumers** close cleanly:
```csharp
finally
{
consumer.Close(); // tells Kafka broker to rebalance partitions immediately
}
```
**RabbitMQ consumers** requeue in-flight messages:
```csharp
catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested)
{
channel.BasicNack(ea.DeliveryTag, multiple: false, requeue: true);
}
```
**The data lake writer** flushes buffered events:
```csharp
finally
{
if (_buffer.Values.Sum(v => v.Count) > 0)
await FlushAsync(consumer, CancellationToken.None);
consumer.Close();
}
```
---
## Background Service Summary
| Service | Pattern | Interval/Trigger | Purpose |
|---------|---------|-----------------|---------|
| `ThresholdCacheLoader` | Initializer | Once at startup | Pre-load Redis cache |
| `KafkaTopicProvisioner` | Initializer | Once at startup | Create Kafka topics |
| `ElasticIndexProvisioner` | Initializer | Once at startup | Create ES indexes |
| `PatientPhiMigrationService` | Initializer | Once at startup | Migrate PHI search tokens |
| `OutboxRelayService` | Periodic | Every 1 second | Outbox → Kafka relay |
| `SepsisEngineService` | Kafka consumer | Per message | qSOFA screening |
| `News2ScoringService` | Kafka consumer | Per message | NEWS2 score computation |
| `GcsScoringService` | Kafka consumer | Per message | GCS aggregation |
| `SofaScoringService` | Kafka consumer | Per message | SOFA organ scoring |
| `TrendAnalyzerService` | Kafka consumer | Per message | Rate-of-change detection |
| `WarningAlertService` | Kafka consumer | Per message | Warning threshold alerts |
| `EsIndexerService` | Kafka consumer | Per message | CQRS projection to ES |
| `DataLakeWriterService` | Kafka consumer | Buffered flush | Parquet files to MinIO |
| `NotificationPublisherService` | Kafka consumer | Per message | Kafka → RabbitMQ bridge |
| `PagingWorkerService` | RabbitMQ consumer | Per message | Page physician, wait for ACK |
| `EscalationWorkerService` | RabbitMQ consumer | Per message | Escalate unacked alerts |
| `DischargeSummaryWorkerService` | RabbitMQ consumer | Per message | Generate discharge PDFs |
| `ClinicalSyncBatchConsumer` | RabbitMQ consumer | Per message | Process gateway sync batches |
| `ReconciliationScheduler` | Periodic timer | Every 30 minutes | Safety checks |
| `SepsisBundleMonitorService` | Periodic timer | Every 5 minutes | Mark overdue bundles |
| `GatewayStaleDetectorService` | Periodic timer | Every 5 minutes | Detect offline gateways |
| `AlertQualityAggregatorService` | Periodic timer | Every 60 minutes | Compute quality metrics |
| `AlertsUnacknowledgedCollector` | Periodic timer | Every 30 seconds | Prometheus gauge |
| `OutboxPendingCollector` | Periodic timer | Every 30 seconds | Prometheus gauge |
| `KafkaConsumerLagCollector` | Periodic timer | Every 30 seconds | Prometheus gauge |
| `WardGatewayMetricsCollector` | Periodic timer | Every 60 seconds | Prometheus gauge |
---
## Key Takeaways
- **Background services are how .NET applications do work outside HTTP requests** — consuming messages, polling databases, collecting metrics.
- **Four patterns cover all use cases**: one-time initializers, Kafka consumers, periodic timers, and RabbitMQ consumers. Learn these four and you can read any service in the codebase.
- **Always create a DI scope per unit of work** — background services are singletons, but database contexts are scoped. The `IServiceScopeFactory` pattern bridges this gap.
- **Handle errors gracefully** — catch, log, and continue to the next cycle/message. A transient database timeout should not kill a background service permanently.
- **Respond to cancellation** — when the application shuts down, clean up resources: close Kafka consumers, requeue RabbitMQ messages, flush buffers.
- **Startup ordering matters** — provisioners (topics, indexes, topology) must run before consumers that depend on them. Use registration order and startup delays.
docs/guides/13-jwt-authentication.md
+284
View File
@@ -0,0 +1,284 @@
# Guide 13: JWT Authentication in ASP.NET Core
## What is JWT Authentication?
**Authentication** answers the question "who are you?" Before your API processes a request, it needs to verify the caller's identity — is this really Dr. Smith, or is someone pretending to be her?
**JWT** (JSON Web Token, pronounced "jot") is one of the most common ways to authenticate API requests. Here's how it works:
1. The user sends their username and password to a login endpoint
2. The server verifies the credentials and creates a **token** — a long string that encodes the user's identity
3. The server sends the token back to the client (e.g., a browser or mobile app)
4. On every subsequent request, the client sends the token in the `Authorization` header
5. The server validates the token and extracts the user's identity from it
A JWT has three parts separated by dots: `header.payload.signature`
```
eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4ifQ.SflKxwRJSMeKKF2QT4fwpM...
```
- **Header**: Metadata (which signing algorithm is used)
- **Payload**: The actual data — user ID, role, expiration time, etc. These key-value pairs are called **claims**
- **Signature**: A cryptographic hash of the header + payload, signed with a secret key. If anyone tampers with the payload (e.g., changes the role from "Nurse" to "Admin"), the signature won't match and the server rejects the token
**Why JWT instead of sessions?** With session-based auth, the server stores session data in memory or a database — every request requires a server-side lookup. With JWT, all the information is embedded in the token itself. The server just validates the signature — no database query needed. This makes JWT ideal for stateless APIs.
---
## How JWT Works in This Project
```
┌──────────┐ POST /api/auth/login ┌──────────────┐
│ Dashboard│ ──────────────────────────► │ AuthService │
│ (Vue.js) │ { username, password } │ │
│ │ │ 1. Verify pwd │
│ │ ◄────────────────────────── │ 2. Generate │
│ │ { token, expires, ... } │ JWT token │
└──────────┘ └──────────────┘
│
│ Authorization: Bearer eyJhbG...
▼
┌──────────────┐ validate token ┌──────────────────┐
│ Any API │ ◄───────────────── │ JwtBearer │
│ Endpoint │ extract claims │ Middleware │
│ │ │ (checks sig, │
│ Knows: who │ │ expiry, issuer) │
│ the user is │ └──────────────────┘
└──────────────┘
```
---
## Step 1: Configuration
### JwtOptions
```csharp
public class JwtOptions
{
public const string Section = "Jwt";
public string Issuer { get; set; } = "VigilCareClinical";
public string Audience { get; set; } = "VigilCareClinical.Dashboard";
public string SigningKey { get; set; } = null!;
public int ExpirationMinutes { get; set; } = 480; // 8 hours
}
```
| Setting | Purpose |
|---------|---------|
| `Issuer` | Who created the token — validated on every request to ensure the token came from this server |
| `Audience` | Who the token is intended for — prevents a token meant for a different service from being accepted |
| `SigningKey` | The secret key used to sign and verify tokens. Must be at least 256 bits (32 bytes) for HMAC-SHA256 |
| `ExpirationMinutes` | How long the token is valid. After 480 minutes (8 hours), the token is rejected and the user must log in again |
### appsettings.json
```json
{
"Jwt": {
"Issuer": "VigilCareClinical",
"Audience": "VigilCareClinical.Dashboard",
"SigningKey": "DEV-ONLY-REPLACE-WITH-256-BIT-SECRET-IN-PRODUCTION-abc123xyz",
"ExpirationMinutes": 480
}
}
```
**Security note**: The signing key in `appsettings.json` is for development only. In production, this would come from an environment variable or a secret manager (like Azure Key Vault or AWS Secrets Manager), never from a file committed to version control.
### Startup Validation
The application fails fast if the signing key is missing or too short:
```csharp
var jwtOptions = builder.Configuration.GetSection(JwtOptions.Section).Get<JwtOptions>()!;
if (string.IsNullOrWhiteSpace(jwtOptions.SigningKey)
|| Encoding.UTF8.GetByteCount(jwtOptions.SigningKey) < 32)
throw new InvalidOperationException(
"Jwt:SigningKey must be configured and at least 256 bits (32 bytes) for HMAC-SHA256.");
```
This prevents the application from starting with an insecure key. HMAC-SHA256 requires at least 256 bits — anything shorter is cryptographically weak.
---
## Step 2: Registering JWT Bearer Authentication
In `Program.cs`:
```csharp
builder.Services.AddAuthentication(JwtBearerDefaults.AuthenticationScheme)
.AddJwtBearer(options =>
{
options.TokenValidationParameters = new TokenValidationParameters
{
ValidateIssuer = true,
ValidateAudience = true,
ValidateLifetime = true,
ValidateIssuerSigningKey = true,
ValidIssuer = jwtOptions.Issuer,
ValidAudience = jwtOptions.Audience,
IssuerSigningKey = new SymmetricSecurityKey(
Encoding.UTF8.GetBytes(jwtOptions.SigningKey))
};
});
```
**What does each validation flag do?**
| Flag | What It Checks | What Happens If It Fails |
|------|---------------|------------------------|
| `ValidateIssuer` | Token's `iss` claim matches `ValidIssuer` | Rejects tokens from other servers |
| `ValidateAudience` | Token's `aud` claim matches `ValidAudience` | Rejects tokens meant for other services |
| `ValidateLifetime` | Token hasn't expired (current time < `exp` claim) | Forces re-login after 8 hours |
| `ValidateIssuerSigningKey` | The signature matches the configured key | Rejects tampered or forged tokens |
**What is `SymmetricSecurityKey`?** In symmetric cryptography, the same key is used to both sign and verify. The server uses this key to create the signature when generating the token, and to verify the signature when validating incoming tokens. This is simpler than asymmetric (public/private key) cryptography but requires the key to remain secret.
The middleware is activated later in the pipeline:
```csharp
app.UseAuthentication(); // Reads the token, validates it, sets HttpContext.User
app.UseAuthorization(); // Checks if the authenticated user has the required permissions
```
---
## Step 3: Generating Tokens (Login)
The `AuthService` handles login and token generation:
```csharp
public class AuthService : IAuthService
{
private readonly AppDbContext _db;
private readonly JwtOptions _jwt;
public async Task<LoginResponse> LoginAsync(LoginRequest req)
{
// 1. Find the user by username
var user = await _db.ClinicalUsers
.FirstOrDefaultAsync(u => u.Username == req.Username && u.IsActive);
// 2. Verify the password using BCrypt
if (user is null || !BCrypt.Net.BCrypt.Verify(req.Password, user.PasswordHash))
throw new ValidationException("Invalid username or password.",
"INVALID_CREDENTIALS");
// 3. Record the login time
user.LastLoginAt = DateTimeOffset.UtcNow;
await _db.SaveChangesAsync();
// 4. Write an audit log entry
_db.ClinicalAuditLogs.Add(new ClinicalAuditLog
{
Action = AuditAction.UserLogin,
EntityType = "ClinicalUser",
EntityId = user.Id,
UserId = user.Id,
UserDisplayName = user.DisplayName,
});
await _db.SaveChangesAsync();
// 5. Generate the JWT token
var expires = DateTimeOffset.UtcNow.AddMinutes(_jwt.ExpirationMinutes);
var token = GenerateToken(user, expires);
return new LoginResponse(token, expires, user.Id, user.Username,
user.DisplayName, user.Role.ToDbString());
}
}
```
### Token Generation
```csharp
private string GenerateToken(ClinicalUser user, DateTimeOffset expires)
{
var claims = new[]
{
new Claim(ClaimTypes.NameIdentifier, user.Id.ToString()),
new Claim(ClaimTypes.Name, user.Username),
new Claim("display_name", user.DisplayName),
new Claim("clinical_role", user.Role.ToDbString()),
};
var key = new SymmetricSecurityKey(Encoding.UTF8.GetBytes(_jwt.SigningKey));
var creds = new SigningCredentials(key, SecurityAlgorithms.HmacSha256);
var token = new JwtSecurityToken(
issuer: _jwt.Issuer,
audience: _jwt.Audience,
claims: claims,
expires: expires.UtcDateTime,
signingCredentials: creds);
return new JwtSecurityTokenHandler().WriteToken(token);
}
```
**What are claims?** Claims are key-value pairs embedded in the token's payload. They describe the authenticated user:
| Claim | Value | Purpose |
|-------|-------|---------|
| `NameIdentifier` | User's GUID | Unique user ID for database lookups |
| `Name` | `"dr.smith"` | Username for logging |
| `display_name` | `"Dr. Sarah Smith"` | Human-readable name for the UI |
| `clinical_role` | `"Physician"` | Role for permission checks (used by the authorization system in Guide 14) |
The claims are not encrypted — anyone can decode a JWT and read the payload (it's just base64). The signature ensures the claims haven't been tampered with, but it doesn't hide them. Never put secrets (passwords, API keys) in JWT claims.
### Password Verification with BCrypt
```csharp
BCrypt.Net.BCrypt.Verify(req.Password, user.PasswordHash)
```
**What is BCrypt?** BCrypt is a password hashing algorithm specifically designed to be slow. Why slow? Because if an attacker steals the database, they'll try to crack passwords by hashing millions of guesses. BCrypt's configurable "cost factor" (default 12) makes each hash attempt take ~250ms — fast enough for a single login, but impossibly slow for brute-force attacks (at 250ms each, trying 1 million passwords would take 70 hours).
Passwords are never stored in plaintext — only the BCrypt hash. `Verify()` hashes the provided password and compares it to the stored hash.
---
## Step 4: How the Dashboard Uses the Token
The Vue.js dashboard stores the token after login and includes it in every API request:
```
GET /api/encounters HTTP/1.1
Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...
```
The `Bearer` prefix is a standard convention that tells the server "the value after this space is a JWT token."
When the token expires (after 8 hours), the API returns `401 Unauthorized`, and the dashboard redirects the user to the login page.
---
## Step 5: The Fallback Policy
```csharp
builder.Services.AddAuthorization(options =>
{
options.FallbackPolicy = new AuthorizationPolicyBuilder()
.RequireAuthenticatedUser()
.Build();
});
```
**What is the fallback policy?** It applies to any endpoint that doesn't have an explicit authorization attribute. By setting it to `RequireAuthenticatedUser()`, every endpoint in the API requires a valid JWT token by default. Endpoints that should be publicly accessible (like health checks) must explicitly opt out with `.AllowAnonymous()`.
This is a security-by-default approach — if a developer forgets to add an authorization attribute to a new endpoint, it's protected rather than exposed.
---
## Key Takeaways
- **JWT is stateless authentication** — the token contains all the user info needed, so the server doesn't need to look up a session database on every request
- **The signing key is the most important secret** — anyone who knows the key can forge tokens for any user. Keep it out of source control.
- **Claims carry identity, not permissions** — the token contains the user's role (`clinical_role`), and the authorization system (Guide 14) maps that role to permissions at request time
- **Token expiration forces periodic re-authentication** — 8 hours matches a clinical shift. After that, the user must log in again.
- **BCrypt protects passwords at rest** — even if the database is compromised, passwords can't be reversed from their hashes
- **The fallback policy ensures no endpoint is accidentally left unprotected** — security by default, with explicit opt-out for public endpoints
docs/guides/14-rbac-dynamic-policies.md
+359
View File
@@ -0,0 +1,359 @@
# Guide 14: Role-Based Access Control (RBAC) with Dynamic Policies
## What is RBAC?
**Authorization** answers the question "what are you allowed to do?" Authentication (Guide 13) verifies identity — you're Dr. Smith. Authorization verifies permissions — Dr. Smith can acknowledge alerts, but can she modify threshold configurations?
**Role-Based Access Control (RBAC)** is the most common authorization model. Instead of assigning permissions directly to each user, you:
1. Define **roles** (Nurse, Physician, Admin, Integration)
2. Assign **permissions** to each role (Nurse can read patients, acknowledge alerts, record observations)
3. Assign each user one role
When a user makes a request, the system checks: "Does this user's role have the required permission for this action?"
The advantage over assigning permissions directly to users: when you hire a new nurse, you assign the "Nurse" role once and they get all the right permissions. If you need to give all nurses a new permission, you change it in one place (the role definition).
---
## How RBAC Works in This Project
```
HTTP Request
│
▼
┌─────────────┐ JWT has claim:
│ JWT Bearer │ "clinical_role": "Nurse"
│ Middleware │
└──────┬──────┘
│
▼
┌─────────────────────────┐ Controller has attribute:
│ [AuthorizePermission( │ "alerts:acknowledge"
│ "alerts:acknowledge")]│
└──────┬──────────────────┘
│
▼
┌─────────────────────────┐ Looks up: does the policy "perm:alerts:acknowledge"
│ PermissionPolicyProvider│ exist? Creates it on-the-fly.
└──────┬──────────────────┘
│
▼
┌─────────────────────────────────┐ Checks the role-permission map:
│ PermissionAuthorizationHandler │ Nurse → { "alerts:acknowledge" ✓ }
│ │
│ If denied: logs warning, │
│ increments Prometheus counter │
└─────────────────────────────────┘
```
---
## Step 1: Define Permissions
All permissions are defined as string constants in one class:
```csharp
public static class ClinicalPermissions
{
public const string PatientsRead = "patients:read";
public const string PatientsWrite = "patients:write";
public const string EncountersRead = "encounters:read";
public const string EncountersWrite = "encounters:write";
public const string ObservationsIngest = "observations:ingest";
public const string AlertsRead = "alerts:read";
public const string AlertsAcknowledge = "alerts:acknowledge";
public const string AlertsResolve = "alerts:resolve";
public const string AlertsFeedback = "alerts:feedback";
public const string ThresholdsRead = "thresholds:read";
public const string ThresholdsWrite = "thresholds:write";
public const string AnalyticsRead = "analytics:read";
public const string OrdersWrite = "orders:write";
public const string MedicationsWrite = "medications:write";
public const string FhirIngest = "fhir:ingest";
public const string FhirRead = "fhir:read";
public const string AuditRead = "audit:read";
public const string UsersAdmin = "users:admin";
}
```
The naming convention `resource:action` makes permissions self-documenting. `"thresholds:write"` clearly means "can modify alert thresholds."
Using `const string` rather than an enum means permissions can be used in attribute arguments (C# requires compile-time constants for attribute parameters).
---
## Step 2: Map Roles to Permissions
The `ClinicalRolePermissionMap` defines which permissions each role has:
```csharp
public static class ClinicalRolePermissionMap
{
private static readonly Dictionary<ClinicalRole, HashSet<string>> _map = new()
{
[ClinicalRole.Nurse] = new()
{
ClinicalPermissions.PatientsRead,
ClinicalPermissions.PatientsWrite,
ClinicalPermissions.EncountersRead,
ClinicalPermissions.EncountersWrite,
ClinicalPermissions.ObservationsIngest,
ClinicalPermissions.AlertsRead,
ClinicalPermissions.AlertsAcknowledge,
ClinicalPermissions.AlertsResolve,
ClinicalPermissions.ThresholdsRead,
ClinicalPermissions.AnalyticsRead,
ClinicalPermissions.OrdersWrite,
ClinicalPermissions.MedicationsWrite,
ClinicalPermissions.AlertsFeedback,
},
[ClinicalRole.Physician] = new()
{
// Same as Nurse in this version
// Physicians and nurses share clinical permissions
},
[ClinicalRole.Admin] = new()
{
// Everything nurses/physicians have, PLUS:
ClinicalPermissions.ThresholdsWrite, // modify alert thresholds
ClinicalPermissions.FhirIngest, // FHIR integration
ClinicalPermissions.FhirRead, // FHIR read/search
ClinicalPermissions.AuditRead, // view audit logs
ClinicalPermissions.UsersAdmin, // manage users
},
[ClinicalRole.Integration] = new()
{
// Narrow set — only what integration systems need
ClinicalPermissions.PatientsWrite,
ClinicalPermissions.EncountersWrite,
ClinicalPermissions.ObservationsIngest,
ClinicalPermissions.MedicationsWrite,
ClinicalPermissions.FhirIngest,
ClinicalPermissions.FhirRead,
},
};
public static bool HasPermission(ClinicalRole role, string permission) =>
_map.TryGetValue(role, out var perms) && perms.Contains(permission);
}
```
Key design choices:
- **Admin has superset permissions** — everything clinical roles have plus administrative actions
- **Integration has minimum permissions** — machine-to-machine integrations can only write data (patients, encounters, observations, medications) and use FHIR. They can't acknowledge alerts, modify thresholds, or view audit logs.
- **The map is in code, not the database** — permission changes require a deployment, which provides a review and audit trail. For systems where permissions change frequently, you'd store them in a database instead.
---
## Step 3: The AuthorizePermission Attribute
Controllers declare which permission is required using a custom attribute:
```csharp
[AuthorizePermission(ClinicalPermissions.EncountersRead)]
[HttpGet]
public async Task<ActionResult> GetEncounters(...)
[AuthorizePermission(ClinicalPermissions.EncountersWrite)]
[HttpPost]
public async Task<ActionResult> OpenEncounter(...)
[AuthorizePermission(ClinicalPermissions.AlertsAcknowledge)]
[HttpPatch("{alertId}/acknowledge")]
public async Task<ActionResult> AcknowledgeAlert(...)
```
The attribute itself is a thin wrapper:
```csharp
public class AuthorizePermissionAttribute : AuthorizeAttribute
{
public AuthorizePermissionAttribute(string permission)
{
Policy = $"perm:{permission}";
}
}
```
**What does `Policy = $"perm:{permission}"` do?** ASP.NET Core's authorization system works with named policies. When you write `[Authorize(Policy = "perm:alerts:acknowledge")]`, ASP.NET asks "does a policy named `perm:alerts:acknowledge` exist?" — and if so, does the current user satisfy it?
The `perm:` prefix is a convention that the `PermissionPolicyProvider` uses to recognize permission-based policies and create them on the fly.
---
## Step 4: Dynamic Policy Resolution
ASP.NET Core expects you to register all policies at startup. But with 18 permissions, you'd need 18 policy registrations — tedious and easy to forget. Instead, the `PermissionPolicyProvider` creates policies dynamically:
```csharp
public class PermissionPolicyProvider : IAuthorizationPolicyProvider
{
private readonly DefaultAuthorizationPolicyProvider _fallback;
public PermissionPolicyProvider(IOptions<AuthorizationOptions> options)
{
_fallback = new DefaultAuthorizationPolicyProvider(options);
}
public Task<AuthorizationPolicy?> GetPolicyAsync(string policyName)
{
if (policyName.StartsWith("perm:", StringComparison.Ordinal))
{
var permission = policyName["perm:".Length..]; // "alerts:acknowledge"
var policy = new AuthorizationPolicyBuilder()
.RequireAuthenticatedUser()
.AddRequirements(new PermissionRequirement(permission))
.Build();
return Task.FromResult<AuthorizationPolicy?>(policy);
}
return _fallback.GetPolicyAsync(policyName);
}
}
```
**How does this work?**
1. ASP.NET Core sees `[AuthorizePermission("alerts:acknowledge")]` on a controller action
2. The attribute sets `Policy = "perm:alerts:acknowledge"`
3. ASP.NET asks `PermissionPolicyProvider.GetPolicyAsync("perm:alerts:acknowledge")`
4. The provider sees the `perm:` prefix, extracts `"alerts:acknowledge"`, and builds a policy that requires authentication + a `PermissionRequirement`
5. If the policy name doesn't start with `perm:`, it falls through to the default provider (for standard ASP.NET policies)
The `PermissionRequirement` is a simple data object:
```csharp
public class PermissionRequirement : IAuthorizationRequirement
{
public string Permission { get; }
public PermissionRequirement(string permission) => Permission = permission;
}
```
---
## Step 5: The Authorization Handler
The `PermissionAuthorizationHandler` does the actual permission check:
```csharp
public class PermissionAuthorizationHandler
: AuthorizationHandler<PermissionRequirement>
{
protected override Task HandleRequirementAsync(
AuthorizationHandlerContext context,
PermissionRequirement requirement)
{
// 1. Extract the role from the JWT claims
var roleClaim = context.User.FindFirst("clinical_role")?.Value;
if (roleClaim is null)
{
LogAuthorizationFailure(context.User, "none", requirement.Permission);
return Task.CompletedTask; // deny (no role claim)
}
// 2. Convert the string to the ClinicalRole enum
var role = ClinicalRoleExtensions.FromDbString(roleClaim);
// 3. Check the role-permission map
if (ClinicalRolePermissionMap.HasPermission(role, requirement.Permission))
{
context.Succeed(requirement); // allow
}
else
{
LogAuthorizationFailure(context.User, roleClaim, requirement.Permission);
// don't call context.Fail() — just don't succeed
// this lets other handlers potentially succeed for the same requirement
}
return Task.CompletedTask;
}
}
```
**Why not call `context.Fail()`?** In ASP.NET Core's authorization pipeline, `Fail()` is a hard denial — no other handler can override it. By simply not calling `Succeed()`, the requirement remains unsatisfied, which still results in a denial, but allows the possibility of other handlers succeeding. This follows the ASP.NET Core best practice for custom handlers.
### Logging and Metrics on Denial
```csharp
private void LogAuthorizationFailure(ClaimsPrincipal user, string role, string permission)
{
_logger.LogWarning(
"Authorization denied: user={User}, role={Role}, " +
"requiredPermission={Permission}, endpoint={Endpoint}",
username, userId, role, permission, endpoint);
_metrics.AuthorizationFailuresTotal.WithLabels(permission, role).Inc();
}
```
Every denied request is:
1. **Logged** with full context (who, what role, what permission, which endpoint) — visible in Seq
2. **Counted** in the `authorization_failures_total` Prometheus metric — visible on the Grafana dashboard
A spike in authorization failures could indicate a misconfigured role, a compromised account trying to access restricted resources, or a frontend bug sending requests to the wrong endpoint.
---
## Step 6: Registration
All authorization components are registered in `Program.cs`:
```csharp
builder.Services.AddSingleton<IAuthorizationPolicyProvider, PermissionPolicyProvider>();
builder.Services.AddSingleton<IAuthorizationHandler, PermissionAuthorizationHandler>();
```
Both are singletons because they're stateless — they don't hold user-specific data.
---
## The Complete Authorization Flow
1. **Request arrives** with `Authorization: Bearer eyJhb...` header
2. **JWT middleware** validates the signature, checks expiry, extracts claims → `HttpContext.User` now has `clinical_role = "Nurse"`
3. **Routing** matches the endpoint → `[AuthorizePermission("alerts:acknowledge")]`
4. **Policy provider** creates a policy requiring `PermissionRequirement("alerts:acknowledge")`
5. **Authorization handler** reads `clinical_role` from claims, checks `ClinicalRolePermissionMap.HasPermission(Nurse, "alerts:acknowledge")` → **true** → `context.Succeed()`
6. **Request proceeds** to the controller action
If step 5 returns false → **403 Forbidden** (authenticated but not authorized).
If step 2 fails → **401 Unauthorized** (not authenticated at all).
---
## Permission Matrix
| Permission | Nurse | Physician | Admin | Integration |
|-----------|-------|-----------|-------|-------------|
| `patients:read` | ✓ | ✓ | ✓ | |
| `patients:write` | ✓ | ✓ | ✓ | ✓ |
| `encounters:read` | ✓ | ✓ | ✓ | |
| `encounters:write` | ✓ | ✓ | ✓ | ✓ |
| `observations:ingest` | ✓ | ✓ | ✓ | ✓ |
| `alerts:read` | ✓ | ✓ | ✓ | |
| `alerts:acknowledge` | ✓ | ✓ | ✓ | |
| `alerts:resolve` | ✓ | ✓ | ✓ | |
| `alerts:feedback` | ✓ | ✓ | ✓ | |
| `thresholds:read` | ✓ | ✓ | ✓ | |
| `thresholds:write` | | | ✓ | |
| `analytics:read` | ✓ | ✓ | ✓ | |
| `orders:write` | ✓ | ✓ | ✓ | |
| `medications:write` | ✓ | ✓ | ✓ | ✓ |
| `fhir:ingest` | | | ✓ | ✓ |
| `fhir:read` | | | ✓ | ✓ |
| `audit:read` | | | ✓ | |
| `users:admin` | | | ✓ | |
---
## Key Takeaways
- **RBAC simplifies permission management** — assign a role once, get all the right permissions. Change the role definition to update everyone with that role.
- **Dynamic policy providers avoid boilerplate** — instead of registering 18 policies manually, the provider creates them on-the-fly from the `perm:` prefix convention.
- **The permission check is a simple map lookup** — `HasPermission(role, permission)` is O(1), adding zero measurable latency to request processing.
- **Authorization failures are observable** — logged to Seq and counted in Prometheus, so security incidents are visible.
- **401 vs 403** — 401 Unauthorized means "I don't know who you are" (missing or invalid token). 403 Forbidden means "I know who you are, but you're not allowed to do this" (valid token, insufficient permissions).
docs/guides/15-api-key-authentication.md
+278
View File
@@ -0,0 +1,278 @@
# Guide 15: API Key Authentication for Machine-to-Machine
## What is API Key Authentication?
Not every system that calls your API has a human user who can type a username and password. Some callers are machines — a ward gateway device reporting vital signs, a FHIR integration engine sending patient data from the hospital's EHR (Electronic Health Record). These systems need to authenticate without a login form.
An **API key** is a pre-shared secret — a long random string that both the client and server know. The client sends it in an HTTP header, and the server checks if it matches. Think of it like a password, but for machines instead of humans.
```
Client (ward gateway) Server (API)
┌─────────────────────┐ ┌──────────────┐
│ Sends request with: │ │ Checks: │
│ X-Api-Key: dev-gw.. │ ─────────────────────► │ Does the key │
│ X-Gateway-Id: 222.. │ │ match config?│
└─────────────────────┘ └──────────────┘
```
**Why not use JWT for machines too?** You could, but JWT adds complexity that machines don't need. JWT involves a login step (exchanging credentials for a token), token expiration, and token refresh. API keys are simpler — one secret, no expiration logic, no login endpoint. The tradeoff: API keys have no built-in expiration, so key rotation must be handled manually.
---
## Two API Key Systems in This Project
This project has two independent API key systems for different purposes:
| System | Header | Who Uses It | What It Protects |
|--------|--------|-------------|-----------------|
| Gateway API key | `X-Api-Key` + `X-Gateway-Id` | Ward gateway devices | Gateway-specific endpoints (heartbeat, sync upload) |
| FHIR API key | `X-Api-Key` | Hospital integration engines (like Mirth Connect) | FHIR R4 endpoints (`/fhir/R4/*`) |
Both use the same header name (`X-Api-Key`) but are handled by different authentication components and configured with different keys.
---
## System 1: Gateway API Key Authentication
### The Authentication Handler
When a ward gateway sends a request with `X-Api-Key`, the `GatewayApiKeyAuthenticationHandler` validates it:
```csharp
public sealed class GatewayApiKeyAuthenticationHandler
: AuthenticationHandler<AuthenticationSchemeOptions>
{
public const string SchemeName = "GatewayApiKey";
protected override Task<AuthenticateResult> HandleAuthenticateAsync()
{
// 1. Check if the request has an X-Api-Key header
if (!Request.Headers.TryGetValue("X-Api-Key", out var suppliedHeader))
return Task.FromResult(AuthenticateResult.NoResult());
// NoResult means "I can't handle this — let another scheme try"
// 2. Load the configured key from appsettings
var configured = _config["ApiKey:Gateway"];
if (string.IsNullOrEmpty(configured))
return Task.FromResult(AuthenticateResult.Fail(
"Gateway API key not configured."));
// 3. Compare using constant-time comparison
if (!FixedTimeEquals(suppliedHeader.ToString(), configured))
return Task.FromResult(AuthenticateResult.Fail("Invalid API key."));
// 4. Create claims for the authenticated gateway
var claims = new List<Claim> { new("client_type", "gateway") };
if (Request.Headers.TryGetValue("X-Gateway-Id", out var gatewayIdHeader)
&& Guid.TryParse(gatewayIdHeader.ToString(), out _))
claims.Add(new Claim("gateway_id", gatewayIdHeader.ToString()!));
// 5. Return a successful authentication result
var identity = new ClaimsIdentity(claims, SchemeName);
var ticket = new AuthenticationTicket(
new ClaimsPrincipal(identity), SchemeName);
return Task.FromResult(AuthenticateResult.Success(ticket));
}
}
```
### Constant-Time Comparison
```csharp
private static bool FixedTimeEquals(string supplied, string configured)
{
var suppliedBytes = Encoding.UTF8.GetBytes(supplied);
var configuredBytes = Encoding.UTF8.GetBytes(configured);
return CryptographicOperations.FixedTimeEquals(suppliedBytes, configuredBytes);
}
```
**Why not just use `==` to compare strings?** Regular string comparison (`==`) is vulnerable to **timing attacks**. When comparing two strings character by character, the comparison fails faster when the first character is wrong than when the last character is wrong. An attacker can measure this timing difference (even over a network) and deduce the key one character at a time.
`CryptographicOperations.FixedTimeEquals` always takes the same amount of time regardless of where the mismatch occurs. It compares every byte even after finding a difference, so the timing reveals nothing about which bytes matched.
This matters in practice: timing attacks have been demonstrated against real APIs over the public internet with as few as ~1000 requests per character.
### Registration
The gateway scheme is registered alongside JWT bearer as a secondary authentication scheme:
```csharp
builder.Services.AddAuthentication(JwtBearerDefaults.AuthenticationScheme)
.AddJwtBearer(options => { /* JWT config */ })
.AddScheme<AuthenticationSchemeOptions, GatewayApiKeyAuthenticationHandler>(
GatewayApiKeyAuthenticationHandler.SchemeName, null);
```
**How does ASP.NET Core choose which scheme to use?** JWT Bearer is the default scheme (first argument to `AddAuthentication`). For most requests, ASP.NET tries JWT first. The gateway handler returns `NoResult()` when there's no `X-Api-Key` header, signaling "this isn't my request." For gateway endpoints that specifically require the `GatewayApiKey` scheme, controllers can specify:
```csharp
[Authorize(AuthenticationSchemes = GatewayApiKeyAuthenticationHandler.SchemeName)]
```
### Configuration
```json
{
"ApiKey": {
"Gateway": "dev-gateway-key-change-in-production"
}
}
```
---
## System 2: FHIR API Key (Middleware-Based)
FHIR endpoints accept either a JWT token OR an API key. This is implemented as middleware (not an authentication handler) because the dual-auth logic needs to run before the standard authentication pipeline:
```csharp
public class FhirApiKeyOrJwtMiddleware
{
public async Task InvokeAsync(HttpContext context)
{
// Only applies to /fhir/* paths
if (!context.Request.Path.StartsWithSegments("/fhir"))
{
await _next(context);
return;
}
// /fhir/R4/metadata is always public (FHIR standard)
if (context.Request.Path.StartsWithSegments("/fhir/R4/metadata"))
{
await _next(context);
return;
}
// If already authenticated via JWT, let it through
if (context.User.Identity?.IsAuthenticated == true)
{
await _next(context);
return;
}
// Try API key authentication
var configuredKeys = GetConfiguredKeys();
if (context.Request.Headers.TryGetValue("X-Api-Key", out var suppliedKey)
&& MatchesAnyKey(suppliedKey!, configuredKeys))
{
// Set up an Integration user identity
var claims = new[]
{
new Claim(ClaimTypes.NameIdentifier, "44444444-..."),
new Claim(ClaimTypes.Name, "integration.mirth"),
new Claim("display_name", "Mirth Connect"),
new Claim("clinical_role", ClinicalRole.Integration.ToDbString()),
};
context.User = new ClaimsPrincipal(
new ClaimsIdentity(claims, "FhirApiKey"));
await _next(context);
return;
}
// X-Api-Key was provided but didn't match — return 401
if (context.Request.Headers.ContainsKey("X-Api-Key"))
{
context.Response.StatusCode = 401;
// Return FHIR OperationOutcome error format
return;
}
// No API key — fall through to normal JWT auth
await _next(context);
}
}
```
**Why set `clinical_role` to `Integration`?** When the API key authenticates successfully, the middleware creates a fake user identity with the `Integration` role. This means the RBAC system (Guide 14) works exactly the same way — the FHIR integration user has `Integration` permissions (can write patients, encounters, observations, but can't acknowledge alerts or modify thresholds).
### Key Rotation Support
The FHIR system supports multiple active keys simultaneously for zero-downtime rotation:
```csharp
public class FhirOptions
{
public string? ApiKey { get; set; } // single key (convenience)
public string[] ApiKeys { get; set; } = []; // multiple keys for rotation
}
```
```json
{
"Fhir": {
"ApiKey": "current-key-abc",
"ApiKeys": ["current-key-abc", "new-key-xyz"]
}
}
```
**How does key rotation work?**
1. Add the new key to `ApiKeys` alongside the old key → deploy
2. Update the integration system to use the new key
3. Remove the old key from `ApiKeys` → deploy
During step 1-2, both keys are valid. The integration system experiences zero downtime.
The key matching checks all configured keys using constant-time comparison:
```csharp
private static bool MatchesAnyKey(string supplied, string[] configuredKeys)
{
var suppliedBytes = Encoding.UTF8.GetBytes(supplied);
var matched = false;
foreach (var configured in configuredKeys)
{
var configuredBytes = Encoding.UTF8.GetBytes(configured);
if (CryptographicOperations.FixedTimeEquals(suppliedBytes, configuredBytes))
matched = true;
// Don't return early — check all keys to prevent timing leaks
}
return matched;
}
```
Note that it checks ALL keys even after finding a match. If it returned immediately on the first match, an attacker could determine how many keys are configured by measuring response time.
---
## How the Two Systems Interact
The middleware pipeline processes requests in order:
```csharp
app.UseMiddleware<CorrelationIdMiddleware>(); // 1. Add correlation ID
app.UseMiddleware<FhirApiKeyOrJwtMiddleware>(); // 2. FHIR API key (sets User if matched)
app.UseMiddleware<ExceptionHandlerMiddleware>(); // 3. Error handling
app.UseAuthentication(); // 4. JWT / GatewayApiKey handlers
app.UseAuthorization(); // 5. Permission checks
```
For a FHIR request with an API key:
- Step 2 matches the key and sets `context.User` with Integration claims
- Step 4 sees that `User` is already authenticated and skips JWT validation
- Step 5 checks the Integration role's permissions normally
For a gateway request with an API key:
- Step 2 doesn't match (`/api/gateways/...` doesn't start with `/fhir`)
- Step 4 runs the `GatewayApiKeyAuthenticationHandler`, which validates the key
- Step 5 checks authorization normally
For a dashboard request with a JWT:
- Step 2 doesn't match (no API key header)
- Step 4 validates the JWT token
- Step 5 checks the user's role permissions
---
## Key Takeaways
- **API keys are for machines, JWTs are for humans** — API keys are simpler (no login step, no expiration logic) but require manual rotation
- **Always use constant-time comparison for secrets** — `CryptographicOperations.FixedTimeEquals` prevents timing attacks that could leak the key character by character
- **Dual auth (key OR token) gives flexibility** — FHIR endpoints accept either, so both integration engines (API key) and admin users (JWT) can access them
- **Key rotation requires supporting multiple keys simultaneously** — add the new key first, migrate clients, then remove the old key
- **API key authentication creates a claims identity** — the `Integration` role feeds into the same RBAC system as JWT-authenticated users, so permissions are managed in one place
- **Return `NoResult()` for unrecognized requests** — this tells ASP.NET "try the next authentication scheme" rather than failing immediately, enabling multiple schemes to coexist