108 lines
5.3 KiB
Markdown
108 lines
5.3 KiB
Markdown
```markdown
|
||
# Decision: Data Lake Architecture — Parquet on MinIO
|
||
|
||
## Context
|
||
|
||
Healthcare regulations in most jurisdictions require medical records to be retained
|
||
for 7–25 years. A medium hospital with 200 concurrent inpatients generates roughly
|
||
17 observations per second at steady state. Over 10 years that is approximately
|
||
5 billion observation rows. Keeping this in PostgreSQL would require continuous
|
||
partitioning and archival work; running population health queries against it would
|
||
compete with live ingest writes.
|
||
|
||
## Decision
|
||
|
||
A separate data lake tier handles long-term retention. The Kafka `data-lake-writer`
|
||
consumer group reads all three topics and writes Parquet files to MinIO under
|
||
date-partitioned prefixes.
|
||
|
||
## Why Parquet over JSON or CSV
|
||
|
||
Parquet is columnar. A population health query — "give me all heart rate values for
|
||
ICU patients in 2024" — reads only the `observation_code` and `value` columns without
|
||
deserializing `encounter_id`, `patient_id`, `unit`, `source`, or `recorded_at`.
|
||
|
||
The repetition of `observation_code` values across millions of rows (thousands of rows
|
||
all with `HEART_RATE`) compresses at 5–10× vs JSON via dictionary encoding. At 10-year
|
||
scale this difference is measured in terabytes of storage cost.
|
||
|
||
CSV is row-oriented like JSON and does not support schema evolution — adding a new
|
||
column requires rewriting every historical file.
|
||
|
||
## Why MinIO (S3-compatible) over PostgreSQL or Elasticsearch for archive
|
||
|
||
PostgreSQL is the operational write layer. Running 10-year-scale queries on the same
|
||
instance that serves live ingest introduces contention. Adding partitioning and tiered
|
||
storage to PostgreSQL adds operational complexity without solving the fundamental
|
||
problem: it is still a row store.
|
||
|
||
Elasticsearch is optimized for search and aggregations, not for full-table scans or
|
||
columnar projections. Storing 5 billion observation rows in Elasticsearch would require
|
||
enormous index memory and produce no benefit for the analytics pattern that justifies
|
||
the archive (population health over multi-year windows).
|
||
|
||
MinIO is S3-compatible. The files it stores can be queried directly by DuckDB
|
||
(single analyst), Apache Spark (cluster analytics), and AWS Athena (serverless queries
|
||
over S3) without any data movement. Switching from MinIO to S3 in production requires
|
||
changing one endpoint URL.
|
||
|
||
## Why not Apache Flink or Spark Streaming for the lake writer
|
||
|
||
At a single-hospital scale — 50 obs/sec peak, 6 Kafka partitions — the consumer lag
|
||
from a simple .NET BackgroundService with in-memory buffering and a 5-minute flush is
|
||
negligible. The overhead of deploying and operating a Flink cluster adds operational
|
||
cost that is not justified by the throughput.
|
||
|
||
The scale inflection point: approximately 10,000+ events/sec sustained, or the need
|
||
for exactly-once semantics at the storage layer. Below that, the simple consumer is
|
||
correct. Above it, Flink's checkpoint-based exactly-once delivery to Parquet
|
||
(via its FileSystem sink) becomes worth the operational cost.
|
||
|
||
## Flush policy rationale
|
||
|
||
**Count-based flush (1,000 events):** Bounds memory. At 17 obs/sec steady state this
|
||
fires approximately once per minute. At 50 obs/sec peak it fires every 20 seconds.
|
||
|
||
**Time-based flush (5 minutes):** Bounds latency. At low load (night shift, few
|
||
monitors active), the count threshold might not fire for hours. A 5-minute ceiling
|
||
means the most recent data is always queryable within 5 minutes of arriving in Kafka.
|
||
|
||
**At-least-once delivery:** Kafka offsets are committed only after at least one Parquet file
|
||
is successfully uploaded in a flush cycle. A process crash between buffering and uploading
|
||
produces duplicate rows on the next startup — the same observation appears in two files with
|
||
different `kafka_offset` values. For a regulatory archive this is acceptable.
|
||
Downstream queries can deduplicate on `observation_id`. Data loss is not acceptable;
|
||
duplicates are.
|
||
|
||
**Graceful shutdown:** On host stop (Ctrl+C), the writer performs a final flush with an
|
||
uncanceled token so in-flight MinIO uploads complete. If every file in a flush fails,
|
||
offsets are not committed and events are re-read on the next start.
|
||
|
||
## `alert.generated` Parquet mapping
|
||
|
||
`DataLakeWriterService.ParseAlertRow` reads these JSON fields from Kafka:
|
||
|
||
| Parquet column | JSON field | Notes |
|
||
|---|---|---|
|
||
| `alert_id` | `alertId` | Required |
|
||
| `encounter_id` | `encounterId` | Required |
|
||
| `patient_id` | `patientId` | Required |
|
||
| `alert_type` | `alertType` | e.g. `WARNING_HEART_RATE`, `CRITICAL_HEART_RATE` |
|
||
| `severity` | `severity` | e.g. `Warning`, `Critical` |
|
||
| `details` | `details` | Required on all new alert producers; parser defaults to `""` if absent |
|
||
| `triggered_at` | `triggeredAt` | ISO-8601 |
|
||
|
||
Phase 11 aligned warning and sepsis outbox producers with critical ingest alerts by
|
||
always including `details`.
|
||
|
||
## Date partition structure
|
||
|
||
Files are partitioned by the **event timestamp** from the payload, not by wall clock
|
||
time at consumption. An observation `recorded_at 2025-01-14T23:59:59Z` consumed at
|
||
`2025-01-15T00:01:00Z` is written to `observations/2025/01/14/...`.
|
||
|
||
This matches how Athena and Spark apply partition pruning: a query with
|
||
`WHERE recorded_at BETWEEN '2025-01-14' AND '2025-01-14'` reads only the
|
||
`observations/2025/01/14/` prefix. Partitioning by ingest time instead would cause
|
||
the query to miss that row.
|
||
``` |