feature: MinIO Data Lake Writer (Parquet, Partitioned)

docs/decisions/data-lake-design.md

@@ -0,0 +1,87 @@
+```markdown
+# Decision: Data Lake Architecture — Parquet on MinIO
+
+## Context
+
+Healthcare regulations in most jurisdictions require medical records to be retained
+for 7–25 years. A medium hospital with 200 concurrent inpatients generates roughly
+17 observations per second at steady state. Over 10 years that is approximately
+5 billion observation rows. Keeping this in PostgreSQL would require continuous
+partitioning and archival work; running population health queries against it would
+compete with live ingest writes.
+
+## Decision
+
+A separate data lake tier handles long-term retention. The Kafka `data-lake-writer`
+consumer group reads all three topics and writes Parquet files to MinIO under
+date-partitioned prefixes.
+
+## Why Parquet over JSON or CSV
+
+Parquet is columnar. A population health query — "give me all heart rate values for
+ICU patients in 2024" — reads only the `observation_code` and `value` columns without
+deserializing `encounter_id`, `patient_id`, `unit`, `source`, or `recorded_at`.
+
+The repetition of `observation_code` values across millions of rows (thousands of rows
+all with `HEART_RATE`) compresses at 5–10× vs JSON via dictionary encoding. At 10-year
+scale this difference is measured in terabytes of storage cost.
+
+CSV is row-oriented like JSON and does not support schema evolution — adding a new
+column requires rewriting every historical file.
+
+## Why MinIO (S3-compatible) over PostgreSQL or Elasticsearch for archive
+
+PostgreSQL is the operational write layer. Running 10-year-scale queries on the same
+instance that serves live ingest introduces contention. Adding partitioning and tiered
+storage to PostgreSQL adds operational complexity without solving the fundamental
+problem: it is still a row store.
+
+Elasticsearch is optimized for search and aggregations, not for full-table scans or
+columnar projections. Storing 5 billion observation rows in Elasticsearch would require
+enormous index memory and produce no benefit for the analytics pattern that justifies
+the archive (population health over multi-year windows).
+
+MinIO is S3-compatible. The files it stores can be queried directly by DuckDB
+(single analyst), Apache Spark (cluster analytics), and AWS Athena (serverless queries
+over S3) without any data movement. Switching from MinIO to S3 in production requires
+changing one endpoint URL.
+
+## Why not Apache Flink or Spark Streaming for the lake writer
+
+At a single-hospital scale — 50 obs/sec peak, 6 Kafka partitions — the consumer lag
+from a simple .NET BackgroundService with in-memory buffering and a 5-minute flush is
+negligible. The overhead of deploying and operating a Flink cluster adds operational
+cost that is not justified by the throughput.
+
+The scale inflection point: approximately 10,000+ events/sec sustained, or the need
+for exactly-once semantics at the storage layer. Below that, the simple consumer is
+correct. Above it, Flink's checkpoint-based exactly-once delivery to Parquet
+(via its FileSystem sink) becomes worth the operational cost.
+
+## Flush policy rationale
+
+**Count-based flush (1,000 events):** Bounds memory. At 17 obs/sec steady state this
+fires approximately once per minute. At 50 obs/sec peak it fires every 20 seconds.
+
+**Time-based flush (5 minutes):** Bounds latency. At low load (night shift, few
+monitors active), the count threshold might not fire for hours. A 5-minute ceiling
+means the most recent data is always queryable within 5 minutes of arriving in Kafka.
+
+**At-least-once delivery:** Kafka offsets are committed only after the Parquet file
+is successfully uploaded. A process crash between buffering and uploading produces
+duplicate rows on the next startup — the same observation appears in two files with
+different `kafka_offset` values. For a regulatory archive this is acceptable.
+Downstream queries can deduplicate on `observation_id`. Data loss is not acceptable;
+duplicates are.
+
+## Date partition structure
+
+Files are partitioned by the **event timestamp** from the payload, not by wall clock
+time at consumption. An observation `recorded_at 2025-01-14T23:59:59Z` consumed at
+`2025-01-15T00:01:00Z` is written to `observations/2025/01/14/...`.
+
+This matches how Athena and Spark apply partition pruning: a query with
+`WHERE recorded_at BETWEEN '2025-01-14' AND '2025-01-14'` reads only the
+`observations/2025/01/14/` prefix. Partitioning by ingest time instead would cause
+the query to miss that row.
+```