```markdown
# Decision: Data Lake Architecture — Parquet on MinIO

## Context

Healthcare regulations in most jurisdictions require medical records to be retained
for 7–25 years. A medium hospital with 200 concurrent inpatients generates roughly
17 observations per second at steady state. Over 10 years that is approximately
5 billion observation rows. Keeping this in PostgreSQL would require continuous
partitioning and archival work; running population health queries against it would
compete with live ingest writes.

## Decision

A separate data lake tier handles long-term retention. The Kafka `data-lake-writer`
consumer group reads all three topics and writes Parquet files to MinIO under
date-partitioned prefixes.

## Why Parquet over JSON or CSV

Parquet is columnar. A population health query — "give me all heart rate values for
ICU patients in 2024" — reads only the `observation_code` and `value` columns without
deserializing `encounter_id`, `patient_id`, `unit`, `source`, or `recorded_at`.

The repetition of `observation_code` values across millions of rows (thousands of rows
all with `HEART_RATE`) compresses at 5–10× vs JSON via dictionary encoding. At 10-year
scale this difference is measured in terabytes of storage cost.

CSV is row-oriented like JSON and does not support schema evolution — adding a new
column requires rewriting every historical file.

## Why MinIO (S3-compatible) over PostgreSQL or Elasticsearch for archive

PostgreSQL is the operational write layer. Running 10-year-scale queries on the same
instance that serves live ingest introduces contention. Adding partitioning and tiered
storage to PostgreSQL adds operational complexity without solving the fundamental
problem: it is still a row store.

Elasticsearch is optimized for search and aggregations, not for full-table scans or
columnar projections. Storing 5 billion observation rows in Elasticsearch would require
enormous index memory and produce no benefit for the analytics pattern that justifies
the archive (population health over multi-year windows).

MinIO is S3-compatible. The files it stores can be queried directly by DuckDB
(single analyst), Apache Spark (cluster analytics), and AWS Athena (serverless queries
over S3) without any data movement. Switching from MinIO to S3 in production requires
changing one endpoint URL.

## Why not Apache Flink or Spark Streaming for the lake writer

At a single-hospital scale — 50 obs/sec peak, 6 Kafka partitions — the consumer lag
from a simple .NET BackgroundService with in-memory buffering and a 5-minute flush is
negligible. The overhead of deploying and operating a Flink cluster adds operational
cost that is not justified by the throughput.

The scale inflection point: approximately 10,000+ events/sec sustained, or the need
for exactly-once semantics at the storage layer. Below that, the simple consumer is
correct. Above it, Flink's checkpoint-based exactly-once delivery to Parquet
(via its FileSystem sink) becomes worth the operational cost.

## Flush policy rationale

**Count-based flush (1,000 events):** Bounds memory. At 17 obs/sec steady state this
fires approximately once per minute. At 50 obs/sec peak it fires every 20 seconds.

**Time-based flush (5 minutes):** Bounds latency. At low load (night shift, few
monitors active), the count threshold might not fire for hours. A 5-minute ceiling
means the most recent data is always queryable within 5 minutes of arriving in Kafka.

**At-least-once delivery:** Kafka offsets are committed only after at least one Parquet file
is successfully uploaded in a flush cycle. A process crash between buffering and uploading
produces duplicate rows on the next startup — the same observation appears in two files with
different `kafka_offset` values. For a regulatory archive this is acceptable.
Downstream queries can deduplicate on `observation_id`. Data loss is not acceptable;
duplicates are.

**Graceful shutdown:** On host stop (Ctrl+C), the writer performs a final flush with an
uncanceled token so in-flight MinIO uploads complete. If every file in a flush fails,
offsets are not committed and events are re-read on the next start.

## `alert.generated` Parquet mapping

`DataLakeWriterService.ParseAlertRow` reads these JSON fields from Kafka:

| Parquet column | JSON field | Notes |
|---|---|---|
| `alert_id` | `alertId` | Required |
| `encounter_id` | `encounterId` | Required |
| `patient_id` | `patientId` | Required |
| `alert_type` | `alertType` | e.g. `WARNING_HEART_RATE`, `CRITICAL_HEART_RATE` |
| `severity` | `severity` | e.g. `Warning`, `Critical` |
| `details` | `details` | Required on all new alert producers; parser defaults to `""` if absent |
| `explanation_json` | `explanation` | Optional JSON object (Phase 34); null for threshold-only alerts |
| `triggered_at` | `triggeredAt` | ISO-8601 |

Phase 11 aligned warning and sepsis outbox producers with critical ingest alerts by
always including `details`.

## Date partition structure

Files are partitioned by the **event timestamp** from the payload, not by wall clock
time at consumption. An observation `recorded_at 2025-01-14T23:59:59Z` consumed at
`2025-01-15T00:01:00Z` is written to `observations/2025/01/14/...`.

This matches how Athena and Spark apply partition pruning: a query with
`WHERE recorded_at BETWEEN '2025-01-14' AND '2025-01-14'` reads only the
`observations/2025/01/14/` prefix. Partitioning by ingest time instead would cause
the query to miss that row.
```