feature: MinIO Data Lake Writer (Parquet, Partitioned)

README.md

@@ -4,7 +4,7 @@ A production-quality clinical backend built with ASP.NET Core 8, PostgreSQL, Apa
 
 ## Domain Model — How It Maps to a Real Clinical System
 
-In a hospital, a patient presents for care and an encounter is opened. Bedside monitors and lab systems post observations continuously against that encounter. A rules engine evaluates each observation against configured thresholds and flags abnormal values as clinical alerts. Clinicians acknowledge and resolve alerts. If a critical alert goes unacknowledged for five minutes, the system escalates to the on-call backup. All events flow through Kafka so the Elasticsearch dashboard, sepsis engine, and data lake each consume the same stream independently.
+In a hospital, a patient presents for care and an encounter is opened. Bedside monitors and lab systems post observations continuously against that encounter. A rules engine evaluates each observation against configured thresholds and flags abnormal values as clinical alerts. Clinicians acknowledge and resolve alerts. If a critical alert goes unacknowledged for five minutes, the system escalates to the on-call backup. All events flow through Kafka so the Elasticsearch dashboard, sepsis engine, and data lake writer consume the same stream independently.
 
 ```
 Patient ─────────────────────────── one patient = one MRN, many lifetime encounters
@@ -55,7 +55,7 @@ An `OutboxEvent` is written in the same transaction as any observation or alert,
 - **Elasticsearch CQRS Projection** — `EsIndexerService` consumer group upserts `patient_encounters` documents, appends to the `observations` index, and updates `openAlertCount` on alert events; patient/encounter search; per-encounter observation trend (hourly avg/min/max); alert volume summary by department and severity; population query (numeric range aggregation across all patients)
 - **Sepsis Early Warning Engine** — `SepsisEngineService` Kafka consumer evaluates SIRS criteria (temperature, heart rate, respiratory rate, WBC) per encounter using Redis keys with a 30-minute TTL sliding window; on ≥2 active criteria, inserts a `SEPSIS_WARNING / CRITICAL` alert idempotently (`INSERT WHERE NOT EXISTS`)
 - **RabbitMQ Notification Workers** — `NotificationPublisherService` reads `alert.generated` from Kafka and publishes paging jobs to `alerts.paging.queue`; `PagingWorkerService` sends the page and waits for acknowledgment; if no ack arrives before timeout it NACKs to `alerts.paging.dlq` with `x-message-ttl = 300000ms`; if the host is stopping, in-flight paging messages are NACKed with `requeue=true` so they are retried after restart and do not false-escalate; `EscalationWorkerService` pages the on-call backup and sets alert status to `escalated`; `DischargeSummaryWorkerService` reads `encounter.status.changed`, generates a discharge summary, and stores it in MinIO under `/discharge-summaries/{encounterId}/summary.pdf`
-- **Data Lake Writer** — Kafka consumer writing partitioned Parquet files to MinIO (`/observations/`, `/alerts/`, `/encounters/` by date); flush policy: 1,000 events or 5 minutes, whichever comes first; columnar format for 10-year regulatory retention
+- **Data Lake Writer (Phase 9 - in progress)** — `DataLakeWriterService` Kafka consumer buffers `observation.recorded`, `alert.generated`, and `encounter.status.changed` events, flushes partitioned Parquet files to MinIO (`/observations/`, `/alerts/`, `/encounters/` by date), and commits offsets after successful uploads
 - **Reconciliation Jobs** — three scheduled checks: (1) unacknowledged CRITICAL alerts older than 30 minutes, (2) pending orders without results after 4 hours, (3) active inpatients with no observation in 2 hours; each finding creates a `reconciliation_alerts` row and publishes to RabbitMQ
 - **Standard Envelope** — all responses use a consistent `{ success, statusCode, data, error }` wrapper; validation errors use the same shape; `ApiBehaviorOptions` overridden so model validation also produces the standard envelope
 - **Observability** — Serilog structured logging enriched with `correlationId`, `encounterId`, `patientId` on alert paths; Seq sink (`http://localhost:5345`); Prometheus (`http://localhost:9101`) scrapes `GET /metrics`; Grafana dashboards (`http://localhost:3101`, admin/admin) for clinical metrics including `alerts_unacknowledged_gauge`; per-request correlation IDs in request logs and response headers
@@ -87,6 +87,7 @@ IHostedServices (background):
   PagingWorkerService    → RabbitMQ paging.queue → log page → NACK on timeout (or requeue on shutdown)
   EscalationWorkerService → RabbitMQ escalation.queue → update alert status
   DischargeSummaryWorkerService → RabbitMQ discharge.queue → MinIO PDF
+  DataLakeWriterService  → Kafka (data-lake-writer) → Parquet files in MinIO
   ReconciliationScheduler → three scheduled safety checks → reconciliation_alerts + RabbitMQ
 ```
 
@@ -182,6 +183,14 @@ VigilCareClinicalAPI/
 │   └── RabbitMqTopologyProvisioner.cs          # Declares exchange, queues, DLQ bindings on startup
 ├── Storage/
 │   └── MinioClientFactory.cs
+├── DataLake/
+│   ├── DataLakeOptions.cs                      # Flush thresholds and bucket settings
+│   ├── DataLakeWriterService.cs                # consumer group: data-lake-writer; Kafka → Parquet → MinIO
+│   └── ParquetFileBuilder.cs                   # Topic row models → Parquet byte arrays
+├── Models/Records/
+│   ├── Observation/ObservationRow.cs           # Parquet row contract for observation events
+│   ├── Alert/AlertRow.cs                       # Parquet row contract for alert events
+│   └── Encounter/EncounterStatusRow.cs         # Parquet row contract for encounter status events
 ├── Data/
 │   ├── AppDbContext.cs                         # EF Core context — entity configs, indexes, constraints
 │   ├── Configurations/                         # IEntityTypeConfiguration per entity
@@ -213,7 +222,9 @@ tests/
     ├── SirsDetectorTests.cs                    # Redis SIRS state SET/DEL/MGET logic
     ├── SirsEvaluatorTests.cs                   # Per-code criterion evaluation
     ├── NotificationPipelineTests.cs            # RabbitMQ topology, DLQ routing
-    └── ReconciliationTests.cs                  # Three reconciliation checks, deduplication, RabbitMQ publish
+    ├── ReconciliationTests.cs                  # Three reconciliation checks, deduplication, RabbitMQ publish
+    ├── ObservabilityPhase8Tests.cs             # /metrics families and correlation header behavior
+    └── DataLakePhase9Tests.cs                  # Kafka → MinIO Parquet flow and schema checks
 ```
 
 ---
@@ -817,4 +828,4 @@ Observation history uses cursor pagination on `(recorded_at DESC, id DESC)`. Off
 | 6 | RabbitMQ exchange and queue topology; `NotificationPublisherService`; `PagingWorkerService`; DLQ escalation (`EscalationWorkerService`); discharge summary (`DischargeSummaryWorkerService` → MinIO); integration tests | Done |
 | 7 | Reconciliation scheduler — unacknowledged critical alerts, stale pending orders, disconnected monitors; `reconciliation_alerts` table; RabbitMQ publish; integration tests | Done |
 | 8 | Prometheus metrics (`GET /metrics`); Grafana dashboards; eight application metric families | In progress |
-| 9 | Data lake writer — Kafka consumer group `data-lake-writer`; Parquet flush to MinIO | Planned |
+| 9 | Data lake writer — Kafka consumer group `data-lake-writer`; Parquet flush to MinIO; integration tests (`DataLakePhase9Tests`) | In progress |