tests: MinIO Data Lake Writer (Parquet, Partitioned) test
This commit is contained in:
voltsrage
@@ -55,7 +55,7 @@ An `OutboxEvent` is written in the same transaction as any observation or alert,
|
||||
- **Elasticsearch CQRS Projection** — `EsIndexerService` consumer group upserts `patient_encounters` documents, appends to the `observations` index, and updates `openAlertCount` on alert events; patient/encounter search; per-encounter observation trend (hourly avg/min/max); alert volume summary by department and severity; population query (numeric range aggregation across all patients)
|
||||
- **Sepsis Early Warning Engine** — `SepsisEngineService` Kafka consumer evaluates SIRS criteria (temperature, heart rate, respiratory rate, WBC) per encounter using Redis keys with a 30-minute TTL sliding window; on ≥2 active criteria, inserts a `SEPSIS_WARNING / CRITICAL` alert idempotently (`INSERT WHERE NOT EXISTS`)
|
||||
- **RabbitMQ Notification Workers** — `NotificationPublisherService` reads `alert.generated` from Kafka and publishes paging jobs to `alerts.paging.queue`; `PagingWorkerService` sends the page and waits for acknowledgment; if no ack arrives before timeout it NACKs to `alerts.paging.dlq` with `x-message-ttl = 300000ms`; if the host is stopping, in-flight paging messages are NACKed with `requeue=true` so they are retried after restart and do not false-escalate; `EscalationWorkerService` pages the on-call backup and sets alert status to `escalated`; `DischargeSummaryWorkerService` reads `encounter.status.changed`, generates a discharge summary, and stores it in MinIO under `/discharge-summaries/{encounterId}/summary.pdf`
|
||||
- **Data Lake Writer (Phase 9 - in progress)** — `DataLakeWriterService` Kafka consumer buffers `observation.recorded`, `alert.generated`, and `encounter.status.changed` events, flushes partitioned Parquet files to MinIO (`/observations/`, `/alerts/`, `/encounters/` by date), and commits offsets after successful uploads
|
||||
- **Data Lake Writer** — `DataLakeWriterService` (consumer group `data-lake-writer`) buffers `observation.recorded`, `alert.generated`, and `encounter.status.changed` events, flushes date-partitioned Parquet files to MinIO (`/observations/`, `/alerts/`, `/encounters/`), and commits Kafka offsets only after successful uploads; `kafka_partition` and `kafka_offset` columns provide audit lineage
|
||||
- **Reconciliation Jobs** — three scheduled checks: (1) unacknowledged CRITICAL alerts older than 30 minutes, (2) pending orders without results after 4 hours, (3) active inpatients with no observation in 2 hours; each finding creates a `reconciliation_alerts` row and publishes to RabbitMQ
|
||||
- **Standard Envelope** — all responses use a consistent `{ success, statusCode, data, error }` wrapper; validation errors use the same shape; `ApiBehaviorOptions` overridden so model validation also produces the standard envelope
|
||||
- **Observability** — Serilog structured logging enriched with `correlationId`, `encounterId`, `patientId` on alert paths; Seq sink (`http://localhost:5345`); Prometheus (`http://localhost:9101`) scrapes `GET /metrics`; Grafana dashboards (`http://localhost:3101`, admin/admin) for clinical metrics including `alerts_unacknowledged_gauge`; per-request correlation IDs in request logs and response headers
|
||||
@@ -225,6 +225,14 @@ tests/
|
||||
├── ReconciliationTests.cs # Three reconciliation checks, deduplication, RabbitMQ publish
|
||||
├── ObservabilityPhase8Tests.cs # /metrics families and correlation header behavior
|
||||
└── DataLakePhase9Tests.cs # Kafka → MinIO Parquet flow and schema checks
|
||||
|
||||
scripts/
|
||||
├── run-phase8-verification.sh # Prometheus + alerts_unacknowledged_gauge checks
|
||||
└── run-phase9-verification.sh # Data lake integration tests + Kafka + MinIO + DuckDB
|
||||
|
||||
docs/
|
||||
├── plans/phase-9-plan.md # Phase 9 implementation and verification guide
|
||||
└── decisions/data-lake-design.md # Parquet vs JSON, partitioning, replay rationale
|
||||
```
|
||||
|
||||
---
|
||||
@@ -337,7 +345,8 @@ On startup the application:
|
||||
3. Pre-loads all thresholds into Redis
|
||||
4. Provisions Kafka topics and Elasticsearch indices
|
||||
5. Declares the RabbitMQ exchange and queue topology
|
||||
6. Starts the reconciliation scheduler (three safety checks on a configurable interval)
|
||||
6. Starts the data lake writer (`data-lake-writer` → Parquet in MinIO)
|
||||
7. Starts the reconciliation scheduler (three safety checks on a configurable interval)
|
||||
|
||||
Swagger UI is available at `http://localhost:<port>/swagger` in Development.
|
||||
|
||||
@@ -349,6 +358,30 @@ dotnet test
|
||||
|
||||
Tests use Testcontainers to spin up a real PostgreSQL instance. No manual setup required.
|
||||
|
||||
### Verification Scripts
|
||||
|
||||
With the API running and Docker Compose up, run phase verification end-to-end:
|
||||
|
||||
```bash
|
||||
./scripts/run-phase8-verification.sh # Prometheus metrics, alerts_unacknowledged_gauge
|
||||
./scripts/run-phase9-verification.sh # Data lake tests, Kafka offsets, MinIO Parquet, DuckDB schema
|
||||
```
|
||||
|
||||
Phase 9 optional tools (install without sudo):
|
||||
|
||||
```bash
|
||||
# MinIO client — object listing and download
|
||||
mkdir -p ~/.local/bin
|
||||
curl -fsSL https://dl.min.io/client/mc/release/linux-amd64/mc -o ~/.local/bin/mc
|
||||
chmod +x ~/.local/bin/mc
|
||||
|
||||
# DuckDB — query Parquet files locally
|
||||
curl https://install.duckdb.org | sh
|
||||
export PATH="$HOME/.duckdb/cli/latest:$HOME/.local/bin:$PATH"
|
||||
```
|
||||
|
||||
See `docs/plans/phase-9-plan.md` for manual Kafka replay and DuckDB query examples.
|
||||
|
||||
---
|
||||
|
||||
## API Reference
|
||||
@@ -828,4 +861,4 @@ Observation history uses cursor pagination on `(recorded_at DESC, id DESC)`. Off
|
||||
| 6 | RabbitMQ exchange and queue topology; `NotificationPublisherService`; `PagingWorkerService`; DLQ escalation (`EscalationWorkerService`); discharge summary (`DischargeSummaryWorkerService` → MinIO); integration tests | Done |
|
||||
| 7 | Reconciliation scheduler — unacknowledged critical alerts, stale pending orders, disconnected monitors; `reconciliation_alerts` table; RabbitMQ publish; integration tests | Done |
|
||||
| 8 | Prometheus metrics (`GET /metrics`); Grafana dashboards; eight application metric families | In progress |
|
||||
| 9 | Data lake writer — Kafka consumer group `data-lake-writer`; Parquet flush to MinIO; integration tests (`DataLakePhase9Tests`) | In progress |
|
||||
| 9 | Data lake writer — `data-lake-writer` consumer group; Parquet flush to MinIO; `DataLakePhase9Tests`; `run-phase9-verification.sh` | Done |
|
||||
|
||||
Reference in New Issue
Block a user