feature: NEWS2 Composite Scoring Engine

This commit is contained in:
voltsrage
2026-06-18 17:39:31 +08:00
parent c3fbc20ddc
commit e6f7989298
31 changed files with 3118 additions and 45 deletions
+24 -3
View File
@@ -67,13 +67,34 @@ fires approximately once per minute. At 50 obs/sec peak it fires every 20 second
monitors active), the count threshold might not fire for hours. A 5-minute ceiling
means the most recent data is always queryable within 5 minutes of arriving in Kafka.
**At-least-once delivery:** Kafka offsets are committed only after the Parquet file
is successfully uploaded. A process crash between buffering and uploading produces
duplicate rows on the next startup — the same observation appears in two files with
**At-least-once delivery:** Kafka offsets are committed only after at least one Parquet file
is successfully uploaded in a flush cycle. A process crash between buffering and uploading
produces duplicate rows on the next startup — the same observation appears in two files with
different `kafka_offset` values. For a regulatory archive this is acceptable.
Downstream queries can deduplicate on `observation_id`. Data loss is not acceptable;
duplicates are.
**Graceful shutdown:** On host stop (Ctrl+C), the writer performs a final flush with an
uncanceled token so in-flight MinIO uploads complete. If every file in a flush fails,
offsets are not committed and events are re-read on the next start.
## `alert.generated` Parquet mapping
`DataLakeWriterService.ParseAlertRow` reads these JSON fields from Kafka:
| Parquet column | JSON field | Notes |
|---|---|---|
| `alert_id` | `alertId` | Required |
| `encounter_id` | `encounterId` | Required |
| `patient_id` | `patientId` | Required |
| `alert_type` | `alertType` | e.g. `WARNING_HEART_RATE`, `CRITICAL_HEART_RATE` |
| `severity` | `severity` | e.g. `Warning`, `Critical` |
| `details` | `details` | Required on all new alert producers; parser defaults to `""` if absent |
| `triggered_at` | `triggeredAt` | ISO-8601 |
Phase 11 aligned warning and sepsis outbox producers with critical ingest alerts by
always including `details`.
## Date partition structure
Files are partitioned by the **event timestamp** from the payload, not by wall clock