feature: NEWS2 Composite Scoring Engine
This commit is contained in:
@@ -67,13 +67,34 @@ fires approximately once per minute. At 50 obs/sec peak it fires every 20 second
|
||||
monitors active), the count threshold might not fire for hours. A 5-minute ceiling
|
||||
means the most recent data is always queryable within 5 minutes of arriving in Kafka.
|
||||
|
||||
**At-least-once delivery:** Kafka offsets are committed only after the Parquet file
|
||||
is successfully uploaded. A process crash between buffering and uploading produces
|
||||
duplicate rows on the next startup — the same observation appears in two files with
|
||||
**At-least-once delivery:** Kafka offsets are committed only after at least one Parquet file
|
||||
is successfully uploaded in a flush cycle. A process crash between buffering and uploading
|
||||
produces duplicate rows on the next startup — the same observation appears in two files with
|
||||
different `kafka_offset` values. For a regulatory archive this is acceptable.
|
||||
Downstream queries can deduplicate on `observation_id`. Data loss is not acceptable;
|
||||
duplicates are.
|
||||
|
||||
**Graceful shutdown:** On host stop (Ctrl+C), the writer performs a final flush with an
|
||||
uncanceled token so in-flight MinIO uploads complete. If every file in a flush fails,
|
||||
offsets are not committed and events are re-read on the next start.
|
||||
|
||||
## `alert.generated` Parquet mapping
|
||||
|
||||
`DataLakeWriterService.ParseAlertRow` reads these JSON fields from Kafka:
|
||||
|
||||
| Parquet column | JSON field | Notes |
|
||||
|---|---|---|
|
||||
| `alert_id` | `alertId` | Required |
|
||||
| `encounter_id` | `encounterId` | Required |
|
||||
| `patient_id` | `patientId` | Required |
|
||||
| `alert_type` | `alertType` | e.g. `WARNING_HEART_RATE`, `CRITICAL_HEART_RATE` |
|
||||
| `severity` | `severity` | e.g. `Warning`, `Critical` |
|
||||
| `details` | `details` | Required on all new alert producers; parser defaults to `""` if absent |
|
||||
| `triggered_at` | `triggeredAt` | ISO-8601 |
|
||||
|
||||
Phase 11 aligned warning and sepsis outbox producers with critical ingest alerts by
|
||||
always including `details`.
|
||||
|
||||
## Date partition structure
|
||||
|
||||
Files are partitioned by the **event timestamp** from the payload, not by wall clock
|
||||
|
||||
Reference in New Issue
Block a user