Run initial test for Climate Resilience Verification Suite

Add first part of Alert Quality Analytics
This commit is contained in:
voltsrage
2026-06-24 03:01:00 +08:00
parent 032cd1d240
commit 185dc93fa1
50 changed files with 4116 additions and 55 deletions
+11 -1
View File
@@ -87,7 +87,7 @@ An `OutboxEvent` is written in the same transaction as any observation or alert,
- **MRN Sequence Generation** — MRN numbers are generated via a PostgreSQL sequence (`mrn_seq`) instead of MAX+1 queries; eliminates race conditions under concurrent patient registration; configurable prefix and digit count via `PatientOptions`
- **FHIR Bundle Transaction Rollback** — `FhirBundleProcessor` wraps all bundle entry processing in a database transaction; on any entry failure, the transaction is rolled back and the response includes the `OperationOutcome` for the failed entry; prevents partial state from orphaned Patient/Encounter records
- **Clinician Feedback Mode** — six quick ratings per alert (useful, too early, too late, false positive, missing context, would act); optional notes; Feedback Summary with aggregate stats and JSON/CSV export; client-side persistence for product research
- **Console Replay Simulator** — standalone `VigilCare.Simulator` .NET console app replays JSON scenario files against the live API with configurable speed (`--speed 0` instant, `60` = 60× faster); commands: `replay`, `replay-all`, `validate`, `dry-run`; optional `--poll` shows alerts, NEWS2, GCS, SOFA, and sepsis bundle state during replay; eleven sample scenarios in `VigilCare.Simulator/Scenarios/List/` (including GCS neurological decline, SOFA sepsis progression, and SpO₂/FiO₂ fallback); user guide in `docs/simulator-guide.md`
- **Console Replay Simulator** — standalone `VigilCare.Simulator` .NET console app replays JSON scenario files against the live API with configurable speed (`--speed 0` instant, `60` = 60× faster); commands: `replay`, `replay-all`, `validate`, `dry-run`; optional `--poll` shows alerts, NEWS2, GCS, SOFA, and sepsis bundle state during replay; `--gateway` targets the ward gateway (`http://localhost:5081`) with `--encounter-id`, `--skip-setup`, and `--gateway-token`; `alert_ack` events poll for open alerts on central before acknowledging (handles async alert pipeline at `--speed 0`); twelve sample scenarios in `VigilCare.Simulator/Scenarios/List/` (including ward outage reconnect, GCS neurological decline, SOFA sepsis progression, and SpO₂/FiO₂ fallback); user guide in `docs/simulator-guide.md`
- **RabbitMQ Notification Workers** — `NotificationPublisherService` reads `alert.generated` from Kafka and publishes paging jobs to `alerts.paging.queue`; `PagingWorkerService` sends the page and waits for acknowledgment; if no ack arrives before timeout it NACKs to `alerts.paging.dlq` with `x-message-ttl = 300000ms`; if the host is stopping, in-flight paging messages are NACKed with `requeue=true` so they are retried after restart and do not false-escalate; `EscalationWorkerService` pages the on-call backup and sets alert status to `escalated`; `DischargeSummaryWorkerService` reads `encounter.status.changed`, generates a discharge summary, and stores it in MinIO under `/discharge-summaries/{encounterId}/summary.pdf`
- **Data Lake Writer** — `DataLakeWriterService` (consumer group `data-lake-writer`) buffers `observation.recorded`, `alert.generated`, and `encounter.status.changed` events, flushes date-partitioned Parquet files to MinIO (`/observations/`, `/alerts/`, `/encounters/`), and commits Kafka offsets only for topic-partitions where all uploads succeeded; failed partition buffers are retained in memory and retried on the next flush cycle (prevents data loss from partial upload failures); shutdown flush uses an uncanceled token so MinIO writes complete on Ctrl+C; `kafka_partition` and `kafka_offset` columns provide audit lineage
- **Reconciliation Jobs** — three scheduled checks: (1) unacknowledged CRITICAL alerts older than 30 minutes, (2) pending orders without results after 4 hours, (3) active inpatients with no observation in 2 hours; each finding creates a `reconciliation_alerts` row and publishes to RabbitMQ
@@ -593,6 +593,7 @@ scripts/
├── run-phase20-verification.sh # Phase 20 — Gateway registry tests + site/gateway/heartbeat curl checks
├── run-phase21-verification.sh # Phase 21 — Ward gateway local-first path, partition tests, sync upload
├── run-phase22-verification.sh # Phase 22 — Dashboard gap analysis fixes (SOFA/GCS/qSOFA history, patient banner, timeline, medication markers)
├── run-phase24-verification.sh # Phase 24 — Ward outage reconnect scenario (central + gateway replay, ack sync)
├── run-phase29-verification.sh # Phase 29 — Simulator scenario expansion + clinical validation
├── run-phase30-verification.sh # Phase 30 — FHIR R4 ingest integration tests + manual bundle/metadata checks
└── run-phase31-verification.sh # Phase 31 — RBAC integration tests + JWT login + audit log query
@@ -791,6 +792,14 @@ dotnet run --project VigilCare.Simulator -- replay \
Other commands: `validate <file>`, `dry-run <file>`, `replay-all <directory>`. See `docs/simulator-guide.md` for the full user guide.
**Ward outage reconnect (Phase 24):** scenario `ward-outage-reconnect-01.json` exercises critical hyperkalemia alerting and nurse acknowledgment during a central outage, then sync back to central when connectivity returns. Manual procedure in `docs/simulator-guide.md` §10; automated end-to-end check:
```bash
./scripts/run-phase24-verification.sh
```
Requires Docker Compose (central API, ward gateway stack) and a running central API on `http://localhost:5270`. Use `SKIP_DOCKER=1` when the stack is already up. Notes from verification runs: `docs/resilience/phase-24-verification-notes.md`.
### Run the Dashboard
With the API running, start the Vue frontend:
@@ -860,6 +869,7 @@ With the API running (`dotnet run`) and Docker Compose up:
```bash
./scripts/run-phase22-verification.sh # Dashboard gap analysis fixes — SOFA/GCS/qSOFA history, patient banner, timeline
./scripts/run-phase21-verification.sh # Ward gateway local-first path, partition tests, sync upload verification
./scripts/run-phase24-verification.sh # Ward outage reconnect — gateway replay, local ack, central sync after restart
./scripts/run-phase20-verification.sh # Gateway registry tests, site/gateway/heartbeat API verification
./scripts/run-phase8-verification.sh # Prometheus target UP, ten metrics, alerts_unacknowledged_gauge live update
./scripts/run-phase9-verification.sh # DataLakePhase9Tests, Kafka consumer group, MinIO Parquet, DuckDB schema