Run initial test for Climate Resilience Verification Suite
Add first part of Alert Quality Analytics
This commit is contained in:
voltsrage
@@ -204,7 +204,7 @@ Documentation complete when:
|
||||
- [ ] [resilience/README.md](resilience/README.md) experiment index populated
|
||||
- [ ] Six chaos experiments documented with baseline/broken/fixed artifacts
|
||||
- [ ] Interview questions #29–32 in [interview-questions.md](interview-questions.md)
|
||||
- [ ] `scripts/verify-phase-24.ps1` passes
|
||||
- [ ] `scripts/run-phase24-verification.sh` passes
|
||||
|
||||
Implementation complete when all phase exit gates in plans 20–24 pass and integration tests green.
|
||||
|
||||
|
||||
@@ -0,0 +1,145 @@
|
||||
# Phase 24 verification notes
|
||||
|
||||
Notes from getting `scripts/run-phase24-verification.sh` to pass against local Docker + dev central API (`http://localhost:5270`).
|
||||
|
||||
## Automated verification
|
||||
|
||||
```bash
|
||||
./scripts/run-phase24-verification.sh
|
||||
```
|
||||
|
||||
Useful env overrides:
|
||||
|
||||
| Variable | Purpose |
|
||||
|----------|---------|
|
||||
| `SKIP_DOCKER=1` | Stack already up; skip compose |
|
||||
| `SKIP_PHASE_A=1` | Skip central baseline replay |
|
||||
| `SCENARIO_SPEED=0` | Instant replay (default) |
|
||||
| `KEEP_CENTRAL_DOWN=1` | Leave central stopped after Phase B (debug) |
|
||||
|
||||
Companion script: `scripts/mint-gateway-jwt.sh` — dev JWT for gateway (`issuer: vigilcare-gateway`).
|
||||
|
||||
---
|
||||
|
||||
## Fixes applied during verification
|
||||
|
||||
### 1. Gateway encounter missing after Phase A
|
||||
|
||||
**Symptom:** Phase B could not find encounter on gateway.
|
||||
|
||||
**Cause:** `docker compose up -d` without recreate did not re-run `SyncOnceAsync` after central had new encounters.
|
||||
|
||||
**Fix:** Script calls `docker compose up -d --force-recreate ward-gateway-api` in `configure_gateway_sync` so sync runs on a fresh gateway container.
|
||||
|
||||
---
|
||||
|
||||
### 2. Phase B observation POST returned 400
|
||||
|
||||
**Symptom:** Gateway rejected observation batches from simulator.
|
||||
|
||||
**Cause:** Central API accepts batch ingest; gateway expects a **single** observation per POST with `observationCode` (not central’s batch shape).
|
||||
|
||||
**Fix:** `VigilCareApiClient.SendObservationBatchAsync` — gateway path posts one observation at a time using gateway field names.
|
||||
|
||||
---
|
||||
|
||||
### 3. Phase B `recordedAt` rejected (future timestamp)
|
||||
|
||||
**Symptom:** 400 — timestamp more than ~5 minutes in the future.
|
||||
|
||||
**Cause:** `--speed 0` uses scenario `offsetMinutes` as simulated future times; gateway validates `recordedAt` against wall clock.
|
||||
|
||||
**Fix:** `ReplayEngine` uses `DateTimeOffset.UtcNow` for observations when `Target == Gateway`.
|
||||
|
||||
**Trade-off:** Gateway replay does not preserve simulated timeline spacing at speed 0; acceptable for outage demo (alerts still fire on values).
|
||||
|
||||
---
|
||||
|
||||
### 4. No critical potassium alert at K+ 6.1
|
||||
|
||||
**Symptom:** Only warning-tier alert; Phase B ack step had nothing to match.
|
||||
|
||||
**Cause:** Seeded threshold `CriticalHigh` for potassium is **6.5 mEq/L**; 6.1 is above warning (5.5) but below critical.
|
||||
|
||||
**Fix:** Scenario `ward-outage-reconnect-01.json` — critical observation value **6.1 → 6.8**.
|
||||
|
||||
---
|
||||
|
||||
### 5. Alert ack / jq verification failed (alert type mismatch)
|
||||
|
||||
**Symptom:** `TryAcknowledgeAlertAsync` returned false; jq filters found no alert.
|
||||
|
||||
**Cause:** API stores/returns enum-style names (e.g. `CriticalPotassiumMeqL`); scenario uses `CRITICAL_POTASSIUM_MEQ_L`.
|
||||
|
||||
**Fix:** `VigilCareApiClient.IsMatchingAlertType` — flexible match (case, underscores, suffix).
|
||||
|
||||
**Fix:** Verification script uses jq filters that accept both naming styles.
|
||||
|
||||
---
|
||||
|
||||
### 6. `acknowledgedBy` was `nurse.demo` instead of `RN-Wu`
|
||||
|
||||
**Symptom:** After gateway ack, synced alert on central showed JWT subject, not scenario nurse.
|
||||
|
||||
**Cause:** Gateway `AcknowledgeAlertRequest` had no body field for clinician; controller used `User.Identity.Name` only. Stale Docker image also hid fixes until rebuild.
|
||||
|
||||
**Fix:** Optional `clinicianId` on gateway ack request; controller uses `req.ClinicianId ?? User.Identity?.Name`. Script rebuilds gateway image before run.
|
||||
|
||||
---
|
||||
|
||||
### 7. Phase C verify timeout
|
||||
|
||||
**Symptom:** Script gave up before central showed synced ack with `RN-Wu`.
|
||||
|
||||
**Cause:** Central restart + gateway sync can exceed 120s on a loaded machine.
|
||||
|
||||
**Fix:** Poll up to **180s** for alert with `syncedFromGateway` and `acknowledgedBy = RN-Wu`.
|
||||
|
||||
---
|
||||
|
||||
### 8. Phase A central ack fails at `--speed 0` (non-fatal in script)
|
||||
|
||||
**Symptom:** Phase A `alert_ack` against central sometimes failed before the poll fix.
|
||||
|
||||
**Cause:** Alert pipeline is async; at speed 0 the ack event could run before the critical alert row existed.
|
||||
|
||||
**Fix:** Central replay now polls for up to 30s before posting `alert_ack` (`TryAcknowledgeAlertAsync` with `waitForAlert`). The verification script still treats Phase A ack as non-fatal; Phase B (gateway outage path) remains the authoritative ack/sync test.
|
||||
|
||||
---
|
||||
|
||||
## Code changes summary (already merged)
|
||||
|
||||
| Area | Files |
|
||||
|------|--------|
|
||||
| Scenario + schema | `ward-outage-reconnect-01.json`, `schema.json`, `ScenarioValidator.cs` |
|
||||
| Gateway replay | `ReplayOptions.cs`, `ReplayCommand.cs`, `ReplayEngine.cs`, `VigilCareApiClient.cs` |
|
||||
| Gateway ack attribution | `AcknowledgeAlertRequest.cs`, `AcknowledgeAlertRequestValidator.cs`, `AlertsController.cs` |
|
||||
| Tests | `WardGatewayLocalPathTests.cs` |
|
||||
| Automation | `run-phase24-verification.sh`, `mint-gateway-jwt.sh` |
|
||||
| Docs | `simulator-guide.md` §10, phase-24 plan footnote on gateway vs central ack |
|
||||
|
||||
---
|
||||
|
||||
## Remaining doc / optional code follow-ups
|
||||
|
||||
| Item | Priority | Notes |
|
||||
|------|----------|--------|
|
||||
| `phase-24-plan.md` embedded scenario snippet still showed K+ 6.1 | Doc | Updated to 6.8 to match scenario + thresholds |
|
||||
| `ext-climate-resilience-roadmap.md` referenced `verify-phase-24.ps1` | Doc | Updated to `run-phase24-verification.sh` |
|
||||
| Central async ack at speed 0 | Done | `ReplayEngine` polls up to 30s before central `alert_ack` |
|
||||
| Gateway has no `/auth/login` | By design | Use `--gateway-token` / `mint-gateway-jwt.sh` |
|
||||
| `CentralApiOptions` default port 5080 vs dev 5270 | OK | Docker compose overrides via `host.docker.internal:5270` |
|
||||
| README mention of Phase 24 script | Done | Ward outage section + verification script list |
|
||||
| Gateway mode unit tests | Optional | No simulator tests for `--gateway` path yet |
|
||||
|
||||
---
|
||||
|
||||
## Expected pass criteria (Phase C)
|
||||
|
||||
After central restarts:
|
||||
|
||||
1. Buffered alert appears on central with `syncedFromGateway` (or equivalent sync marker).
|
||||
2. `acknowledgedBy` = **RN-Wu** (not gateway JWT subject).
|
||||
3. Gateway buffer depth returns to **0** (or heartbeat shows no pending acks).
|
||||
|
||||
Last successful run: `./scripts/run-phase24-verification.sh` with `SKIP_DOCKER=1` — all phases passed.
|
||||
@@ -17,6 +17,7 @@ Think of it as a flight simulator, but for clinical decision support.
|
||||
7. [Reading the Output](#7-reading-the-output)
|
||||
8. [Creating Your Own Scenarios](#8-creating-your-own-scenarios)
|
||||
9. [Troubleshooting](#9-troubleshooting)
|
||||
10. [Ward Outage Scenario (Climate Resilience)](#10-ward-outage-scenario-climate-resilience)
|
||||
|
||||
---
|
||||
|
||||
@@ -76,6 +77,12 @@ dotnet run --project VigilCare.Simulator -- replay <scenario-file> [options]
|
||||
| `--base-url <url>` | `http://localhost:5270` | API address (change if your API runs elsewhere) |
|
||||
| `--poll` | off | Show alerts and scores after each set of vitals |
|
||||
| `--poll-interval <seconds>` | 5 | How often to check for alerts when polling |
|
||||
| `--username <name>` | `physician.demo` | API login username |
|
||||
| `--password <secret>` | (demo default) | API login password |
|
||||
| `--gateway` | off | Target the ward gateway API at `http://localhost:5081` |
|
||||
| `--encounter-id <guid>` | — | Use an existing encounter (required with `--gateway`; also used with `--skip-setup`) |
|
||||
| `--skip-setup` | off | Skip patient/encounter registration — requires `--encounter-id` |
|
||||
| `--gateway-token <jwt>` | `$GATEWAY_JWT` | Bearer token for gateway replay (required with `--gateway`; use `./scripts/mint-gateway-jwt.sh`) |
|
||||
|
||||
**Example -- run the stable baseline scenario in real-time with polling:**
|
||||
|
||||
@@ -200,6 +207,9 @@ The key concept is **offsetMinutes** -- each event happens at a certain number o
|
||||
| `medication` | A drug being administered | Ceftriaxone 1g IV |
|
||||
| `order` | A clinical order being placed | "Blood cultures", "Chest X-ray" |
|
||||
| `order_result` | Result of a prior order | "Positive for E. coli" |
|
||||
| `alert_ack` | Acknowledgement of an open alert | RN acknowledges critical potassium alert |
|
||||
|
||||
`alert_ack` events require `alertType` and `clinicianId` in `data`; optional `note`. On **gateway** replay (`--gateway`), the simulator sends `clinicianId` in the acknowledge request so the ward records the bedside nurse label (e.g. `RN-Wu`) and syncs it to central. On **central** replay, attribution comes from the logged-in API user.
|
||||
|
||||
### Vital Sign Codes
|
||||
|
||||
@@ -236,6 +246,7 @@ The simulator ships with 8 scenarios covering different clinical situations:
|
||||
| **dka-electrolyte-01** | Diabetic ketoacidosis with potassium and glucose derangement. | Varies |
|
||||
| **hypothermia-elderly-01** | Elderly patient with severe hypothermia. Slow HR, dropping temperature. | Varies |
|
||||
| **medication-false-alarm-01** | Beta-blocker causing bradycardia. Tests whether the system correctly handles medication-induced vital changes. | 3 hours |
|
||||
| **ward-outage-reconnect-01** | ICU patient with critical hyperkalemia during simulated central outage. Validates gateway-local alerting and alert acknowledgement sync. | 90 minutes |
|
||||
|
||||
All scenario files are in: `VigilCare.Simulator/Scenarios/List/`
|
||||
|
||||
@@ -398,3 +409,83 @@ Use `--base-url`:
|
||||
dotnet run --project VigilCare.Simulator -- replay scenario.json \
|
||||
--base-url http://192.168.1.50:5270
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 10. Ward Outage Scenario (Climate Resilience)
|
||||
|
||||
The `ward-outage-reconnect-01` scenario validates Tier 1 safety during a central API outage. Observations and alerts continue on the ward gateway; acknowledgements are recorded locally and synced when the uplink returns.
|
||||
|
||||
### Automated verification (recommended)
|
||||
|
||||
From the repo root, with Docker running:
|
||||
|
||||
```bash
|
||||
./scripts/run-phase24-verification.sh
|
||||
```
|
||||
|
||||
This script runs all three phases automatically: central baseline replay, gateway replay during simulated central outage, and post-reconnect sync checks. Logs go to `/tmp/vigilcare-phase24-*`.
|
||||
|
||||
Useful flags:
|
||||
|
||||
| Env var | Effect |
|
||||
|---------|--------|
|
||||
| `SKIP_DOCKER=1` | Assume `docker compose` stack is already up |
|
||||
| `SKIP_PHASE_A=1` | Skip central replay; use an existing ICU encounter on the gateway |
|
||||
|
||||
Helper for manual gateway API calls (ward gateway has no login endpoint):
|
||||
|
||||
```bash
|
||||
export GATEWAY_JWT=$(./scripts/mint-gateway-jwt.sh nurse.demo NURSE)
|
||||
```
|
||||
|
||||
### Manual procedure
|
||||
|
||||
1. Full stack running with the ward gateway profile:
|
||||
|
||||
```bash
|
||||
docker compose --profile ward-gateway up -d
|
||||
```
|
||||
|
||||
2. Central API running and the gateway encounter replica synced
|
||||
3. Note an active ICU encounter id from the gateway:
|
||||
|
||||
```bash
|
||||
curl "http://localhost:5081/api/v1/encounters?status=ACTIVE&department=ICU" \
|
||||
-H "Authorization: Bearer $JWT"
|
||||
```
|
||||
|
||||
### Procedure
|
||||
|
||||
**Phase A — Baseline on central (optional):**
|
||||
|
||||
```bash
|
||||
dotnet run --project VigilCare.Simulator -- replay \
|
||||
VigilCare.Simulator/Scenarios/List/ward-outage-reconnect-01.json \
|
||||
--speed 0 --base-url http://localhost:5270
|
||||
```
|
||||
|
||||
**Phase B — Stop central, replay against gateway:**
|
||||
|
||||
```bash
|
||||
# Stop central API process/container
|
||||
dotnet run --project VigilCare.Simulator -- replay \
|
||||
VigilCare.Simulator/Scenarios/List/ward-outage-reconnect-01.json \
|
||||
--gateway --encounter-id <ENCOUNTER-GUID> --speed 60 --poll
|
||||
```
|
||||
|
||||
The `--gateway` flag targets `http://localhost:5081` automatically. `--encounter-id` must reference an encounter already replicated on the gateway. The `alert_ack` event at T+50 min sends `clinicianId: "RN-Wu"` from the scenario; the gateway honors this in `acknowledged_by` and in the sync buffer (login user is only used when `clinicianId` is omitted).
|
||||
|
||||
**Phase A note:** Central replay attributes acks to the JWT user (`physician.demo` by default). Use `--username nurse.demo` if you want a nurse role on central; gateway Phase B is the authoritative climate-resilience path for scenario attribution.
|
||||
|
||||
**Phase C — Restart central, verify sync:**
|
||||
|
||||
- Wait for `SyncUploaderService` to drain the buffer
|
||||
- Poll `GET /api/v1/operations/gateways` — buffer depth should reach 0
|
||||
- Confirm observations on central with preserved `recorded_at` timestamps
|
||||
|
||||
### Success criteria
|
||||
|
||||
- Critical potassium alert created on gateway at T+45 min while central is down
|
||||
- Ack recorded locally at T+50 min with `acknowledged_by` = `RN-Wu`
|
||||
- After reconnect: central has observations, alert, and ack; no duplicate paging logs
|
||||
|
||||
Reference in New Issue
Block a user