run test on phase 24 and 33

2026-06-24 04:35:15 +08:00
parent 68c397350b
commit 279add1e45
45 changed files with 732 additions and 99 deletions
@@ -0,0 +1,9 @@
+{
+  "experiment": "ack-before-alert-in-batch",
+  "timestamp": "2026-06-24T04:19:00Z",
+  "testClass": "ClinicalSyncBatchTests",
+  "testMethod": "Processor_ConflictOnAckBeforeAlert",
+  "gatewayId": "22222222-2222-2222-2222-222222222222",
+  "batchOrdering": "acknowledgment before alert event",
+  "notes": "Pre-test baseline — integration fixture ready; batch will contain ack-only items with no preceding alert"
+}
@@ -0,0 +1,42 @@
+# Experiment: Ack Before Alert in Batch
+Date: 2026-06-24
+Phase: 24 — Climate Resilience Verification
+
+## 1. Hypothesis
+If I upload a sync batch containing an alert acknowledgment before the corresponding
+alert event, I expect the batch processor to record **CONFLICT** with reason
+`ALERT_NOT_YET_SYNCED`, because clinical sync enforces temporal ordering dependencies.
+
+## 2. Baseline
+- Test: `ClinicalSyncBatchTests.Processor_ConflictOnAckBeforeAlert`
+- Batch contains: `SyncedAlertAcknowledgment` only (no matching `SyncedAlertEvent`)
+- Client alert id: newly generated Guid
+
+## 3. Break
+```bash
+dotnet test VigilCareClinicalAPI.Tests \
+  --filter "FullyQualifiedName~ClinicalSyncBatchTests.Processor_ConflictOnAckBeforeAlert"
+```
+
+Batch payload (wrong order):
+```csharp
+new ClinicalSyncBatchRequest(batchRef, gatewayId, siteId, capturedAt,
+    observations: [],
+    alertEvents: [],
+    alertAcknowledgments: [new SyncedAlertAcknowledgment(..., clientAlertId, "RN-Smith", ...)],
+    alertResolutions: []);
+```
+
+## 4. Observe
+- Batch uploaded successfully (HTTP 201)
+- After `ProcessBatchAsync`: batch status = **CONFLICT**
+- Conflicts collection contains single entry with `ConflictReason: ALERT_NOT_YET_SYNCED`
+- Test result: **Passed** (309 ms)
+- No alert row created from premature ack
+
+## 5. Fix / Outcome
+- Bug found? **no** — conflict is the correct server-authoritative response
+- Fix applied: no fix needed — reorder and re-upload in a new batch (see plan Step 3 for full reorder test)
+
+## 6. Verify
+(Conflict assertions captured in `2026-06-24-ack-before-alert-fixed.json`)
@@ -0,0 +1,12 @@
+{
+  "experiment": "ack-before-alert-in-batch",
+  "timestamp": "2026-06-24T04:19:00Z",
+  "testClass": "ClinicalSyncBatchTests",
+  "testMethod": "Processor_ConflictOnAckBeforeAlert",
+  "testResult": "Passed",
+  "durationMs": 309,
+  "batchStatus": "CONFLICT",
+  "conflictReason": "ALERT_NOT_YET_SYNCED",
+  "conflictCount": 1,
+  "notes": "Premature ack rejected with ordering conflict; gateway must re-upload alert then ack in correct order"
+}
@@ -0,0 +1,7 @@
+Clinical sync correctly rejects out-of-order acknowledgments: when an ack arrives before
+its alert event, the batch processor marks the batch CONFLICT with `ALERT_NOT_YET_SYNCED`
+rather than silently creating orphan ack rows. This differs from POS inventory sync where
+ordering is less critical — in clinical workflows, temporal dependencies matter for audit
+trails and paging attribution. The integration test passed in 309 ms. A follow-up reorder
+retry (alert then ack in a new batch → APPLIED) is specified in Phase 24 Step 3
+`ClimateResilienceTests` for full end-to-end coverage.
@@ -0,0 +1,10 @@
+{
+  "experiment": "central-api-down-30min",
+  "timestamp": "2026-06-24T04:06:00Z",
+  "encounterId": "0f81a139-9f33-47ad-9df9-09c7a6c011c8",
+  "gatewayStatus": "ONLINE",
+  "gatewayBufferDepth": 0,
+  "centralObservationCount": 7,
+  "scenario": "ward-outage-reconnect-01",
+  "notes": "Pre-outage baseline after Phase A central replay — gateway heartbeat ONLINE, buffer empty, encounter replicated to ward gateway"
+}
@@ -0,0 +1,49 @@
+# Experiment: Central API Down 30 Minutes
+Date: 2026-06-24
+Phase: 24 — Climate Resilience Verification
+
+## 1. Hypothesis
+If I stop the central API while the ward gateway continues running and replay
+`ward-outage-reconnect-01` against the gateway, I expect critical potassium
+observations to alert locally on the gateway with zero new rows on central,
+because Tier 1 safety is ward-local and sync is deferred until reconnect.
+
+## 2. Baseline
+- Encounter: `0f81a139-9f33-47ad-9df9-09c7a6c011c8`
+- Gateway status: ONLINE
+- Gateway buffer depth: 0
+- Central observation count (encounter): 7 (Phase A replay complete)
+
+## 3. Break
+```bash
+SKIP_DOCKER=1 ./scripts/run-phase24-verification.sh
+# Internally: pkill central VigilCareClinicalAPI process, wait ~35s for gateway
+# to detect central unreachable, then replay scenario against gateway.
+```
+
+Manual equivalent:
+```bash
+pkill -f VigilCareClinicalAPI
+sleep 35
+dotnet run --project VigilCare.Simulator -- replay \
+  VigilCare.Simulator/Scenarios/List/ward-outage-reconnect-01.json \
+  --gateway --encounter-id 0f81a139-9f33-47ad-9df9-09c7a6c011c8 \
+  --gateway-token "$GATEWAY_JWT" --speed 0
+```
+
+## 4. Observe
+- Gateway created `CriticalPotassiumMeqL` alert at scenario T+45 min (K+ 6.8 mEq/L)
+- RN-Wu ack recorded locally at T+50 min (`status: Acknowledged`, `acknowledgedBy: RN-Wu`)
+- Central API unreachable during outage — no new central observations ingested
+- Gateway buffer depth (unsynced `buffered_sync_items`): **10**
+- Phase B replay sent 7 observations, 1 critical alert, 1 ack (plus warning-tier alert)
+- Gateway local alerts after Phase B:
+  - `CriticalPotassiumMeqL` — Acknowledged by RN-Wu
+  - `WarningPotassiumMeqL` — Open
+
+## 5. Fix / Outcome
+- Bug found? **no** — behavior matches design (local Tier 1 path + deferred sync)
+- Fix applied: no fix needed — by design
+
+## 6. Verify
+(Post-reconnect results captured in `2026-06-24-central-down-fixed.json`)
@@ -0,0 +1,21 @@
+{
+  "experiment": "central-api-down-30min",
+  "timestamp": "2026-06-24T04:08:30Z",
+  "encounterId": "0f81a139-9f33-47ad-9df9-09c7a6c011c8",
+  "gatewayStatus": "ONLINE",
+  "gatewayBufferDepth": 0,
+  "reportedBufferDepth": 0,
+  "centralObservationCount": 14,
+  "centralAlertCount": 1,
+  "syncedCriticalAlert": {
+    "alertType": "CriticalPotassiumMeqL",
+    "status": "Acknowledged",
+    "acknowledgedBy": "RN-Wu",
+    "syncedFromGateway": true
+  },
+  "centralAlertAcknowledged": true,
+  "duplicatePagingEvents": 0,
+  "syncBatchesApplied": 10,
+  "verificationScript": "scripts/run-phase24-verification.sh",
+  "notes": "After central restart + sync drain — 7 Phase A observations plus 7 gateway-synced observations; critical alert and RN-Wu ack reconciled on central with syncedFromGateway=true; buffer depth returned to 0"
+}
@@ -0,0 +1,11 @@
+Tier 1 invariants held on live run 2026-06-24: the observe-alert-acknowledge loop for
+critical hyperkalemia (K+ 6.8 mEq/L) completed entirely on the ward gateway while the
+central API was stopped. Ten sync items buffered locally during the outage; central
+received no new observations until reconnect. After central restart, `SyncUploaderService`
+drained the buffer within the 180s verification window — gateway reported buffer depth 0,
+and central showed the critical alert with `acknowledgedBy: RN-Wu` and
+`syncedFromGateway: true`. Central paging was suppressed for the gateway-synced alert
+(no duplicate critical pages). The run used instant replay (`--speed 0`) rather than a
+full 30-minute wall-clock outage, but the architectural property — ward-local safety
+with deferred idempotent reconciliation — was demonstrated end-to-end via
+`run-phase24-verification.sh`.
@@ -0,0 +1,10 @@
+{
+  "experiment": "duplicate-batch-re-upload",
+  "timestamp": "2026-06-24T04:19:00Z",
+  "testClass": "ClinicalSyncBatchTests",
+  "testMethod": "DuplicateBatchReference_ReturnsExisting",
+  "gatewayId": "22222222-2222-2222-2222-222222222222",
+  "batchReferenceStrategy": "same Guid posted twice",
+  "expectedBatchRowsInDb": 1,
+  "notes": "Pre-test baseline — isolated integration fixture with seeded gateway registry and active encounter"
+}
@@ -0,0 +1,34 @@
+# Experiment: Duplicate Batch Re-upload
+Date: 2026-06-24
+Phase: 24 — Climate Resilience Verification
+
+## 1. Hypothesis
+If I POST the same `batchReference` twice to `/api/v1/sync/batches`, I expect the
+second request to return the existing batch id with only one row in
+`clinical_sync_batches`, because batch reference is the idempotency key for gateway sync.
+
+## 2. Baseline
+- Test: `ClinicalSyncBatchTests.DuplicateBatchReference_ReturnsExisting`
+- Fixture: `ApiFixture` with `GatewayRegistrySeeder` and active encounter
+- Gateway auth: `X-Api-Key` + `X-Gateway-Id`
+
+## 3. Break
+```bash
+dotnet test VigilCareClinicalAPI.Tests \
+  --filter "FullyQualifiedName~ClinicalSyncBatchTests.DuplicateBatchReference_ReturnsExisting"
+```
+
+The test posts identical `ClinicalSyncBatchRequest` (same `batchReference`) twice.
+
+## 4. Observe
+- First POST: HTTP 201 Created, returns `batchId`
+- Second POST: returns **same** `batchId` as first response
+- Database: exactly **1** row in `clinical_sync_batches` for that `batchReference`
+- Test result: **Passed** (702 ms)
+
+## 5. Fix / Outcome
+- Bug found? **no**
+- Fix applied: no fix needed — by design
+
+## 6. Verify
+(Assertions captured in `2026-06-24-duplicate-batch-fixed.json`)
@@ -0,0 +1,11 @@
+{
+  "experiment": "duplicate-batch-re-upload",
+  "timestamp": "2026-06-24T04:19:00Z",
+  "testClass": "ClinicalSyncBatchTests",
+  "testMethod": "DuplicateBatchReference_ReturnsExisting",
+  "testResult": "Passed",
+  "durationMs": 702,
+  "secondBatchIdEqualsFirst": true,
+  "clinicalSyncBatchRowsForReference": 1,
+  "notes": "Duplicate POST returned existing batch id; no duplicate batch rows created"
+}
@@ -0,0 +1,6 @@
+Batch reference idempotency works as specified: replaying the same gateway sync payload
+does not create a second batch row or re-process observations. This is the clinical
+equivalent of POS offline sync deduplication and protects against network retries after
+partial ACK. The integration test `DuplicateBatchReference_ReturnsExisting` passed in
+702 ms, giving CI-guaranteed evidence for interview demos without requiring a manual
+chaos run.
@@ -0,0 +1,9 @@
+{
+  "experiment": "gateway-restart-with-buffer",
+  "timestamp": "2026-06-24T04:21:00Z",
+  "encounterId": "0f81a139-9f33-47ad-9df9-09c7a6c011c8",
+  "gatewayStatus": "ONLINE",
+  "gatewayBufferDepth": 2,
+  "centralApiStatus": "stopped",
+  "notes": "Pre-restart baseline — central API stopped, 3 new observations posted to gateway, 5 total unsynced buffer items"
+}
@@ -0,0 +1,35 @@
+# Experiment: Gateway Restart With Buffer
+Date: 2026-06-24
+Phase: 24 — Climate Resilience Verification
+
+## 1. Hypothesis
+If I restart the ward gateway API container while unsynced items remain in
+`buffered_sync_items`, I expect the buffer count to survive the restart unchanged,
+because persistence lives in gateway PostgreSQL, not the container process.
+
+## 2. Baseline
+- Encounter: `0f81a139-9f33-47ad-9df9-09c7a6c011c8`
+- Central API: stopped (simulated uplink loss)
+- Observations posted to gateway: 3 (HR 142, SpO2 91%, Temp 38.2°C)
+- Unsynced buffer count: **5** (includes prior buffered items from earlier experiments)
+
+## 3. Break
+```bash
+# Central already stopped
+docker restart $(docker ps -qf name=ward-gateway-api)
+# Wait ~35s for gateway health checks
+```
+
+## 4. Observe
+- Gateway container restarted successfully
+- Unsynced buffer count **before restart: 5**
+- Unsynced buffer count **after restart: 5** (unchanged)
+- No buffered rows lost from gateway PostgreSQL
+- Gateway `/health/ready` returned Healthy after ~35s
+
+## 5. Fix / Outcome
+- Bug found? **no**
+- Fix applied: no fix needed — by design (buffer in ward DB)
+
+## 6. Verify
+(Post-sync results in `2026-06-24-gateway-restart-buffer-fixed.json`)
@@ -0,0 +1,12 @@
+{
+  "experiment": "gateway-restart-with-buffer",
+  "timestamp": "2026-06-24T04:25:00Z",
+  "encounterId": "0f81a139-9f33-47ad-9df9-09c7a6c011c8",
+  "idempotencyPrefix": "gateway-restart-exp6-1782246096",
+  "bufferBeforeRestart": 5,
+  "bufferAfterRestart": 5,
+  "bufferAfterCentralSync": 0,
+  "observationsSyncedToCentral": 3,
+  "centralRestartWaitSeconds": 120,
+  "notes": "Buffer survived gateway container restart; after central API restart all 3 new observations synced and buffer drained to 0"
+}
@@ -0,0 +1,7 @@
+Gateway restart did not erase the sync backlog: unsynced `buffered_sync_items` count
+remained at 5 through the container restart because the ward PostgreSQL database is
+independent of the API process lifecycle. After central API was restarted, sync resumed
+automatically — all three experiment observations (`gateway-restart-exp6-1782246096-*`)
+appeared on central and the buffer drained to 0 within 120 seconds. This validates the
+UPS + ward-server deployment model: brief gateway process restarts (deployments, OOM kills)
+do not lose clinical events already accepted at the bedside.
@@ -0,0 +1,9 @@
+{
+  "experiment": "kafka-down-during-sync-apply",
+  "timestamp": "2026-06-24T04:17:00Z",
+  "encounterId": "0f81a139-9f33-47ad-9df9-09c7a6c011c8",
+  "outboxPendingEvents": 0,
+  "centralObservationCount": 306,
+  "kafkaStatus": "running",
+  "notes": "Pre-break baseline — Kafka up, outbox empty, central API accepting sync batches"
+}
@@ -0,0 +1,40 @@
+# Experiment: Kafka Down During Sync Apply
+Date: 2026-06-24
+Phase: 24 — Climate Resilience Verification
+
+## 1. Hypothesis
+If I stop Kafka while a clinical sync batch is processed, I expect observations to
+land in PostgreSQL but outbox events to remain pending, because the transactional
+outbox pattern decouples durable writes from async Kafka publication.
+
+## 2. Baseline
+- Encounter: `0f81a139-9f33-47ad-9df9-09c7a6c011c8`
+- Outbox pending (`processed_at IS NULL`): 0
+- Central observation count: 306
+- Kafka: running
+
+## 3. Break
+```bash
+docker stop $(docker ps -qf name=kafka)
+
+curl -X POST http://localhost:5270/api/v1/sync/batches \
+  -H "X-Api-Key: dev-gateway-key-change-in-production" \
+  -H "X-Gateway-Id: 22222222-2222-2222-2222-222222222222" \
+  -H "Content-Type: application/json" \
+  -d '{ "batchReference": "<uuid>", "gatewayId": "...", "siteId": "...",
+        "capturedAtUtc": "<now>", "observations": [...], "alertEvents": [],
+        "alertAcknowledgments": [], "alertResolutions": [] }'
+```
+
+## 4. Observe
+- Batch `37618ec9-5fc8-45bf-8edd-fcfcfc47230a` reached status **APPLIED**
+- Observation `kafka-exp3-1782245857` present in PostgreSQL (count = 1)
+- Outbox pending events rose to **1** while Kafka was stopped
+- `OutboxRelayService` could not publish `observation.recorded` until broker returned
+
+## 5. Fix / Outcome
+- Bug found? **no**
+- Fix applied: no fix needed — by design (outbox holds events until Kafka heals)
+
+## 6. Verify
+(Post-heal results in `2026-06-24-kafka-down-sync-fixed.json`)
@@ -0,0 +1,14 @@
+{
+  "experiment": "kafka-down-during-sync-apply",
+  "timestamp": "2026-06-24T04:19:00Z",
+  "batchReference": "64bee532-e76f-4799-9831-76c450e3fe69",
+  "batchId": "37618ec9-5fc8-45bf-8edd-fcfcfc47230a",
+  "batchStatus": "APPLIED",
+  "idempotencyKey": "kafka-exp3-1782245857",
+  "obsInPostgresDuringKafkaDown": 1,
+  "outboxPendingBefore": 0,
+  "outboxPendingDuringKafkaDown": 1,
+  "outboxPendingAfterKafkaRestart": 0,
+  "kafkaHealWaitSeconds": 60,
+  "notes": "Observation durable in PostgreSQL during Kafka outage; outbox drained to 0 within 60s of broker restart"
+}
@@ -0,0 +1,6 @@
+Stopping Kafka mid-sync did not block clinical data persistence: the sync batch processor
+applied the observation to PostgreSQL and left one outbox row pending (`processed_at IS NULL`).
+When Kafka restarted, `OutboxRelayService` drained the backlog within 60 seconds
+(outbox pending 1 → 0). This demonstrates the intended separation between Tier 1 durable
+writes and Tier 2 async scoring pipelines — NEWS2 and downstream consumers may lag during
+broker outages, but observations are not lost and replay correctly once Kafka is available.
@@ -0,0 +1,9 @@
+{
+  "experiment": "network-partition-mid-batch",
+  "timestamp": "2026-06-24T04:12:00Z",
+  "encounterId": "0f81a139-9f33-47ad-9df9-09c7a6c011c8",
+  "gatewayStatus": "ONLINE",
+  "gatewayBufferDepth": 0,
+  "centralObservationCount": 14,
+  "notes": "Pre-partition baseline — gateway ONLINE, empty buffer, central reachable"
+}
@@ -0,0 +1,41 @@
+# Experiment: Network Partition Mid-Batch
+Date: 2026-06-24
+Phase: 24 — Climate Resilience Verification
+
+## 1. Hypothesis
+If I disconnect the ward gateway from the central Docker network while it accepts
+observations, I expect events to buffer locally and fleet status to show DEGRADED,
+because sync upload cannot reach central until the partition heals.
+
+## 2. Baseline
+- Encounter: `0f81a139-9f33-47ad-9df9-09c7a6c011c8`
+- Gateway status: ONLINE
+- Gateway buffer depth: 0
+- Central observation count: 14
+
+## 3. Break
+```bash
+GATEWAY_CONTAINER=$(docker ps -qf name=ward-gateway-api)
+NETWORK=$(docker network ls --format '{{.Name}}' | grep vigilcare | head -1)
+docker network disconnect "$NETWORK" "$GATEWAY_CONTAINER"
+
+curl -X POST "http://localhost:5081/api/v1/encounters/$ENCOUNTER_ID/observations" \
+  -H "Authorization: Bearer $GATEWAY_JWT" \
+  -H "Content-Type: application/json" \
+  -d '{"observationCode":"HEART_RATE","value":165,"unit":"bpm","source":"DEVICE",
+       "recordedAt":"<UTC-now>","idempotencyKey":"partition-exp2-1782245574"}'
+```
+
+## 4. Observe
+- Gateway accepted observation locally (HTTP 201)
+- Fleet status transitioned to **DEGRADED** (`reportedBufferDepth: 2`)
+- Gateway DB unsynced buffer count: **2**
+- Central could not receive the observation during partition
+- `minutesSinceHeartbeat` remained near zero (gateway process still running)
+
+## 5. Fix / Outcome
+- Bug found? **no**
+- Fix applied: no fix needed — by design
+
+## 6. Verify
+(Post-heal results in `2026-06-24-partition-mid-batch-fixed.json`)
@@ -0,0 +1,14 @@
+{
+  "experiment": "network-partition-mid-batch",
+  "timestamp": "2026-06-24T04:14:00Z",
+  "encounterId": "0f81a139-9f33-47ad-9df9-09c7a6c011c8",
+  "idempotencyKey": "partition-exp2-1782245574",
+  "gatewayStatus": "ONLINE",
+  "gatewayBufferDepth": 0,
+  "centralObservationCountBefore": 14,
+  "centralObservationCountAfter": 15,
+  "duplicateObservationRows": 1,
+  "healCommand": "docker network connect $NETWORK $GATEWAY_CONTAINER",
+  "syncWaitSeconds": 60,
+  "notes": "After partition heal — buffer drained to 0, exactly one central row for partition idempotency key, no duplicates"
+}
@@ -0,0 +1,7 @@
+Network partition behaved as designed: the gateway continued accepting bedside observations
+while disconnected from central, fleet ops showed DEGRADED with buffer depth 2, and
+reconnecting the Docker network allowed `SyncUploaderService` to drain the backlog within
+60 seconds. The partition observation (`partition-exp2-1782245574`) appeared exactly once
+on central (count 14 → 15), confirming idempotent sync rather than duplicate rows. This
+validates the Phase 23 ops visibility story: charge nurses and IT can see degraded gateways
+before data loss occurs, and healing the uplink reconciles without manual intervention.
@@ -0,0 +1,32 @@
+# Experiment: {name}
+Date: {YYYY-MM-DD}
+Phase: 24 — Climate Resilience Verification
+
+## 1. Hypothesis
+If I {break}, I expect {observation} because {reason}.
+
+## 2. Baseline
+- Gateway status: {ONLINE/DEGRADED/OFFLINE}
+- Gateway buffer depth: {N}
+- Central observation count (encounter): {N}
+- Grafana snapshot: `{date}-baseline.png`
+
+## 3. Break
+- Command: `{exact command}`
+- Single fault only — no other changes
+
+## 4. Observe
+- What happened:
+- Metrics at T+5 min:
+  - `ward_gateways_offline_gauge`: 
+  - `ward_gateway_buffer_depth`: 
+  - `clinical_sync_batches_total`: 
+  - `outbox_pending_events`: 
+
+## 5. Fix / Outcome
+- Bug found? {yes/no}
+- Fix applied: {PR link or "no fix needed — by design"}
+
+## 6. Verify
+- Repeated measurement after fix/heal:
+- Learned (one paragraph):
@@ -0,0 +1,39 @@
+# Climate Resilience Chaos Experiments
+
+Six intentional failure injections validating Phases 20–23 ward gateway architecture.
+Methodology follows the six-step loop: BASELINE → HYPOTHESIS → BREAK → OBSERVE → FIX → VERIFY.
+
+| # | Experiment | Break | Success criterion | Automation |
+|---|---|---|---|---|
+| 1 | Central API down 30 min | Stop central API container/process | Local critical alert + ack on gateway; zero central observations during outage | Manual + `ward-outage-reconnect-01` |
+| 2 | Network partition mid-batch | `docker network disconnect` during sync upload | Partial retry; no duplicate observations on central | `scripts/demo-network-partition.sh` |
+| 3 | Kafka down during sync apply | `docker stop kafka` before batch process; restart after | Observations in PostgreSQL; `outbox_pending_events` drops after Kafka up | Grafana |
+| 4 | Duplicate batch re-upload | POST same `batchReference` twice | Second returns existing batch id; one set of rows | `ClinicalSyncBatchTests` |
+| 5 | Ack before alert in batch | Construct batch with ack before alert item | `CONFLICT` + `ALERT_NOT_YET_SYNCED`; reorder retry succeeds | `ClinicalSyncBatchTests` |
+| 6 | Gateway restart with buffer | `docker restart ward-gateway-api` before sync completes | Buffered items survive in gateway DB; sync resumes | Manual |
+
+## Artifact sets
+
+Each experiment has four files dated `YYYY-MM-DD-<scenario>-{baseline.json,broken.md,fixed.json,learned.md}`.
+
+| Experiment | Date run | Status |
+|---|---|---|
+| 1 — central-down | 2026-06-24 | Complete |
+| 2 — partition-mid-batch | 2026-06-24 | Complete |
+| 3 — kafka-down-sync | 2026-06-24 | Complete |
+| 4 — duplicate-batch | 2026-06-24 | Complete (integration test) |
+| 5 — ack-before-alert | 2026-06-24 | Complete (integration test) |
+| 6 — gateway-restart-buffer | 2026-06-24 | Complete |
+
+## Grafana panels to watch
+
+- `ward_gateways_offline_gauge`
+- `ward_gateway_buffer_depth`
+- `clinical_sync_batches_total`
+- `outbox_pending_events`
+- `alerts_unacknowledged_gauge`
+
+## Related scripts
+
+- `./scripts/demo-network-partition.sh`
+- `./scripts/run-phase24-verification.sh`