Climate Resilience Chaos Experiments
Six intentional failure injections validating Phases 20–23 ward gateway architecture. Methodology follows the six-step loop: BASELINE → HYPOTHESIS → BREAK → OBSERVE → FIX → VERIFY.
| # | Experiment | Break | Success criterion | Automation |
|---|---|---|---|---|
| 1 | Central API down 30 min | Stop central API container/process | Local critical alert + ack on gateway; zero central observations during outage | Manual + ward-outage-reconnect-01 |
| 2 | Network partition mid-batch | docker network disconnect during sync upload |
Partial retry; no duplicate observations on central | scripts/demo-network-partition.sh |
| 3 | Kafka down during sync apply | docker stop kafka before batch process; restart after |
Observations in PostgreSQL; outbox_pending_events drops after Kafka up |
Grafana |
| 4 | Duplicate batch re-upload | POST same batchReference twice |
Second returns existing batch id; one set of rows | ClinicalSyncBatchTests |
| 5 | Ack before alert in batch | Construct batch with ack before alert item | CONFLICT + ALERT_NOT_YET_SYNCED; reorder retry succeeds |
ClinicalSyncBatchTests |
| 6 | Gateway restart with buffer | docker restart ward-gateway-api before sync completes |
Buffered items survive in gateway DB; sync resumes | Manual |
Artifact sets
Each experiment has four files dated YYYY-MM-DD-<scenario>-{baseline.json,broken.md,fixed.json,learned.md}.
| Experiment | Date run | Status |
|---|---|---|
| 1 — central-down | 2026-06-24 | Complete |
| 2 — partition-mid-batch | 2026-06-24 | Complete |
| 3 — kafka-down-sync | 2026-06-24 | Complete |
| 4 — duplicate-batch | 2026-06-24 | Complete (integration test) |
| 5 — ack-before-alert | 2026-06-24 | Complete (integration test) |
| 6 — gateway-restart-buffer | 2026-06-24 | Complete |
Grafana panels to watch
ward_gateways_offline_gaugeward_gateway_buffer_depthclinical_sync_batches_totaloutbox_pending_eventsalerts_unacknowledged_gauge
Related scripts
./scripts/demo-network-partition.sh./scripts/run-phase24-verification.sh