40 lines
2.1 KiB
Markdown
40 lines
2.1 KiB
Markdown
# Climate Resilience Chaos Experiments
|
||
|
||
Six intentional failure injections validating Phases 20–23 ward gateway architecture.
|
||
Methodology follows the six-step loop: BASELINE → HYPOTHESIS → BREAK → OBSERVE → FIX → VERIFY.
|
||
|
||
| # | Experiment | Break | Success criterion | Automation |
|
||
|---|---|---|---|---|
|
||
| 1 | Central API down 30 min | Stop central API container/process | Local critical alert + ack on gateway; zero central observations during outage | Manual + `ward-outage-reconnect-01` |
|
||
| 2 | Network partition mid-batch | `docker network disconnect` during sync upload | Partial retry; no duplicate observations on central | `scripts/demo-network-partition.sh` |
|
||
| 3 | Kafka down during sync apply | `docker stop kafka` before batch process; restart after | Observations in PostgreSQL; `outbox_pending_events` drops after Kafka up | Grafana |
|
||
| 4 | Duplicate batch re-upload | POST same `batchReference` twice | Second returns existing batch id; one set of rows | `ClinicalSyncBatchTests` |
|
||
| 5 | Ack before alert in batch | Construct batch with ack before alert item | `CONFLICT` + `ALERT_NOT_YET_SYNCED`; reorder retry succeeds | `ClinicalSyncBatchTests` |
|
||
| 6 | Gateway restart with buffer | `docker restart ward-gateway-api` before sync completes | Buffered items survive in gateway DB; sync resumes | Manual |
|
||
|
||
## Artifact sets
|
||
|
||
Each experiment has four files dated `YYYY-MM-DD-<scenario>-{baseline.json,broken.md,fixed.json,learned.md}`.
|
||
|
||
| Experiment | Date run | Status |
|
||
|---|---|---|
|
||
| 1 — central-down | 2026-06-24 | Complete |
|
||
| 2 — partition-mid-batch | 2026-06-24 | Complete |
|
||
| 3 — kafka-down-sync | 2026-06-24 | Complete |
|
||
| 4 — duplicate-batch | 2026-06-24 | Complete (integration test) |
|
||
| 5 — ack-before-alert | 2026-06-24 | Complete (integration test) |
|
||
| 6 — gateway-restart-buffer | 2026-06-24 | Complete |
|
||
|
||
## Grafana panels to watch
|
||
|
||
- `ward_gateways_offline_gauge`
|
||
- `ward_gateway_buffer_depth`
|
||
- `clinical_sync_batches_total`
|
||
- `outbox_pending_events`
|
||
- `alerts_unacknowledged_gauge`
|
||
|
||
## Related scripts
|
||
|
||
- `./scripts/demo-network-partition.sh`
|
||
- `./scripts/run-phase24-verification.sh`
|