# Climate Resilience Chaos Experiments

Six intentional failure injections validating Phases 20–23 ward gateway architecture.
Methodology follows the six-step loop: BASELINE → HYPOTHESIS → BREAK → OBSERVE → FIX → VERIFY.

| # | Experiment | Break | Success criterion | Automation |
|---|---|---|---|---|
| 1 | Central API down 30 min | Stop central API container/process | Local critical alert + ack on gateway; zero central observations during outage | Manual + `ward-outage-reconnect-01` |
| 2 | Network partition mid-batch | `docker network disconnect` during sync upload | Partial retry; no duplicate observations on central | `scripts/demo-network-partition.sh` |
| 3 | Kafka down during sync apply | `docker stop kafka` before batch process; restart after | Observations in PostgreSQL; `outbox_pending_events` drops after Kafka up | Grafana |
| 4 | Duplicate batch re-upload | POST same `batchReference` twice | Second returns existing batch id; one set of rows | `ClinicalSyncBatchTests` |
| 5 | Ack before alert in batch | Construct batch with ack before alert item | `CONFLICT` + `ALERT_NOT_YET_SYNCED`; reorder retry succeeds | `ClinicalSyncBatchTests` |
| 6 | Gateway restart with buffer | `docker restart ward-gateway-api` before sync completes | Buffered items survive in gateway DB; sync resumes | Manual |

## Artifact sets

Each experiment has four files dated `YYYY-MM-DD-<scenario>-{baseline.json,broken.md,fixed.json,learned.md}`.

| Experiment | Date run | Status |
|---|---|---|
| 1 — central-down | 2026-06-24 | Complete |
| 2 — partition-mid-batch | 2026-06-24 | Complete |
| 3 — kafka-down-sync | 2026-06-24 | Complete |
| 4 — duplicate-batch | 2026-06-24 | Complete (integration test) |
| 5 — ack-before-alert | 2026-06-24 | Complete (integration test) |
| 6 — gateway-restart-buffer | 2026-06-24 | Complete |

## Grafana panels to watch

- `ward_gateways_offline_gauge`
- `ward_gateway_buffer_depth`
- `clinical_sync_batches_total`
- `outbox_pending_events`
- `alerts_unacknowledged_gauge`

## Related scripts

- `./scripts/demo-network-partition.sh`
- `./scripts/run-phase24-verification.sh`