run test on phase 24 and 33

2026-06-24 04:35:15 +08:00
parent 68c397350b
commit 279add1e45
45 changed files with 732 additions and 99 deletions
@@ -0,0 +1,39 @@
+# Climate Resilience Chaos Experiments
+
+Six intentional failure injections validating Phases 20–23 ward gateway architecture.
+Methodology follows the six-step loop: BASELINE → HYPOTHESIS → BREAK → OBSERVE → FIX → VERIFY.
+
+| # | Experiment | Break | Success criterion | Automation |
+|---|---|---|---|---|
+| 1 | Central API down 30 min | Stop central API container/process | Local critical alert + ack on gateway; zero central observations during outage | Manual + `ward-outage-reconnect-01` |
+| 2 | Network partition mid-batch | `docker network disconnect` during sync upload | Partial retry; no duplicate observations on central | `scripts/demo-network-partition.sh` |
+| 3 | Kafka down during sync apply | `docker stop kafka` before batch process; restart after | Observations in PostgreSQL; `outbox_pending_events` drops after Kafka up | Grafana |
+| 4 | Duplicate batch re-upload | POST same `batchReference` twice | Second returns existing batch id; one set of rows | `ClinicalSyncBatchTests` |
+| 5 | Ack before alert in batch | Construct batch with ack before alert item | `CONFLICT` + `ALERT_NOT_YET_SYNCED`; reorder retry succeeds | `ClinicalSyncBatchTests` |
+| 6 | Gateway restart with buffer | `docker restart ward-gateway-api` before sync completes | Buffered items survive in gateway DB; sync resumes | Manual |
+
+## Artifact sets
+
+Each experiment has four files dated `YYYY-MM-DD-<scenario>-{baseline.json,broken.md,fixed.json,learned.md}`.
+
+| Experiment | Date run | Status |
+|---|---|---|
+| 1 — central-down | 2026-06-24 | Complete |
+| 2 — partition-mid-batch | 2026-06-24 | Complete |
+| 3 — kafka-down-sync | 2026-06-24 | Complete |
+| 4 — duplicate-batch | 2026-06-24 | Complete (integration test) |
+| 5 — ack-before-alert | 2026-06-24 | Complete (integration test) |
+| 6 — gateway-restart-buffer | 2026-06-24 | Complete |
+
+## Grafana panels to watch
+
+- `ward_gateways_offline_gauge`
+- `ward_gateway_buffer_depth`
+- `clinical_sync_batches_total`
+- `outbox_pending_events`
+- `alerts_unacknowledged_gauge`
+
+## Related scripts
+
+- `./scripts/demo-network-partition.sh`
+- `./scripts/run-phase24-verification.sh`