vigilcare-clinical/docs/resilience/climate/README.md

Climate Resilience Chaos Experiments

Six intentional failure injections validating Phases 20–23 ward gateway architecture. Methodology follows the six-step loop: BASELINE → HYPOTHESIS → BREAK → OBSERVE → FIX → VERIFY.

#	Experiment	Break	Success criterion	Automation
1	Central API down 30 min	Stop central API container/process	Local critical alert + ack on gateway; zero central observations during outage	Manual + ward-outage-reconnect-01
2	Network partition mid-batch	docker network disconnect during sync upload	Partial retry; no duplicate observations on central	scripts/demo-network-partition.sh
3	Kafka down during sync apply	docker stop kafka before batch process; restart after	Observations in PostgreSQL; outbox_pending_events drops after Kafka up	Grafana
4	Duplicate batch re-upload	POST same batchReference twice	Second returns existing batch id; one set of rows	ClinicalSyncBatchTests
5	Ack before alert in batch	Construct batch with ack before alert item	CONFLICT + ALERT_NOT_YET_SYNCED; reorder retry succeeds	ClinicalSyncBatchTests
6	Gateway restart with buffer	docker restart ward-gateway-api before sync completes	Buffered items survive in gateway DB; sync resumes	Manual

Artifact sets

Each experiment has four files dated YYYY-MM-DD-<scenario>-{baseline.json,broken.md,fixed.json,learned.md}.

Experiment	Date run	Status
1 — central-down	2026-06-24	Complete
2 — partition-mid-batch	2026-06-24	Complete
3 — kafka-down-sync	2026-06-24	Complete
4 — duplicate-batch	2026-06-24	Complete (integration test)
5 — ack-before-alert	2026-06-24	Complete (integration test)
6 — gateway-restart-buffer	2026-06-24	Complete

Grafana panels to watch

• ward_gateways_offline_gauge
• ward_gateway_buffer_depth
• clinical_sync_batches_total
• outbox_pending_events
• alerts_unacknowledged_gauge

Related scripts

• ./scripts/demo-network-partition.sh
• ./scripts/run-phase24-verification.sh