41 lines
1.5 KiB
Markdown
41 lines
1.5 KiB
Markdown
# Experiment: Kafka Down During Sync Apply
|
|
Date: 2026-06-24
|
|
Phase: 24 — Climate Resilience Verification
|
|
|
|
## 1. Hypothesis
|
|
If I stop Kafka while a clinical sync batch is processed, I expect observations to
|
|
land in PostgreSQL but outbox events to remain pending, because the transactional
|
|
outbox pattern decouples durable writes from async Kafka publication.
|
|
|
|
## 2. Baseline
|
|
- Encounter: `0f81a139-9f33-47ad-9df9-09c7a6c011c8`
|
|
- Outbox pending (`processed_at IS NULL`): 0
|
|
- Central observation count: 306
|
|
- Kafka: running
|
|
|
|
## 3. Break
|
|
```bash
|
|
docker stop $(docker ps -qf name=kafka)
|
|
|
|
curl -X POST http://localhost:5270/api/v1/sync/batches \
|
|
-H "X-Api-Key: dev-gateway-key-change-in-production" \
|
|
-H "X-Gateway-Id: 22222222-2222-2222-2222-222222222222" \
|
|
-H "Content-Type: application/json" \
|
|
-d '{ "batchReference": "<uuid>", "gatewayId": "...", "siteId": "...",
|
|
"capturedAtUtc": "<now>", "observations": [...], "alertEvents": [],
|
|
"alertAcknowledgments": [], "alertResolutions": [] }'
|
|
```
|
|
|
|
## 4. Observe
|
|
- Batch `37618ec9-5fc8-45bf-8edd-fcfcfc47230a` reached status **APPLIED**
|
|
- Observation `kafka-exp3-1782245857` present in PostgreSQL (count = 1)
|
|
- Outbox pending events rose to **1** while Kafka was stopped
|
|
- `OutboxRelayService` could not publish `observation.recorded` until broker returned
|
|
|
|
## 5. Fix / Outcome
|
|
- Bug found? **no**
|
|
- Fix applied: no fix needed — by design (outbox holds events until Kafka heals)
|
|
|
|
## 6. Verify
|
|
(Post-heal results in `2026-06-24-kafka-down-sync-fixed.json`)
|