Experiment: Kafka Down During Sync Apply

Date: 2026-06-24 Phase: 24 — Climate Resilience Verification

1. Hypothesis

If I stop Kafka while a clinical sync batch is processed, I expect observations to land in PostgreSQL but outbox events to remain pending, because the transactional outbox pattern decouples durable writes from async Kafka publication.

2. Baseline

Encounter: 0f81a139-9f33-47ad-9df9-09c7a6c011c8
Outbox pending (processed_at IS NULL): 0
Central observation count: 306
Kafka: running

3. Break

docker stop $(docker ps -qf name=kafka)

curl -X POST http://localhost:5270/api/v1/sync/batches \
  -H "X-Api-Key: dev-gateway-key-change-in-production" \
  -H "X-Gateway-Id: 22222222-2222-2222-2222-222222222222" \
  -H "Content-Type: application/json" \
  -d '{ "batchReference": "<uuid>", "gatewayId": "...", "siteId": "...",
        "capturedAtUtc": "<now>", "observations": [...], "alertEvents": [],
        "alertAcknowledgments": [], "alertResolutions": [] }'

4. Observe

Batch 37618ec9-5fc8-45bf-8edd-fcfcfc47230a reached status APPLIED
Observation kafka-exp3-1782245857 present in PostgreSQL (count = 1)
Outbox pending events rose to 1 while Kafka was stopped
OutboxRelayService could not publish observation.recorded until broker returned

5. Fix / Outcome

Bug found? no
Fix applied: no fix needed — by design (outbox holds events until Kafka heals)

6. Verify

(Post-heal results in 2026-06-24-kafka-down-sync-fixed.json)

1.5 KiB Raw Permalink Blame History