Files

T

voltsrage 185dc93fa1 Run initial test for Climate Resilience Verification Suite

Add first part of Alert Quality Analytics

2026-06-24 03:01:00 +08:00

12 KiB

Raw Blame History

VigilCare Clinical — Climate Resilience Extension Roadmap

Internal document | Phases 20–24 | June 2026

Prerequisites: Phases 1–19 complete — full clinical CDSS pipeline, ward dashboard, simulator, and clinician feedback mode.

Reference implementation: POS System API Phase 10 (phase-10-plan.md) — offline batch upload, async processor, two-layer idempotency, partial conflict model.

Problem Statement

Hospital clinical monitoring assumes reliable connectivity between the ward, the hospital datacenter, and any cloud-hosted central services. Climate and infrastructure shocks break that assumption:

Scenario	What fails	Clinical risk
Ward WiFi outage	Central API unreachable from bedside	Alerts never reach the ward dashboard
Hospital uplink loss (typhoon, fiber cut)	Central VigilCare unreachable for hours	Same — cloud-dependent alerting stops
Regional cloud outage	Central stack down	Population analytics lost; ward safety at risk if ward depends on cloud
Power blip (UPS handoff)	Gateway host restart	Buffered events lost if not persisted locally
Post-disaster partial recovery	Intermittent connectivity	Duplicate alerts or lost audit trail on sloppy sync

VigilCare Phases 1–19 run as a central clinical backend. Observations ingest to PostgreSQL, fan out through Kafka, and drive alerting, scoring, and archival. That architecture is correct when the network is healthy. It is not sufficient when the ward must operate alone for typhoon-length outages.

Climate resilience means: Tier 1 patient safety runs at the ward even when central is unreachable; Tier 3 analytics and archival catch up idempotently when connectivity returns.

This extension does not change the clinical domain (hospital wards, sepsis bundles, NEWS2). It adds a ward-first deployment model.

Design Principle — Three Tiers

Tier	Responsibility	Runs where	Must work offline?
Tier 1 — Safety	Critical + warning threshold alerts, ack/resolve, local paging, ward dashboard reads	`VigilCare.WardGateway`	Yes
Tier 2 — Reconciliation	Buffered event upload, idempotent apply, conflict reporting	Gateway → central sync API	On reconnect
Tier 3 — Analytics	NEWS2, SIRS, qSOFA, trend, sepsis bundle, Elasticsearch, Parquet	Central `VigilCareClinicalAPI`	No — replays after sync

Invariant (carry into every phase plan): Append-only observations, synchronous critical / asynchronous warning split, idempotency keys, DLQ escalation — same rules as Phases 1–19. The gateway is not a different clinical system; it is a degraded-mode replica of the hot path.

Target Architecture

┌─────────────────────────────────────────────────────────────────────────┐
│  Ward (UPS-backed server) — VigilCare.WardGateway                       │
│  ┌──────────────┐  ┌─────────────────┐  ┌──────────────────────────┐  │
│  │ Local ingest │→ │ Critical+warning│→ │ Local RabbitMQ paging    │  │
│  │ + idempotency│  │ eval (Tier 1)   │  │ + DLQ escalation         │  │
│  └──────────────┘  └─────────────────┘  └──────────────────────────┘  │
│         │                    │                      │                   │
│         └────────────────────┴──────────────────────┘                   │
│                              │                                          │
│                    ┌─────────▼─────────┐                                │
│                    │ Local PostgreSQL  │                                │
│                    │ + sync buffer     │                                │
│                    └─────────┬─────────┘                                │
└──────────────────────────────┼──────────────────────────────────────────┘
                               │ POST /sync/batches (when link up)
┌──────────────────────────────▼──────────────────────────────────────────┐
│  Central — VigilCareClinicalAPI                                         │
│  ClinicalSyncBatchProcessor → ObservationService → outbox → Kafka       │
│  → NEWS2 / SIRS / qSOFA / trend / ES / Parquet (Tier 3 replay)          │
└─────────────────────────────────────────────────────────────────────────┘

Phase Map

Phase	Name	Deliverable
20	Site & Gateway Registry	`ClinicalSite`, `WardGateway`, API key auth, `VigilCare.ClinicalContracts` sync DTOs
21	Ward Gateway Service	New `VigilCare.WardGateway` project — local ingest, Tier 1 eval, buffer, local paging
22	Clinical Sync Batch Engine	Central batch upload + `ClinicalSyncBatchProcessor` (POS Phase 10 adapted)
23	Degraded Operations Visibility	Ops fleet APIs, dashboard ops view, Grafana panels, partition demo script
24	Climate Resilience Verification	Outage scenario, 6 chaos experiments, interview Q&A, verify script

Dependency graph

Phase 20 (registry + contracts)
    ├── Phase 21 (ward gateway service)
    └── Phase 22 (central sync engine)
            ├── Phase 23 (ops visibility)
            └── Phase 24 (verification)

Implementation order: 20 → 21 → 22 → 23 → 24. Phase plans for 21 and 22 can be drafted in parallel once Phase 20 defines the sync contract.

Sync Payload Contract (summary)

Shared types live in VigilCare.ClinicalContracts (class library referenced by central API and WardGateway). Scaffold and manage projects per dotnet-solution-scaffolding.md. Full field definitions in phase-20-plan.md Step 1.

{
  "batchReference": "550e8400-e29b-41d4-a716-446655440000",
  "gatewayId": "...",
  "siteId": "...",
  "capturedAtUtc": "2026-06-20T14:30:00Z",
  "observations": [
    {
      "clientRef": "...",
      "idempotencyKey": "device-retry-key-001",
      "encounterId": "...",
      "observationCode": "HEART_RATE",
      "value": 118,
      "recordedAt": "2026-06-20T14:28:00Z",
      "source": "bedside_monitor"
    }
  ],
  "alertEvents": [
    {
      "clientAlertId": "...",
      "encounterId": "...",
      "alertType": "CRITICAL_HEART_RATE",
      "severity": "CRITICAL",
      "details": "...",
      "generatedAt": "2026-06-20T14:28:01Z"
    }
  ],
  "alertAcknowledgments": [
    {
      "clientRef": "...",
      "clientAlertId": "...",
      "clinicianId": "RN-Smith",
      "acknowledgedAt": "2026-06-20T14:29:00Z",
      "note": "At bedside"
    }
  ]
}

Replay order inside batch processor: observations → alert events → acknowledgments → resolve events. Ack before alert in the same batch → ClinicalSyncConflict with reason ALERT_NOT_YET_SYNCED.

Idempotency layers (mirror POS):

Batch level — unique index on batch_reference; re-upload returns existing batch.
Item level — observation idempotency_key; alert client_alert_id; silent skip on duplicate.

What Is Explicitly Out of Scope

Defer to vigilcare-clinical-roadmap.md:

FHIR read endpoints, real notification channels (SMS/Twilio), department-specific threshold configuration
Full local NEWS2 / SIRS / qSOFA / trend / sepsis bundle on the gateway (central Kafka replay after sync instead)
HIPAA / TFDA certification language
Home / infant monitoring domain pivots

Interview Talking Points

"How is this different from just running VigilCare in the hospital datacenter?"

Datacenter deployment still fails when the ward-to-datacenter link fails — common during typhoons, construction, or router failures. The gateway colocates Tier 1 safety with the ward LAN. Central adds cross-ward analytics when available.

"Why not run everything locally and skip central?"

NEWS2 composite scoring, SIRS window state, sepsis bundle compliance, Elasticsearch population views, and Parquet regulatory archival require the full Kafka multi-consumer pipeline. Duplicating that on every ward gateway is operationally expensive. The split is: safety local, intelligence central.

"How do you prevent duplicate critical pages after reconnect?"

Observation and alert idempotency keys survive the sync boundary. The batch processor silently skips items already applied. Alert events use client_alert_id — if central already has the alert from a partial sync, no second RabbitMQ page fires.

"What happens if a nurse acknowledges an alert locally but central never receives it?"

The acknowledgment is buffered. On sync, it applies to the matching alert by client_alert_id. If central never receives it (gateway lost before sync), the reconciliation job (Phase 7) still flags unacknowledged critical alerts on central — but during outage, local escalation (DLQ) is authoritative. Ops dashboard (Phase 23) shows buffer depth so IT knows reconciliation is pending.

"How does this relate to POS offline sync?"

Same pattern: edge node buffers writes, uploads batch with batchReference, central applies idempotently with partial conflicts, async processor via outbox + RabbitMQ. POS conflicts are inventory; clinical conflicts are ordering dependencies (ack before alert) or duplicate device retries.

Document Index

Document	Purpose
phase-20-plan.md	Site/gateway registry + shared contracts
phase-21-plan.md	Ward gateway service
phase-22-plan.md	Central sync batch engine
phase-23-plan.md	Ops visibility + dashboard
phase-24-plan.md	Verification suite
resilience-learning-plan.md	Chaos experiment methodology

After Phase 24, the portfolio claim becomes: "Clinical monitoring that continues during ward isolation and reconciles honestly when the link returns — with measured failure-mode evidence."

Verification Checklist (Phase 24)

Documentation complete when:

Phase plans 20–24 written (phase-20-plan.md … phase-24-plan.md)
ward-outage-reconnect-01.json scenario present
resilience/README.md experiment index populated
Six chaos experiments documented with baseline/broken/fixed artifacts
Interview questions #29–32 in interview-questions.md
scripts/run-phase24-verification.sh passes

Implementation complete when all phase exit gates in plans 20–24 pass and integration tests green.

12 KiB Raw Blame History Unescape Escape