feature: Ward Gateway Service (Local-First Clinical Path)

2026-06-23 16:45:38 +08:00
parent d8e142fffe
commit 1bf8359097
100 changed files with 5474 additions and 4 deletions
@@ -2,7 +2,7 @@

 A production-quality clinical backend built with ASP.NET Core 8, PostgreSQL, Apache Kafka, RabbitMQ, Elasticsearch, Redis, and MinIO. The domain models the observe-alert-acknowledge lifecycle at the center of any clinical monitoring system: patient encounters, continuous vital sign and lab result ingest, real-time sepsis and NEWS2 scoring, and clinician notification with automatic escalation.

-**Implementation status:** Twenty-seven planned phases are complete through Phase 31 (plus Phase 20) — from schema and CRUD through Kafka, Elasticsearch CQRS, sepsis detection, RabbitMQ paging with DLQ escalation, reconciliation jobs, Prometheus/Grafana observability, the MinIO Parquet data lake, clinical data model expansion, warning alerts and orders, the NEWS2 composite scoring engine, trend detection with alert suppression, qSOFA bedside screening, medication administration with alert correlation annotations, the console replay simulator, the **Vue 3 ward dashboard**, clinician feedback mode, **Glasgow Coma Scale (GCS) scoring**, **SOFA organ-dysfunction scoring with baseline tracking and delta sepsis alerts**, the **Sepsis-3 clinical refactor** (SIRS removed, qSOFA repositioned as screening, SOFA delta ≥ 2 triggers bundles), **frontend GCS entry and SOFA display**, **expanded simulator scenarios with clinical validation**, the **Site & Gateway Registry** with dual authentication, shared clinical sync contracts, and fleet health Prometheus gauges, the **FHIR R4 Inbound Facade** for EHR integration, and **Role-Based Access Control (RBAC) with clinical audit logging**. Post-phase hardening includes health check endpoints, Kafka poison pill protection, outbox dead-letter with retry tracking, data lake partial-commit safety, MRN sequence-based generation, FHIR bundle transaction rollback, **FHIR R4 read/search endpoints** (Patient and Encounter), **alert threshold deletion with audit trail**, **FHIR API key rotation** (constant-time multi-key validation), **authorization failure logging** with Prometheus metrics, and **JWT signing key validation** at startup. See [Implemented Phases](#implemented-phases) for the full breakdown. Guides: [dashboard-guide.md](docs/dashboard-guide.md) (technical), [clinical-testing-guide.md](docs/clinical-testing-guide.md) (doctors & nurses).
+**Implementation status:** Twenty-seven planned phases are complete through Phase 31 (plus Phase 20) — from schema and CRUD through Kafka, Elasticsearch CQRS, sepsis detection, RabbitMQ paging with DLQ escalation, reconciliation jobs, Prometheus/Grafana observability, the MinIO Parquet data lake, clinical data model expansion, warning alerts and orders, the NEWS2 composite scoring engine, trend detection with alert suppression, qSOFA bedside screening, medication administration with alert correlation annotations, the console replay simulator, the **Vue 3 ward dashboard**, clinician feedback mode, **Glasgow Coma Scale (GCS) scoring**, **SOFA organ-dysfunction scoring with baseline tracking and delta sepsis alerts**, the **Sepsis-3 clinical refactor** (SIRS removed, qSOFA repositioned as screening, SOFA delta ≥ 2 triggers bundles), **frontend GCS entry and SOFA display**, **expanded simulator scenarios with clinical validation**, the **Site & Gateway Registry** with dual authentication, shared clinical sync contracts, and fleet health Prometheus gauges, the **FHIR R4 Inbound Facade** for EHR integration, and **Role-Based Access Control (RBAC) with clinical audit logging**. Post-phase hardening includes health check endpoints, Kafka poison pill protection, outbox dead-letter with retry tracking, data lake partial-commit safety, MRN sequence-based generation, FHIR bundle transaction rollback, **FHIR R4 read/search endpoints** (Patient and Encounter), **alert threshold deletion with audit trail**, **FHIR API key rotation** (constant-time multi-key validation), **authorization failure logging** with Prometheus metrics, **JWT signing key validation** at startup, and **concurrency hardening** (transactional sepsis bundle creation, unique active encounter constraint). See [Implemented Phases](#implemented-phases) for the full breakdown. Guides: [dashboard-guide.md](docs/dashboard-guide.md) (technical), [clinical-testing-guide.md](docs/clinical-testing-guide.md) (doctors & nurses).

 ## Domain Model — How It Maps to a Real Clinical System

@@ -53,7 +53,7 @@ An `OutboxEvent` is written in the same transaction as any observation or alert,
 ## Features

 - **Patient Registration** — register patients with MRN generation; optional blood type, allergies, and emergency contact; paginated list with name (`ILIKE`) and MRN (exact) search; patient detail with active encounter summary
- **Encounter Management** — open encounters against a patient with optional room/bed and admission reason; encounter status state machine (`scheduled → active → discharged / cancelled`) with 409 on illegal transitions; optional discharge diagnosis on discharge; encounter timeline as a merged chronological view across status changes, observation summaries, and alerts
+- **Encounter Management** — open encounters against a patient with optional room/bed and admission reason; encounter status state machine (`scheduled → active → discharged / cancelled`) with 409 on illegal transitions; unique active encounter per type per patient enforced by partial unique index (concurrent duplicate attempts return 409); optional discharge diagnosis on discharge; encounter timeline as a merged chronological view across status changes, observation summaries, and alerts
 - **Alert Threshold Management** — configure per-observation-code numeric bounds (`criticalLow`, `warningLow`, `warningHigh`, `criticalHigh`) for 12 observation codes; thresholds pre-loaded into Redis on startup; write-through cache invalidation on update and delete; `DELETE /alert-thresholds/{id}` removes a threshold with `THRESHOLD_DELETED` audit trail and Redis cache invalidation
 - **Observation Ingest** — `POST /encounters/:id/observations` accepts single or small batch (up to 10); idempotency via `Idempotency-Key` header (partial unique index); plausibility validation per observation code; synchronous critical alert creation within the ingest transaction; warning-range breaches evaluated asynchronously by `WarningAlertService` (Kafka consumer group `warning-evaluator`); outbox event written in the same commit; cursor-paginated history on `(encounter_id, observation_code, recorded_at DESC)`
 - **Warning Threshold Alerts** — `WarningEvaluator` reads thresholds from Redis; creates `WARNING`-severity alerts for values above `warningHigh` or below `warningLow` that are not also critical breaches; idempotent `INSERT WHERE NOT EXISTS` per encounter and alert type while status is `OPEN` or `ACKNOWLEDGED`; warning alerts are indexed in Elasticsearch but not published to the RabbitMQ paging queue
@@ -439,6 +439,8 @@ tests/
    ├── QsofaCurrentTests.cs                    # qSOFA current API — Redis state, criteria breakdown
    ├── GcsScoringTests.cs                      # GCS component scoring, alerts, NEWS2/qSOFA integration paths
    ├── SofaScoringTests.cs                     # SOFA organ scores, baseline, delta alerts, carry-forward, vasopressors
+    ├── BackgroundServiceTests.cs                # Outbox relay, Kafka consumer, sepsis bundle monitor, reconciliation
+    ├── ConcurrencyTests.cs                      # Parallel patient MRN, sepsis bundle, observation idempotency, encounter open
    ├── GatewayRegistryTests.cs                 # Gateway register, heartbeat, API key auth, department filter
    ├── Helpers/GatewayAuthHelper.cs             # WithGatewayApiKey extension method for test clients
    ├── Auth/
@@ -459,6 +461,47 @@ VigilCare.ClinicalContracts/                      # Phase 20 — shared sync DTO
 VigilCare.ClinicalContracts.Tests/                # Contracts round-trip serialization tests
 └── ClinicalContractsTests.cs

+VigilCare.WardGateway/                            # Phase 21 — local-first ward edge API (separate DB/Redis/RabbitMQ)
+├── Dockerfile
+├── Program.cs
+├── Data/
+│   ├── GatewayDbContext.cs
+│   └── Configurations/                           # snake_case mappings mirroring central API patterns
+├── Domain/
+│   ├── Entities/                                 # ReplicaPatient, ReplicaEncounter, LocalObservation, …
+│   └── Enums/                                    # AlertType, EncounterStatus, BufferedSyncItemType, …
+├── Models/Records/ThresholdCacheEntry.cs
+├── Models/Records/Observation/IngestObservationRequest.cs
+├── Models/Records/Alert/AcknowledgeAlertRequest.cs
+├── Models/Central/CentralSyncDtos.cs              # DTOs for central API sync responses
+├── Models/Central/CentralApiJson.cs
+├── Validators/IngestObservationRequestValidator.cs
+├── Validators/AcknowledgeAlertRequestValidator.cs
+├── Services/
+│   ├── PlausibilityValidator.cs
+│   ├── LocalObservationService.cs
+│   ├── LocalWarningEvaluator.cs
+│   ├── LocalAlertService.cs
+│   ├── LocalPagingPublisher.cs
+│   ├── ObservationQueryService.cs
+│   ├── EncounterReadService.cs
+│   └── BufferedSyncWriter.cs
+├── Controllers/
+│   ├── ObservationsController.cs
+│   ├── AlertsController.cs
+│   ├── EncountersController.cs
+│   └── CentralRequiredController.cs
+├── Notifications/RabbitMqTopologyProvisioner.cs
+├── BackgroundService/
+│   ├── EncounterReplicaSyncService.cs
+│   ├── ThresholdCacheLoader.cs
+│   ├── CentralReachabilityService.cs
+│   ├── LocalPagingWorkerService.cs
+│   └── LocalEscalationWorkerService.cs
+├── Configurations/                               # GatewayOptions, CentralApiOptions, RabbitMqOptions
+├── Infrastructure/                               # Health checks (Redis, RabbitMQ, encounter replica ready)
+└── Migrations/                                   # InitialGatewaySchema
+
 VigilCare.Simulator/                              # Phase 16 — console replay simulator (HTTP-only, no direct DB/Kafka)
 ├── Program.cs                                    # CLI: replay, replay-all, validate, dry-run
 ├── Commands/                                     # System.CommandLine command handlers
@@ -598,10 +641,26 @@ This two-tier design prevents false-positive bundle activations — SIRS criteri

 ### Start Infrastructure

+**Central stack only** (default — no Compose profile):
+
 ```bash
 docker compose up -d
 ```

+**Ward gateway stack** (separate PostgreSQL, Redis, RabbitMQ, and gateway API on port 5081):
+
+```bash
+docker compose --profile ward-gateway up -d
+```
+
+**Everything** (central + ward):
+
+```bash
+docker compose --profile full up -d
+```
+
+Ward services use Compose profiles and do **not** start with a plain `docker compose up`. See `docs/docker-compose-usage-and-troubleshooting.md` §10.
+
 All services join the `vigilcare_net` bridge network so containers can reach each other by service name (e.g. Grafana → `http://prometheus:9090`). Connection strings in `appsettings.json` use **host** ports when running `dotnet run` on your machine.

 | Service | Host Port | Notes |
@@ -616,6 +675,15 @@ All services join the `vigilcare_net` bridge network so containers can reach eac
 | Prometheus 2.52 | 9101 | UI at `http://localhost:9101` — scrapes `GET /metrics` on the API |
 | Grafana 10.4 | 3101 | UI at `http://localhost:3101` — login: `admin` / `admin` |

+**Ward gateway stack** (`--profile ward-gateway` or `full`):
+
+| Service | Host Port | Notes |
+|---|---|---|
+| Ward PostgreSQL 16 | 5437 | Database: `vigilcare_ward`, user: `postgres`, password: `password` |
+| Ward Redis 7 | 6383 | No auth |
+| Ward RabbitMQ 3.13 | 5675 (AMQP), 15675 (UI) | login: `guest` / `guest` |
+| Ward Gateway API | 5081 | `VigilCare.WardGateway` — local-first clinical path (Phase 21) |
+
 **Seq first-run:** `SEQ_FIRSTRUN_ADMINPASSWORD=admin` is set in `docker-compose.yml`. This password is only applied on the very first container start (when the `/data` volume is empty). After initialization, the password is stored in the volume and this env var is ignored.

 ### Docker notes for Linux
@@ -728,6 +796,7 @@ Integration tests use `WebApplicationFactory` with a `Testing` environment and T
 | `ClinicalContractsTests` | 20 | ClinicalSyncBatchRequest JSON round-trip serialization |
 | `FhirIngestTests` | 30 | FHIR R4 patient upsert idempotency, LOINC observation mapping, unknown code 422, transaction bundle |
 | `RbacTests` | 31 | Unauthenticated 401, nurse 403 on threshold write, admin threshold update with audit log creation, alert acknowledge uses authenticated user |
+| `ConcurrencyTests` | — | Parallel patient registration unique MRNs, parallel sepsis alerts single bundle, parallel observation idempotency, parallel encounter open duplicate rejection |

 ### Verification Scripts

@@ -1350,7 +1419,7 @@ dischargedAt        DateTimeOffset?
 createdAt           DateTimeOffset
 ```

-Indexes: `(patient_id, admitted_at DESC)`, partial `(status, admitted_at DESC) WHERE status = 'active'`
+Indexes: `(patient_id, admitted_at DESC)`, partial `(status, admitted_at DESC) WHERE status = 'active'`, partial unique `(patient_id, encounter_type) WHERE status = 'ACTIVE'` (prevents duplicate active encounters of the same type per patient)

 ### AlertThreshold

@@ -1777,7 +1846,7 @@ When a `SOFA_SEPSIS` alert fires (delta ≥ 2 from baseline), `SepsisAlertHandle
 7. When all four elements are complete: `complianceStatus = COMPLIANT` (within deadline) or `NON_COMPLIANT` (past deadline); Prometheus `sepsis_bundle_compliance_total{status}` incremented
 8. If the deadline passes with incomplete elements, `SepsisBundleMonitorService` (polling every 5 min) marks the bundle `NON_COMPLIANT` — elements remain `PENDING` but the bundle status reflects the missed deadline

-**Idempotency:** Only one in-progress bundle can exist per encounter, enforced by a partial unique index `(encounter_id) WHERE compliance_status = 'IN_PROGRESS'`. A second alert for the same encounter returns early without creating a duplicate — the database constraint prevents TOCTOU race conditions even under concurrent SOFA scoring events.
+**Idempotency:** Only one in-progress bundle can exist per encounter, enforced by a partial unique index `(encounter_id) WHERE compliance_status = 'IN_PROGRESS'`. Bundle creation wraps order and bundle inserts in a single database transaction — if the unique constraint rejects a concurrent duplicate, the transaction rolls back all associated orders, preventing orphaned order records. A second alert for the same encounter returns early without side effects.

 ---

@@ -1921,6 +1990,8 @@ Twenty-seven phases from the project roadmap are implemented and verified, inclu

 **Site & Gateway Registry (Phase 20):** Central API manages clinical sites and ward edge nodes (gateways). Gateways authenticate via API key for heartbeat and future sync upload. Shared `VigilCare.ClinicalContracts` class library defines sync DTOs consumed by both central API and ward gateway projects. Prometheus fleet health gauges track offline gateways and buffer depth per site. Foundation for Phase 21 (WardGateway standalone service) and Phase 22 (sync batch upload).

+**Ward Gateway (Phase 21, in progress):** `VigilCare.WardGateway` is a separate ASP.NET deployable with its own PostgreSQL (`vigilcare_ward`), Redis, and RabbitMQ. Docker Compose services use the `ward-gateway` profile — start with `docker compose --profile ward-gateway up -d`. Domain entities mirror central `Patient`/`Encounter`/`Observation`/`ClinicalAlert` as replica or local types; see `docs/plans/phase-21-plan.md` Step 2.
+
 **Scoring pipeline (Phases 25–26):** GCS components → `gcs_scores` + `gcs.scored` → SOFA CNS organ system; SOFA lab/vital observations → `sofa_scores` with baseline tracking → delta sepsis alerts when organ dysfunction worsens.

 **Sepsis-3 refactor (Phases 27–29):** SIRS removed; qSOFA repositioned as bedside screening (`QSOFA_SCREEN`); SOFA delta ≥ 2 triggers `SOFA_SEPSIS` → sepsis bundle. Frontend gains GCS entry form and SOFA score panel. Eleven simulator scenarios validate the full clinical pipeline end-to-end.
@@ -1935,6 +2006,7 @@ Twenty-seven phases from the project roadmap are implemented and verified, inclu
 - **FHIR API key rotation** — `Fhir:ApiKeys` array alongside existing `Fhir:ApiKey` for zero-downtime key rotation; constant-time comparison via `CryptographicOperations.FixedTimeEquals` prevents timing attacks
 - **Authorization failure logging** — `PermissionAuthorizationHandler` logs denied requests with structured details (username, user ID, role, required permission, endpoint); Prometheus `authorization_failures_total` counter with `permission` and `role` labels
 - **JWT signing key validation** — startup guard rejects keys shorter than 256 bits (HMAC-SHA256 minimum); prevents silent misconfiguration that would weaken token verification
+- **Concurrency hardening** — `SepsisBundleService.TryCreateBundleAsync` wraps order + bundle creation in a single database transaction so the unique constraint rollback also reverts orphaned orders; `PatientService.OpenEncounterAsync` enforced by new partial unique index `ix_encounters_patient_active_type` on `(patient_id, encounter_type) WHERE status = 'ACTIVE'` with constraint-violation catch returning 409 Conflict; `ConcurrencyTests` validates parallel patient registration, sepsis bundle creation, observation idempotency, and encounter open race conditions
 - **New Prometheus metrics** — `fhir_read_total` (resource_type, interaction, outcome), `authorization_failures_total` (permission, role)
 - **New audit actions** — `THRESHOLD_DELETED`, `AUTHORIZATION_DENIED`