test: run verification test

This commit is contained in:
voltsrage
2026-06-18 16:16:57 +08:00
parent 7d9e53fb8d
commit ddde7fee31
2 changed files with 395 additions and 8 deletions
+61 -8
View File
@@ -50,7 +50,9 @@ An `OutboxEvent` is written in the same transaction as any observation or alert,
- **Patient Registration** — register patients with MRN generation; optional blood type, allergies, and emergency contact; paginated list with name (`ILIKE`) and MRN (exact) search; patient detail with active encounter summary
- **Encounter Management** — open encounters against a patient with optional room/bed and admission reason; encounter status state machine (`scheduled → active → discharged / cancelled`) with 409 on illegal transitions; optional discharge diagnosis on discharge; encounter timeline as a merged chronological view across status changes, observation summaries, and alerts
- **Alert Threshold Management** — configure per-observation-code numeric bounds (`criticalLow`, `warningLow`, `warningHigh`, `criticalHigh`) for 12 observation codes; thresholds pre-loaded into Redis on startup; write-through cache invalidation on update
- **Observation Ingest** — `POST /encounters/:id/observations` accepts single or small batch (up to 10); idempotency via `Idempotency-Key` header (partial unique index); plausibility validation per observation code; synchronous critical alert creation within the ingest transaction; warning breach deferred to Kafka consumer; outbox event written in the same commit; cursor-paginated history on `(encounter_id, observation_code, recorded_at DESC)`
- **Observation Ingest** — `POST /encounters/:id/observations` accepts single or small batch (up to 10); idempotency via `Idempotency-Key` header (partial unique index); plausibility validation per observation code; synchronous critical alert creation within the ingest transaction; warning-range breaches evaluated asynchronously by `WarningAlertService` (Kafka consumer group `warning-evaluator`); outbox event written in the same commit; cursor-paginated history on `(encounter_id, observation_code, recorded_at DESC)`
- **Warning Threshold Alerts** — `WarningEvaluator` reads thresholds from Redis; creates `WARNING`-severity alerts for values above `warningHigh` or below `warningLow` that are not also critical breaches; idempotent `INSERT WHERE NOT EXISTS` per encounter and alert type while status is `OPEN` or `ACKNOWLEDGED`; warning alerts are indexed in Elasticsearch but not published to the RabbitMQ paging queue
- **Clinical Order Management** — `POST /encounters/:id/orders` create; `GET /encounters/:id/orders` list with optional status filter; `GET /orders/:id` detail; `PATCH /orders/:id/status` status transitions; `PATCH /orders/:id/result` record result and transition to `Resulted`; status machine enforces `Pending → InProgress → Resulted` and terminal `Cancelled`
- **Clinical Alert Lifecycle** — paginated alert list per encounter and globally; acknowledge with clinician ID and optional note; resolve (must be acknowledged first); global list filterable by status, severity, and department
- **Outbox Relay** — `IHostedService` polling every 500ms; reads unprocessed outbox rows, publishes to Kafka, marks processed; partitioned by `encounterId` for per-encounter ordering
- **Kafka Pipeline** — three topics (`observation.recorded`, `alert.generated`, `encounter.status.changed`) with six partitions each; KRaft mode, no Zookeeper; `KAFKA_AUTO_CREATE_TOPICS_ENABLE=false` — topics are provisioned explicitly by `KafkaTopicProvisioner`
@@ -59,7 +61,8 @@ An `OutboxEvent` is written in the same transaction as any observation or alert,
- **RabbitMQ Notification Workers** — `NotificationPublisherService` reads `alert.generated` from Kafka and publishes paging jobs to `alerts.paging.queue`; `PagingWorkerService` sends the page and waits for acknowledgment; if no ack arrives before timeout it NACKs to `alerts.paging.dlq` with `x-message-ttl = 300000ms`; if the host is stopping, in-flight paging messages are NACKed with `requeue=true` so they are retried after restart and do not false-escalate; `EscalationWorkerService` pages the on-call backup and sets alert status to `escalated`; `DischargeSummaryWorkerService` reads `encounter.status.changed`, generates a discharge summary, and stores it in MinIO under `/discharge-summaries/{encounterId}/summary.pdf`
- **Data Lake Writer** — `DataLakeWriterService` (consumer group `data-lake-writer`) buffers `observation.recorded`, `alert.generated`, and `encounter.status.changed` events, flushes date-partitioned Parquet files to MinIO (`/observations/`, `/alerts/`, `/encounters/`), and commits Kafka offsets only after successful uploads; `kafka_partition` and `kafka_offset` columns provide audit lineage
- **Reconciliation Jobs** — three scheduled checks: (1) unacknowledged CRITICAL alerts older than 30 minutes, (2) pending orders without results after 4 hours, (3) active inpatients with no observation in 2 hours; each finding creates a `reconciliation_alerts` row and publishes to RabbitMQ
- **Standard Envelope** — all responses use a consistent `{ success, statusCode, data, error }` wrapper; validation errors use the same shape; `ApiBehaviorOptions` overridden so model validation also produces the standard envelope
- **Standard Envelope** — all responses use a consistent `{ success, statusCode, data, error }` wrapper; validation errors use the same shape; `ApiBehaviorOptions` overridden so model validation also produces the standard envelope with field-level `details`
- **Input Validation** — FluentValidation validators on all request DTOs (patient registration, encounter open, observation ingest, alert acknowledge, alert thresholds, orders); invalid requests return 400 before reaching the service layer
- **Observability** — Serilog structured logging enriched with `correlationId`, `encounterId`, `patientId` on alert paths; Seq sink (`http://localhost:5345`); Prometheus (`http://localhost:9101`) scrapes `GET /metrics`; eight application metric families via `ClinicalMetrics` and three background collectors (`AlertsUnacknowledgedCollector`, `OutboxPendingCollector`, `KafkaConsumerLagCollector`); Grafana clinical dashboard (`http://localhost:3101`, admin/admin) with `alerts_unacknowledged_gauge` as the primary safety panel; per-request correlation IDs in request logs and `X-Correlation-Id` response headers
- **Swagger UI** — OpenAPI spec via Swashbuckle (Development only)
@@ -85,6 +88,7 @@ IHostedServices (background):
OutboxRelayService → PostgreSQL outbox → Kafka (every 500ms)
EsIndexerService → Kafka → Elasticsearch (consumer group: es-indexer)
SepsisEngineService → Kafka → Redis SIRS state → PostgreSQL alert (consumer group: sepsis-engine)
WarningAlertService → Kafka → WarningEvaluator → PostgreSQL WARNING alert (consumer group: warning-evaluator)
NotificationPublisherService → Kafka → RabbitMQ paging.queue (consumer group: notification-publisher)
PagingWorkerService → RabbitMQ paging.queue → log page → NACK on timeout (or requeue on shutdown)
EscalationWorkerService → RabbitMQ escalation.queue → update alert status
@@ -117,6 +121,7 @@ IHostedServices (background):
| Dashboards | Prometheus 2.52 + Grafana 10.4 |
| Data lake format | Parquet.Net 4.x |
| Docs | Swagger / OpenAPI (Swashbuckle) |
| Validation | FluentValidation.AspNetCore |
| Testing | xUnit + Testcontainers + WebApplicationFactory |
---
@@ -133,6 +138,7 @@ VigilCareClinicalAPI/
│ ├── ObservationsController.cs # Ingest POST, cursor-paginated GET
│ ├── AlertThresholdsController.cs # Threshold CRUD + cache invalidation
│ ├── AlertsController.cs # Alert list (global + per-encounter), acknowledge, resolve
│ ├── OrdersController.cs # Order create, list, get, status transition, record result
│ └── AnalyticsController.cs # Elasticsearch-backed patient search, trend, alert summary, population
├── Domains/
│ ├── Entities/
@@ -149,7 +155,7 @@ VigilCareClinicalAPI/
│ ├── EncounterType.cs # Inpatient, Outpatient, Emergency
│ ├── AlertSeverity.cs # Warning, Critical
│ ├── AlertStatus.cs # Open, Acknowledged, Resolved, Escalated
│ ├── AlertType.cs # Threshold breach, sepsis, systolic BP, AVPU, glucose, …
│ ├── AlertType.cs # Threshold breach, sepsis, warning*, systolic BP, AVPU, glucose, …
│ ├── BloodType.cs # A+, O-, AB-, … with ToDbString/FromDbString
│ ├── ObservationSource.cs # Device, Manual, Lab
│ └── OrderType.cs / ReconciliationCheckType.cs / Department.cs / OrderStatus.cs
@@ -165,8 +171,11 @@ VigilCareClinicalAPI/
│ ├── ObservationService.cs # Ingest transaction: idempotency → plausibility → threshold → alert → outbox; emits Prometheus counters
│ ├── ObservationQueryService.cs # Cursor-paginated history
│ ├── AlertService.cs # Acknowledge, resolve, list
│ ├── OrderService.cs # Order lifecycle; status machine; ConflictException on illegal transitions
│ ├── WarningEvaluator.cs # Warning-range threshold evaluation; idempotent alert INSERT
│ ├── AnalyticsService.cs # Elasticsearch query wrappers
│ └── PlausibilityValidator.cs # Per-code numeric range guard
├── Validators/ # FluentValidation — RegisterPatient, OpenEncounter, IngestObservation, …
├── Observability/
│ └── Metrics/
│ └── ClinicalMetrics.cs # Eight Prometheus metric families (counters, histogram, gauges)
@@ -182,6 +191,7 @@ VigilCareClinicalAPI/
│ │ ├── ElasticIndexProvisioner.cs # Creates patient_encounters, observations, clinical_alerts indices
│ │ └── EsIndexerService.cs # consumer group: es-indexer; upserts Elasticsearch documents
│ ├── SepsisEngineService.cs # consumer group: sepsis-engine; SIRS eval via Redis TTL keys
│ ├── WarningAlertService.cs # consumer group: warning-evaluator; observation.recorded → WARNING alerts
│ ├── Notifications/
│ │ ├── NotificationPublisherService.cs # consumer group: notification-publisher; alert.generated → RabbitMQ paging.queue
│ │ ├── PagingWorkerService.cs # RabbitMQ consumer; logs page; NACK on ack timeout → DLQ, requeue on graceful shutdown
@@ -250,7 +260,10 @@ tests/
├── ReconciliationTests.cs # Three reconciliation checks, deduplication, RabbitMQ publish
├── ObservabilityPhase8Tests.cs # /metrics families and correlation header behavior
├── DataLakePhase9Tests.cs # Kafka → MinIO Parquet flow and schema checks
── ClinicalDemographicsAndObservationTests.cs # Patient/encounter enrichment, expanded observation alerts
── ClinicalDemographicsAndObservationTests.cs # Patient/encounter enrichment, expanded observation alerts
├── WarningAlertTests.cs # WarningEvaluator — warning created, normal/critical skipped, idempotent
├── OrderLifecycleTests.cs # Orders API — create, list, record result, illegal transition 409
└── ValidationTests.cs # FluentValidation — empty fields, threshold ordering, order description
scripts/
├── run-api-redis-tests.sh # Phase 1 — patient/encounter/threshold + Redis cache
@@ -261,10 +274,11 @@ scripts/
├── run-reconciliation-tests.sh # Phase 7 — reconciliation scheduler checks
├── run-phase8-verification.sh # Phase 8 — Prometheus metrics, alerts_unacknowledged_gauge, correlation headers
├── run-phase9-verification.sh # Phase 9 — data lake tests, Kafka offsets, MinIO Parquet, DuckDB schema
── run-phase10-verification.sh # Phase 10 — 12 Redis thresholds, clinical enrichment, ES pipeline, integration tests
── run-phase10-verification.sh # Phase 10 — 12 Redis thresholds, clinical enrichment, ES pipeline, integration tests
└── run-phase11-verification.sh # Phase 11 — warning alerts, orders API, validation, integration tests
docs/
├── plans/ # Phase 110 implementation and verification guides
├── plans/ # Phase 111 implementation and verification guides
├── decisions/
│ ├── data-lake-design.md # Parquet vs JSON, partitioning, replay rationale
│ └── sepsis-engine-design.md # SIRS sliding window and idempotent alert design
@@ -405,6 +419,9 @@ Integration tests use `WebApplicationFactory` with a `Testing` environment and T
| `ObservabilityPhase8Tests` | 8 | All eight `/metrics` families, correlation headers, ingest counter increment |
| `DataLakePhase9Tests` | 9 | Kafka → MinIO Parquet flow and schema checks |
| `ClinicalDemographicsAndObservationTests` | 10 | Patient clinical fields, encounter enrichment, expanded observation codes, critical glucose alert |
| `WarningAlertTests` | 11 | WarningEvaluator — warning HR alert, normal/critical skipped, duplicate idempotent |
| `OrderLifecycleTests` | 11 | Orders API create, list, record result, cancel-resulted 409 |
| `ValidationTests` | 11 | FluentValidation 400 on empty first name, invalid threshold order, empty order description |
### Verification Scripts
@@ -414,6 +431,7 @@ With the API running (`dotnet run`) and Docker Compose up:
./scripts/run-phase8-verification.sh # Prometheus target UP, eight metrics, alerts_unacknowledged_gauge live update
./scripts/run-phase9-verification.sh # DataLakePhase9Tests, Kafka consumer group, MinIO Parquet, DuckDB schema
./scripts/run-phase10-verification.sh # 12 Redis thresholds, clinical enrichment, ES pipeline, Phase 10 integration tests
./scripts/run-phase11-verification.sh # Warning alert pipeline, orders API, FluentValidation, Phase 11 integration tests
```
Per-phase test runners (subset of `dotnet test`):
@@ -656,6 +674,39 @@ open → acknowledged → resolved
| `clinicianId` | string | yes | Clinician identifier |
| `note` | string | no | Optional acknowledgment note |
### Orders
| Method | Path | Description |
|---|---|---|
| POST | `/encounters/{id}/orders` | Create a clinical order for an active encounter |
| GET | `/encounters/{id}/orders` | List orders for an encounter; optional `status`, `page`, `pageSize` |
| GET | `/orders/{id}` | Order detail with encounter |
| PATCH | `/orders/{id}/status` | Transition order status |
| PATCH | `/orders/{id}/result` | Record a result; transitions to `Resulted` |
**Order status machine:**
```
pending → in_progress → resulted
→ cancelled
```
`PATCH /orders/{id}/status` and `PATCH /orders/{id}/result` return **409** on illegal transitions (e.g. cancelling a resulted order).
**POST body:**
| Field | Type | Required | Description |
|---|---|---|---|
| `orderType` | string | yes | `Lab`, `Imaging`, `Medication`, `Procedure` |
| `description` | string | yes | Order description |
| `orderedBy` | string | yes | Ordering clinician |
**PATCH `/orders/{id}/result` body:**
| Field | Type | Required | Description |
|---|---|---|---|
| `resultSummary` | string | no | Free-text result summary |
### Analytics (Elasticsearch)
| Method | Path | Description |
@@ -775,6 +826,7 @@ orderedBy string required (max 200)
status string pending | in_progress | resulted | cancelled (default: pending)
orderedAt DateTimeOffset
resultedAt DateTimeOffset?
resultSummary string? Free-text result summary (set on record result)
```
Indexes: `(encounter_id, ordered_at DESC)`, partial `(status, ordered_at) WHERE status IN ('pending', 'in_progress')`
@@ -879,7 +931,7 @@ Exchange: `clinical.notifications.exchange` (direct)
| Topic | Partition key | Consumer groups |
|---|---|---|
| `observation.recorded` | `encounterId` | `es-indexer`, `sepsis-engine`, `data-lake-writer` |
| `observation.recorded` | `encounterId` | `es-indexer`, `sepsis-engine`, `warning-evaluator`, `data-lake-writer` |
| `alert.generated` | `encounterId` | `es-indexer`, `notification-publisher`, `data-lake-writer` |
| `encounter.status.changed` | `encounterId` | `es-indexer`, `data-lake-writer` |
@@ -979,7 +1031,7 @@ Observation history uses cursor pagination on `(recorded_at DESC, id DESC)`. Off
## Implemented Phases
All ten phases from the project roadmap are implemented and covered by integration tests and/or verification scripts.
Eleven phases from the project roadmap are implemented. Phases 110 and Step 4 of Phase 11 are covered by integration tests (`dotnet test` — 64 passing). Phase 11 Step 5 manual verification (docker compose end-to-end) is documented in `docs/plans/phase-11-plan.md`.
| Phase | Feature | Status |
|---|---|---|
@@ -993,5 +1045,6 @@ All ten phases from the project roadmap are implemented and covered by integrati
| 8 | Prometheus metrics (`GET /metrics`); eight metric families and three collectors; Grafana clinical dashboard; `ObservabilityPhase8Tests`; `run-phase8-verification.sh` | Done |
| 9 | Data lake writer — `data-lake-writer` consumer group; date-partitioned Parquet flush to MinIO; `DataLakePhase9Tests`; `run-phase9-verification.sh`; design doc in `docs/decisions/data-lake-design.md` | Done |
| 10 | Clinical data model expansion — `BloodType`, patient allergies/emergency contact, encounter room/bed/admission/discharge fields; five new observation codes (`SYSTOLIC_BP`, `DIASTOLIC_BP`, `LACTATE_MMOL_L`, `AVPU`, `SUPPLEMENTAL_O2`); `GLUCOSE_MG_DL` threshold fix; 12 seeded thresholds; `ClinicalDemographicsAndObservationTests`; `run-phase10-verification.sh` | Done |
| 11 | Warning alert consumer (`WarningAlertService` / `warning-evaluator`); 10 `Warning*` alert types; Orders API (`OrdersController`, `OrderService`); FluentValidation on all request DTOs; `WarningAlertTests`, `OrderLifecycleTests`, `ValidationTests`; `run-phase11-verification.sh` | Done (Step 5 E2E verification via script) |
**Optional follow-up:** execute and document the Kafka replay demonstration for the data lake (reset `data-lake-writer` offsets, clear MinIO prefixes, restart API, confirm Parquet rebuild). See `docs/plans/phase-9-plan.md` § Replay demonstration.