feature: Prometheus Metrics, Supervisor Dashboard, and Promotion Retry Job
This commit is contained in:
voltsrage
@@ -2,7 +2,7 @@
|
||||
|
||||
A clinical records digitization platform built with ASP.NET Core 8, PostgreSQL, MinIO, Redis, and a Vue 3 workstation UI. The domain models the scan-to-approved lifecycle at the center of any paper chart digitization system: scanned document upload, human data entry, dual-human verification with separation of duties, site-configurable clinical approval routing, atomic promotion to live clinical tables, and governed correction via supersession.
|
||||
|
||||
**Implementation status:** Phases 1–6 are complete (API core through Track B live capture). Phase 7 delivers the digitization workstation UI (`vigilcare-records-web`). Phase 8 is partially implemented — work queue overview endpoint and supervisor dashboard UI are in place; Prometheus metrics and promotion retry remain planned. See [Implemented Phases](#implemented-phases) for the full breakdown.
|
||||
**Implementation status:** Phases 1–8 are complete. Phase 7 delivers the digitization workstation UI (`vigilcare-records-web`). Phase 8 adds Prometheus metrics, supervisor work-queue overview, cursor-paginated batch audit trails, promotion retry with exponential backoff, and a Docker monitoring stack (Prometheus on **9095**, Grafana on **3013**). Phase 9 (E2E verification and clinical scenario docs) is partially implemented. See [Implemented Phases](#implemented-phases) for the full breakdown.
|
||||
|
||||
## Domain Model — How It Maps to a Real Clinical System
|
||||
|
||||
@@ -51,12 +51,13 @@ Append-only audit log entry for every state transition, field-level correction,
|
||||
- **Submit for Verification** — validates completeness per batch type (e.g. vitals sheet requires linked patient, encounter context, and at least one observation with `recordedAt`); transitions `IN_ENTRY → PENDING_VERIFICATION`; returns `422` with missing fields if incomplete
|
||||
- **Verification and Rejection** — verifier reviews entry against the scan with field-level checks (`fieldName`, `status: ok|warning|error`, optional `note`); verify pass transitions to `VERIFIED` or `AWAITING_CLINICAL_APPROVAL` based on site configuration for the batch type; verify fail transitions to `REJECTED` with mandatory reason; **separation of duties** enforced: entry clerk cannot verify their own batch (`409 SEPARATION_OF_DUTIES_VIOLATION`)
|
||||
- **Clinical Approval Routing** — site-configurable per batch type (`SiteConfig.ClinicalApprovalRequired`); high-stakes batch types (encounter summary, vitals sheet, lab results, medication list, mixed) route to `AWAITING_CLINICAL_APPROVAL` after verification for physician sign-off; low-stakes types (patient registration, allergy update) go directly to `VERIFIED`
|
||||
- **Approval and Promotion** — `POST /digitization-batches/:id/approve` atomically promotes draft data to live VigilCareClinical tables (`patients`, `encounters`, `observations`) in a single PostgreSQL transaction; generates MRN via PostgreSQL sequence (`VCR-000001`); patient deduplication by name + DOB; encounter matching by patient + department + active status; writes `observation.created` outbox events for downstream consumers; separation of duties enforced (approver cannot be entry clerk or verifier); `Idempotency-Key` header required for safe retries with 24-hour TTL; retroactive alert policy (`enableRetroactiveAlerts`) controls whether backfill observations emit outbox events
|
||||
- **Approval and Promotion** — `POST /digitization-batches/:id/approve` (on `ApprovalController` only) atomically promotes draft data to live VigilCareClinical tables (`patients`, `encounters`, `observations`) in a single PostgreSQL transaction; generates MRN via PostgreSQL sequence (`VCR-000001`); patient deduplication by name + DOB; encounter matching by patient + department + active status; writes `observation.created` outbox events for downstream consumers; separation of duties enforced (approver cannot be entry clerk or verifier); `Idempotency-Key` header required for safe retries with 24-hour TTL; retroactive alert policy (`enableRetroactiveAlerts`) controls whether backfill observations emit outbox events; on transient infrastructure failure returns **202** with `PROMOTION_DEFERRED` — batch stays `APPROVED` and `PromotionRetryService` retries via `POST /digitization-batches/:id/promote`
|
||||
- **Promotion Result Query** — `GET /digitization-batches/:id/promotion-result` returns live entity IDs (patient, MRN, encounter, observations) created during promotion
|
||||
- **Corrections and Supersession** — approved live observations are never silently edited; a correction uploads a new batch with `supersedesBatchId`, goes through the full entry → verify → approve cycle, and on promotion marks the original batch's `live_observations` rows as `is_superseded` (append-only — never deleted); `422 SUPERSEDED_BATCH_NOT_PROMOTED`, `409 BATCH_ALREADY_SUPERSEDED`, and `404 SUPERSEDED_BATCH_NOT_FOUND` guard invalid supersession targets; correction batches inherit patient context and reuse the original promotion encounter
|
||||
- **Patient Digitization History** — `GET /patients/:id/digitization-history` returns all batches for a patient with correction chain metadata (`isCorrection`, `hasBeenSuperseded`, `supersededByBatchId`), live vs superseded observation counts, summary totals, and per-batch audit trails; `404 PATIENT_HISTORY_NOT_FOUND` when no batches exist for the patient
|
||||
- **Patient Registry Search** — `GET /patients/search?q=` searches live patients by MRN or full name (minimum 2 characters); used by the intake workstation to link uploads to existing patients
|
||||
- **Work Queues** — `GET /work-queue/verification` (batches in `PENDING_VERIFICATION`, sorted by submission time ASC); `GET /work-queue/entry` (batches in `UPLOADED`, `IN_ENTRY`, or `REJECTED`); `GET /work-queue/clinical-approval` (batches in `AWAITING_CLINICAL_APPROVAL`); `GET /work-queue/overview` (aggregate status counts, average queue age, reject rate, oldest pending verification — administrator only); role-restricted access
|
||||
- **Batch Audit Trail API** — `GET /digitization-batches/:id/events` returns cursor-paginated digitization events with actor username and full name; accessible by administrator, verifier, and clinical approver roles
|
||||
- **User Directory** — `GET /users?role=` lists active users for batch assignment (intake clerks assign entry clerks via the workstation UI)
|
||||
- **Digitization Workstation UI** — Vue 3 SPA at `vigilcare-records-web`: role-based routing (intake, entry, verification, supervisor dashboard), split-pane scan viewer with zoom/pan/rotate, draft entry with auto-save, field-level verification checkboxes, presigned URL refresh for long sessions, JWT refresh interceptor
|
||||
- **Live Capture (Track B)** — `POST /live-capture/encounters/{encounterId}/observations` and `POST /live-capture/encounters` for credentialed clinicians entering vitals at bedside; clinician attestation + password re-confirm replaces the dual-human verification gate; observations promote synchronously to live tables in a single transaction with `source = live_capture`; critical threshold evaluation runs before the response returns, with inline `criticalAlert` per observation, committed `ClinicalAlert` rows, and `observation.recorded` / `alert.generated` outbox events; `422 ATTESTATION_REQUIRED`, `422 PASSWORD_CONFIRM_INVALID`, `422 EMPTY_OBSERVATIONS`, `409 ENCOUNTER_NOT_ACTIVE`, `409 ACTIVE_ENCOUNTER_EXISTS`
|
||||
@@ -65,7 +66,7 @@ Append-only audit log entry for every state transition, field-level correction,
|
||||
- **Role-Based Access** — six roles (`INTAKE_CLERK`, `DATA_ENTRY_CLERK`, `VERIFIER`, `CLINICAL_APPROVER`, `CLINICIAN`, `ADMINISTRATOR`) with role-based endpoint authorization; twelve seeded demo users (two per role)
|
||||
- **Auth Audit Events** — append-only `auth_audit_events` table records login, logout, token refresh, and failed login attempts with user ID, IP address, and timestamp
|
||||
- **Standard Envelope** — all responses use `{ success, statusCode, data, error }` wrapper; validation errors use the same shape with stable error codes
|
||||
- **Observability** — Serilog structured logging with Seq sink; correlation IDs via `CorrelationIdMiddleware`; `ExceptionHandlerMiddleware` for consistent error responses
|
||||
- **Observability** — Serilog structured logging with Seq sink; correlation IDs via `CorrelationIdMiddleware`; `ExceptionHandlerMiddleware` for consistent error responses; Prometheus metrics at `GET /metrics` (HTTP request histograms, .NET runtime stats, custom gauges for batch counts by status and queue age, promotion duration histogram, rejection counter by reason category); `MetricsCollectorService` refreshes DB-backed gauges every 30 seconds; Prometheus scrapes the API via `prometheus.yml`; Grafana available at `http://localhost:3013` (dashboard panels configured manually)
|
||||
- **Swagger UI** — OpenAPI spec via Swashbuckle (Development only) at `http://localhost:5217/swagger`
|
||||
|
||||
---
|
||||
@@ -89,6 +90,9 @@ HTTP request
|
||||
├── IdempotencyService (Idempotency-Key storage + replay for safe promotion retries)
|
||||
├── MrnGenerator (PostgreSQL sequence-backed MRN generation: VCR-000001)
|
||||
├── WorkQueueService (verification, entry, clinical approval queues, supervisor overview metrics)
|
||||
├── BatchEventService (cursor-paginated batch audit trail)
|
||||
├── MetricsCollectorService (periodic DB gauge refresh for Prometheus)
|
||||
├── PromotionRetryService (exponential backoff retry for deferred promotions)
|
||||
├── PatientRegistryService (live patient search by MRN or name)
|
||||
├── UserDirectoryService (active user listing for batch assignment)
|
||||
├── AttestationService (clinician role + password re-confirm for live capture)
|
||||
@@ -134,6 +138,7 @@ HTTP request
|
||||
| Authentication | JWT Bearer (Microsoft.AspNetCore.Authentication.JwtBearer) |
|
||||
| Password hashing | BCrypt.Net-Next |
|
||||
| Logging | Serilog + Seq sink |
|
||||
| Metrics | Prometheus (`prometheus-net`) + Grafana |
|
||||
| Docs | Swagger / OpenAPI (Swashbuckle) |
|
||||
| Testing | xUnit + FluentAssertions + WebApplicationFactory |
|
||||
|
||||
@@ -147,9 +152,9 @@ VigilCareRecords/
|
||||
│ ├── Program.cs # Service registration, middleware, seed on startup
|
||||
│ ├── appsettings.json # Connection strings, Redis, MinIO, Seq, JWT, site config
|
||||
│ ├── Controllers/
|
||||
│ │ ├── ApprovalController.cs # Batch approval and promotion to live clinical tables
|
||||
│ │ ├── ApprovalController.cs # Batch approval, promotion deferral (202), promotion to live tables
|
||||
│ │ ├── AuthController.cs # JWT login, token refresh, logout, authenticated user profile
|
||||
│ │ ├── DigitizationBatchesController.cs # Batch CRUD, document upload, assignment
|
||||
│ │ ├── DigitizationBatchesController.cs # Batch CRUD, document upload, assignment, events, promote
|
||||
│ │ ├── DraftController.cs # Draft data entry: patient, encounter, observations, submit
|
||||
│ │ ├── PatientsController.cs # Patient search and digitization history
|
||||
│ │ ├── UsersController.cs # User directory for batch assignment
|
||||
@@ -173,7 +178,9 @@ VigilCareRecords/
|
||||
│ ├── vite.config.ts # Dev server on port 3028; proxies /api → localhost:5217
|
||||
│ └── tailwind.config.js # Clinical color palette and layout component classes
|
||||
├── tests/
|
||||
│ └── VigilCareRecordsAPI.Tests/ # Integration tests (Phases 1–6)
|
||||
│ └── VigilCareRecordsAPI.Tests/ # Integration tests (Phases 1–8)
|
||||
├── prometheus.yml # Prometheus scrape config (API on host.docker.internal:5217)
|
||||
├── docker-compose.yml # PostgreSQL, Redis, Seq, MinIO, Prometheus, Grafana
|
||||
├── scripts/
|
||||
│ ├── run-vigilcare-records-verification.sh # Phase 1
|
||||
│ ├── run-vigilcare-records-phase-2-verification.sh
|
||||
@@ -181,6 +188,7 @@ VigilCareRecords/
|
||||
│ ├── run-vigilcare-records-phase-4-verification.sh
|
||||
│ ├── run-vigilcare-records-phase-5-verification.sh
|
||||
│ ├── run-vigilcare-records-phase-6-verification.sh
|
||||
│ ├── run-vigilcare-records-phase-8-verification.sh # Prometheus metrics, overview, events, retry
|
||||
│ └── run-vigilcare-records-verification-p9.sh # E2E workflow + work queue overview smoke test
|
||||
└── docs/
|
||||
├── plans/ # Phase 1–9 implementation guides
|
||||
@@ -315,6 +323,8 @@ docker compose up -d
|
||||
| Redis 7 | 6383 | No auth |
|
||||
| Seq | 5346 | UI at `http://localhost:5346`, login: `admin` / `seqadmin` |
|
||||
| MinIO | 9012 (S3 API), 9013 (console) | login: `minioadmin` / `minioadmin` |
|
||||
| Prometheus | 9095 | Scrapes API at `host.docker.internal:5217/metrics`; UI at `http://localhost:9095` |
|
||||
| Grafana | 3013 | UI at `http://localhost:3013`; add Prometheus data source `http://prometheus:9090` |
|
||||
|
||||
### Install and Run
|
||||
|
||||
@@ -385,6 +395,7 @@ With the API running (`dotnet run`) and Docker Compose up:
|
||||
./scripts/run-vigilcare-records-phase-4-verification.sh # Phase 4 — approval, promotion, idempotency, patient dedup
|
||||
./scripts/run-vigilcare-records-phase-5-verification.sh # Phase 5 — corrections, supersession, digitization history
|
||||
./scripts/run-vigilcare-records-phase-6-verification.sh # Phase 6 — live capture, attestation, critical alerts
|
||||
./scripts/run-vigilcare-records-phase-8-verification.sh # Phase 8 — Prometheus metrics, work-queue overview, batch events, promotion retry
|
||||
./scripts/run-vigilcare-records-verification-p9.sh # E2E workflow + work queue overview smoke test
|
||||
```
|
||||
|
||||
@@ -1018,7 +1029,7 @@ Response shape:
|
||||
|
||||
## Implemented Phases
|
||||
|
||||
Phases 1–6 are fully implemented and verified via integration tests and per-phase scripts. Phase 7 (workstation UI) and parts of Phase 8 (supervisor overview) are implemented.
|
||||
Phases 1–8 are fully implemented and verified via integration tests and per-phase scripts. Phase 9 (E2E verification and clinical scenario documentation) is partially implemented.
|
||||
|
||||
| Phase | Feature | Status |
|
||||
|---|---|---|
|
||||
@@ -1029,5 +1040,5 @@ Phases 1–6 are fully implemented and verified via integration tests and per-ph
|
||||
| 5 | Correction batches via `supersedesBatchId`, supersession validation on create (`422`/`404`/`409`), append-only `live_observations` supersession flags (`is_superseded`, `superseded_by_batch_id`, `superseded_at`), correction promotion reuses original encounter and linked patient, `correction_uploaded`/`correction_promoted`/`superseded` audit events, `GET /patients/:id/digitization-history` with correction chain and per-batch audit trails, `CorrectionSupersessionTests` integration tests, Phase 5 verification script | Done |
|
||||
| 6 | Track B live capture: `LiveCaptureController` with clinician-only endpoints, `AttestationService` (role + password re-confirm), synchronous promotion via `LiveCaptureService`, Redis-backed critical threshold evaluation, inline critical alerts + committed `ClinicalAlert` rows, `observation.recorded` and `alert.generated` outbox events, open-encounter + vitals outpatient workflow, `LiveCaptureIntegrationTests`, Phase 6 verification script | Done |
|
||||
| 7 | Digitization workstation UI (`vigilcare-records-web`): Vue 3 + Pinia + Tailwind, role-based routing, split-pane scan viewer, draft entry with auto-save, verification checkboxes, patient search, assign-clerk dialog, presigned URL refresh, supervisor dashboard shell | Done |
|
||||
| 8 | `GET /work-queue/overview`, supervisor dashboard UI, patient search API, user directory API | Partial — Prometheus metrics and promotion retry job planned |
|
||||
| 8 | Prometheus metrics (`GET /metrics`), custom digitization gauges/histograms/counters, `MetricsCollectorService`, supervisor `GET /work-queue/overview`, cursor-paginated `GET /digitization-batches/:id/events`, promotion deferral (202 / `PROMOTION_DEFERRED`), `PromotionRetryService` with exponential backoff, Docker Prometheus (9095) + Grafana (3013), Phase 8 verification script | Done |
|
||||
| 9 | E2E verification script (`run-vigilcare-records-verification-p9.sh`), clinical scenario documentation | Partial |
|
||||
|
||||
Reference in New Issue
Block a user