Trent/vigilcare-records

Fork 0

Files

T

voltsrage a4faf7eadd update prd

2026-06-28 02:06:58 +08:00

52 KiB

Raw Blame History

PRD: VigilCare Records — Paper Chart Digitization & Approval Platform

Implementation Status

Phases 1–13 are complete. Post-phase hardening (health checks, user management, auth rate limiting, document access audit, batch cancellation, list sorting, unified promotion retry, normalized patient deduplication, assignment-time IN_ENTRY transitions, API-proxied document streaming, CORS) is also done. See the gap analysis summary matrix for remaining open items.

Phase	Scope	Status
1	Schema, auth, roles, batch CRUD, MinIO upload, status machine	Done
2	Draft entry API, plausibility validation, submit-for-verification	Done
3	Verification, rejection, separation of duties, work queues	Done
4	Promotion to VigilCareClinical live tables, outbox, idempotency	Done
5	Corrections / supersession, patient digitization history	Done
6	Track B live capture with clinician attestation	Done
7	Digitization workstation UI (`vigilcare-records-web`)	Done
8	Prometheus metrics, supervisor overview, batch events API, promotion retry	Done
9	Extended seed data, E2E verification script, clinical scenario docs	Done
10	Barcode/QR cover sheets for high-volume intake	Done
11	HL7 FHIR R4 read-only API and FHIR Explorer UI	Done
12	Backend-driven batch-type field requirements (`fieldRequirements` metadata)	Done
13	Optional OCR-assisted draft pre-fill (Azure or Tesseract; disabled by default)	Done

Verification scripts: ./scripts/run-vigilcare-records-verification-p9.sh (full workflow + metrics), ./scripts/run-vigilcare-records-phase-10-verification.sh, ./scripts/run-vigilcare-records-phase-11-verification.sh, ./scripts/run-vigilcare-records-phase-13-verification.sh.

See README.md for API reference, quick start, and digitization-workstation-guide.md for operator workflows.

Overview

A clinical records intake system that converts paper-based patient charts into structured, human-verified digital records before they enter the VigilCareClinical alerting pipeline. Designed for small hospitals, district clinics, and island health systems where the majority of historical and day-to-day records still exist on paper.

The workflow is deliberately human-governed at every quality gate:

Scan / upload → Human data entry → Human verification → Approved patient record

Every structured field is ultimately verified by a human who reads the scan. A second human compares the entry against the original image before the record becomes clinically authoritative. Unapproved drafts never trigger alerts, scoring, or surveillance.

Optional OCR assist (Phase 13): When Ocr:Enabled is true, a background service may pre-fill draft fields from the scan after upload. The entry clerk still reviews every value against the image, corrects errors, and submits for verification — OCR converts "type everything" into "review and correct" but does not bypass entry or verification gates. Disabled by default.

VigilCareClinical (vigilcare-clinical-api-prd.md) remains the downstream intelligence layer: threshold alerting, NEWS2, sepsis detection, ward dashboard, and long-term archival. VigilCare Records is the precursor that creates the fuel — patient identity, encounters, and observations — in a governed, auditable way.

This project maps to sd-mid-009 (Outbox Pattern for approval promotion events), sd-mid-013 (CQRS — draft vs live read models), sd-junior-004 (RBAC), and senior trade-off conversations around data quality gates vs time-to-value in resource-constrained health systems.

Stack: .NET 8 Web API, PostgreSQL 16, MinIO (scanned document storage), Redis 7 (batch assignment locks and live-capture threshold cache), Serilog → Seq, Prometheus → Grafana, xUnit, Docker Compose. Vue 3 digitization workstation UI in vigilcare-records-web/ (Vite, Pinia, Tailwind CSS). Optional OCR via Azure Document Intelligence or Tesseract (Ocr:Enabled). Read-only FHIR R4 API at /fhir (Firely SDK).

Prerequisite / companion: VigilCareClinicalAPI Phases 1–2 minimum (patient, encounter, observation ingest). Full VigilCare value unlocks as approved observations flow into the existing Kafka alert pipeline.

Goals

Provide a complete scan-to-approved workflow for paper chart conversion with human-verified data quality at every gate (optional OCR pre-fill to accelerate entry, never to replace it)
Enforce separation of duties: the person who enters data cannot verify their own entry
Promote approved structured data into VigilCareClinical's live domain model (Patient, Encounter, Observation) atomically
Maintain a full audit trail linking every approved field to its source scan, entry clerk, verifier, and approver
Support two operational modes: Track A (historical backfill with full dual-human gate) and Track B (live bedside capture with clinician attestation, lighter gate)
Produce a deployable precursor that makes VigilCareClinical credible in paper-only facilities — not as a standalone EMR replacement

Non-Goals

OCR bypassing human review — OCR may pre-fill draft fields (Phase 13) but never skips entry, verification, or approval; fully automated extraction without human review remains out of scope
Full HL7v2 or FHIR write compliance — Phase 11 delivers read-only FHIR R4; HL7v2 ADT, FHIR write, SMART on FHIR, and LIS instrument integration remain out of scope
Full EMR functionality (billing, pharmacy inventory, scheduling beyond basic encounter open)
Replacing VigilCareClinical's alert engine, scoring, or ward dashboard
HIPAA-certified or jurisdiction-specific medical device registration (model the patterns; certification scoped per deployment)
Multi-facility federated identity across islands (single-tenant deployment per site in v1)

Problem Statement

VigilCareClinical assumes structured, timestamped clinical data already exists. In paper-based facilities:

Patient identity lives in folders, ward books, or duplicate index cards — no stable MRN workflow
Vitals and lab results are handwritten — illegible, untimestamped at minute precision, or lost between visits
There is no encounter boundary — "the patient" is not the same as "this admission" or "this clinic visit"
Clinicians cannot trust machine-generated alerts if the underlying values were guessed from poor handwriting

Without a digitization and approval layer, VigilCare has nothing to observe. With it, even a 20-bed district hospital can convert charts incrementally and activate real-time alerting as live capture replaces paper forms.

Relationship to VigilCareClinical

┌─────────────────────────────────────────────────────────────────────┐
│  VigilCare Records (this PRD)                                     │
│                                                                     │
│  Scan → Entry → Verify → Approve                                    │
│       ↓                                                             │
│  Draft tables (never alert)                                         │
│       ↓ on approval                                                 │
│  Promotion service ──────────────────────────────────────────────┐  │
└──────────────────────────────────────────────────────────────────│──┘
                                                                   │
┌──────────────────────────────────────────────────────────────────▼──┐
│  VigilCareClinicalAPI                                             │
│                                                                     │
│  Patient → Encounter → Observation → Outbox → Kafka → Alerts      │
└─────────────────────────────────────────────────────────────────────┘

Invariant: Draft observations are invisible to VigilCareClinical's alert engine, Elasticsearch ward projection, and scoring consumers. Only batches that have passed approval are promoted to live tables. Promotion is idempotent — re-running an approval job for the same batch produces no duplicate live rows.

Track A (backfill): Full pipeline — scan, entry, verification, approval. Historical vitals and labs enter as observations with recordedAt taken from the chart (not scan time). Alerts on backfilled critical values are suppressed by default unless the facility explicitly opts in per batch (enableRetroactiveAlerts: false default).

Track B (live capture): Credentialed clinician enters vitals at bedside. Skips verification queue; requires clinicianAttestation: true on submit. Observations promote immediately to VigilCareClinical with source: live_capture. Every live-capture submission still creates a DigitizationBatch (audit unit) with draft observations, DigitizationEvent entries, and live rows in a single transaction — see Track B audit model.

Deployment model (v1): Integrated PostgreSQL — Records draft tables and VigilCareClinical live tables (patients, encounters, observations, outbox_events) share one database instance (separate schemas). Promotion runs in a single local transaction. Split deployment with HTTP + saga retry is documented as a future deployment option, not the portfolio default.

API Conventions

Same response envelope as VigilCareClinical and other portfolio projects. Prefix: /api/v1.

Success:

{
  "success": true,
  "statusCode": 200,
  "data": {},
  "error": null
}

Error:

{
  "success": false,
  "statusCode": 422,
  "data": null,
  "error": {
    "message": "Verifier cannot approve a batch they entered.",
    "code": "SEPARATION_OF_DUTIES_VIOLATION"
  }
}

Pagination: Work queues and batch lists use offset pagination (?page=1&pageSize=20). Audit event history uses cursor pagination on (occurred_at DESC, id DESC).

Idempotency: POST /api/v1/digitization-batches/:id/approve accepts an Idempotency-Key header. Duplicate approval requests return the same promotion result without creating duplicate live observations in VigilCareClinical.

Roles and Permissions

Role	Permissions
Intake clerk	Upload scans, create batches, assign patient (existing MRN or new registration draft)
Data entry clerk	Edit draft fields on batches in `uploaded` or `rejected` state; submit for verification
Verifier	Review batches in `pending_verification`; field-level pass/fail; reject with reason; cannot verify batches they entered
Clinical approver	Final promotion trigger via `POST .../approve` for batches in `verified` or `awaiting_clinical_approval`; cannot approve batches they entered
Clinician	Track B live capture; attestation on own entries (attestation satisfies verify + approve for that batch only)
Administrator	User management, batch type config, retroactive alert policy, work-queue reassignment

Verification vs approval (Track A)

Track A uses two distinct human gates before promotion:

Verifier — compares structured draft fields against the scan (POST .../verify or POST .../reject). This is the dual-human data-quality check.
Clinical approver — authorizes promotion (POST .../approve), which creates live clinical records. Every Track A batch requires this step regardless of batch type.

Optional third gate: For high-stakes batchType values, site configuration routes verify-pass batches to awaiting_clinical_approval instead of verified. The clinical approver then reviews after the verifier before promotion. See Site configuration below.

After verifier pass	Next status	Who calls `approve`
Site config: clinical sign-off not required for this `batchType`	`verified`	Clinical approver (or administrator)
Site config: clinical sign-off required for this `batchType`	`awaiting_clinical_approval`	Clinical approver (or administrator)

Separation of duties is enforced at the service layer, not only in the UI. enteredByUserId === currentUserId blocks verify and approve actions with 409 SEPARATION_OF_DUTIES_VIOLATION. Verifiers never call approve; clinical approvers never call verify.

Site configuration

Per-site ClinicalApprovalRequired maps batchType to whether verify-pass routes to awaiting_clinical_approval (physician queue) vs verified (ready for immediate approver sign-off). Default portfolio configuration:

batchType	Clinical sign-off after verify?
`patient_registration`	No
`allergy_update`	No
`encounter_summary`	Yes
`vitals_sheet`	Yes
`lab_results`	Yes
`medication_list`	Yes
`mixed`	Yes

Administrators may override this map per deployment in appsettings.json (SiteConfig.ClinicalApprovalRequired).

Domain Model

DigitizationBatch

The unit of work for one digitization effort — typically one scanned document or one logical chart section (vitals sheet, lab report, admission face sheet).

Field	Description
`id`	UUID
`status`	See state machine below
`batchType`	`patient_registration`, `encounter_summary`, `vitals_sheet`, `lab_results`, `medication_list`, `allergy_update`, `mixed`
`patientId`	Nullable until linked; may reference draft or live patient
`encounterDraftId`	Nullable; encounter context for vitals/labs
`documentRef`	MinIO object key for the scanned PDF/image
`documentSha256`	Content hash for integrity verification
`track`	`backfill` (Track A) or `live_capture` (Track B)
`enableRetroactiveAlerts`	Default `false`; if `true` on approval, promoted observations participate in alerting
`enteredByUserId`	Set on first draft save
`verifiedByUserId`	Set on verification pass
`approvedByUserId`	Set on final approval
`rejectionReason`	Nullable; required when status → `rejected`
`promotedAt`	Nullable; timestamp when live records created
`promotionEncounterId`	VigilCareClinical encounter ID after promotion
`supersedesBatchId`	Nullable; links a correction batch to the batch it replaces
`clinicianAttestation`	`true` for Track B batches attested at bedside

DraftPatient / DraftPatientUpdate

Structured patient fields extracted from paper — demographics, allergies, blood type, emergency contact, medications (for medication_list batches). On approval of a patient_registration or allergy_update batch, merges into VigilCareClinical Patient (create or patch). medicationsJson is stored on the digitization audit record in v1; medication rows are not promoted to a pharmacy module (out of scope).

DraftEncounter

A clinical episode extracted from the chart: admission date, department, room/bed, admission reason, discharge diagnosis (if applicable). Promotes to VigilCareClinical Encounter.

DraftObservation

A single measurable value: observation code, numeric value, unit, recordedAt (from chart, required), optional note. Subject to the same plausibility ranges as VigilCareClinical ingest. Never written to live observations until batch approval.

DigitizationEvent

Append-only audit log entry for every state transition and field-level correction.

{
  "id": "uuid",
  "batchId": "uuid",
  "eventType": "uploaded | entry_started | submitted_for_verification | rejected | verified | awaiting_clinical_approval | approved | promoted | live_capture_attested | correction_requested",
  "actorUserId": "uuid",
  "occurredAt": "2026-06-22T14:30:00Z",
  "metadata": {
    "rejectionReason": "SpO2 value unclear on scan — decimal ambiguous",
    "fieldsChanged": ["observations[2].value"]
  }
}

ScannedDocument

Stored in MinIO. Original paper is the legal source; the scan is the working reference for entry and verification. Retention: minimum 7 years (configurable per jurisdiction). Scanned documents are never deleted when a batch is rejected — only the draft is returned for correction.

Batch Status State Machine

                    ┌──────────────┐
                    │   uploaded   │
                    └──────┬───────┘
                           │ assign / first save
                           ▼
                    ┌──────────────┐
         ┌──────────│  in_entry    │◄─────────┐
         │          └──────┬───────┘          │
         │                 │ submit           │ reject
         │                 ▼                  │
         │          ┌──────────────┐         │
         │          │   pending    │─────────┘
         │          │ verification │
         │          └──────┬───────┘
         │                 │
         │    verify fail  │ verify pass
         │        ─────────┤
         │                 │
         │   site config   │ site config
         │   = false       │ = true
         │        ┌────────┴────────┐
         │        ▼                 ▼
         │  ┌──────────────┐  ┌──────────────────────┐
         │  │   verified   │  │ awaiting_clinical    │
         │  └──────┬───────┘  │     _approval          │
         │         │          └──────────┬─────────────┘
         │         └──────────┬──────────┘
         │                    │ clinical approver: approve
         │                    ▼
         │             ┌──────────────┐
         └────────────►│   approved   │
                       └──────┬───────┘
                              │ promotion (sync or retry job)
                              ▼
                       ┌──────────────┐
                       │  promoted    │  (terminal — live records exist)
                       └──────────────┘

Track B shortcut: live_capture creates a batch already in promoted — see Track B audit model.

Allowed transitions (Track A):

From	To
`uploaded`	`in_entry`
`in_entry`	`pending_verification`
`pending_verification`	`verified`, `awaiting_clinical_approval`, `rejected`
`rejected`	`in_entry`
`verified`	`approved`
`awaiting_clinical_approval`	`approved`, `rejected`
`approved`	`promoted`
`promoted`	(none — terminal)

Illegal transitions return 409 with a stable error code. A promoted batch cannot return to any earlier state. Corrections require a new batch referencing supersedesBatchId.

Track B audit model

Track B is not a bypass of the batch model — it is a shortcut through the workflow states, not the audit unit. Every live-capture submission:

Creates a DigitizationBatch with track: live_capture, batchType: vitals_sheet (or appropriate type), documentRef: "live-capture", and a synthetic documentSha256 derived from clinician + encounter + timestamp (no scan file).
Writes DraftObservation rows for every entered value (audit trail).
Sets enteredByUserId, verifiedByUserId, and approvedByUserId to the attesting clinician (attestation replaces the dual-human gate for that batch only).
Inserts live Observation rows and outbox events in the same transaction, leaving the batch in promoted immediately.
Emits DigitizationEvent entries: live_capture_attested, then promoted.

This keeps digitization history, patient coverage stats, and Prometheus batch metrics consistent across both tracks.

Features

1. Document Upload and Batch Creation — implemented (Phase 1)

Description: Intake clerk scans or uploads a paper record. System stores the file in MinIO, computes SHA-256, creates a batch in uploaded state, and writes a DigitizationEvent.

Endpoints:

POST /api/v1/digitization-batches — multipart upload: file (PDF, JPEG, PNG; max 25 MB), batchType, optional patientId, optional track (default backfill), optional coverSheetCode (Phase 10 — auto-populates type/track/patient from barcode cover sheet)
GET /api/v1/digitization-batches/:id — batch detail with document presigned URL (15-minute expiry)
GET /api/v1/digitization-batches — filter by status, batchType, assignedTo, track; paginated
PATCH /api/v1/digitization-batches/:id/assign — assign to entry clerk (Redis lock prevents double-assignment; transitions uploaded → in_entry immediately)

Validation:

Accepted MIME types: application/pdf, image/jpeg, image/png
Reject empty files
Duplicate detection (when patientId is set): reject if documentSha256 matches an existing batch for the same patient within 24 hours (409 DUPLICATE_DOCUMENT) — prevents accidental double-scan of the same chart page
Cross-patient duplicates: the same physical form scanned for two different patients is allowed (distinct patients, distinct batches). Operators may still see a warning in the UI if the SHA matches any batch site-wide (informational only in v1)

Concepts practiced: Object storage for immutable document artifacts, content-addressed deduplication, presigned URLs for secure document viewing without proxying binary through the API.

2. Draft Data Entry — implemented (Phase 2)

Description: Entry clerk views the scan alongside structured form fields. Saves draft patient updates, encounter context, and observations. Batch moves to in_entry on assignment or first save. Form sections (allergies, medications, encounter summary, observations) are driven by backend fieldRequirements metadata (Phase 12). When OCR is enabled, pre-filled fields include confidence indicators (Phase 13).

Endpoints:

GET /api/v1/digitization-batches/:id/draft — full draft payload: patient fields, encounter, observations[]
PUT /api/v1/digitization-batches/:id/draft/patient — upsert draft patient demographics or updates
PUT /api/v1/digitization-batches/:id/draft/encounter — upsert draft encounter fields
POST /api/v1/digitization-batches/:id/draft/observations — add observation row
PUT /api/v1/digitization-batches/:id/draft/observations/:obsId — edit observation
DELETE /api/v1/digitization-batches/:id/draft/observations/:obsId — remove observation from draft
POST /api/v1/digitization-batches/:id/submit-for-verification — validates completeness, transitions to pending_verification

Required fields before submit (by batch type):

batchType	Required draft content
`patient_registration`	Full name, date of birth, sex; MRN generated on approval if new
`encounter_summary`	Linked patient; encounter with admission date, department, admission reason
`vitals_sheet`	Linked patient, encounter context, ≥1 observation with `recordedAt`
`lab_results`	Linked patient, encounter, ≥1 lab observation code, `recordedAt`
`medication_list`	Linked patient; `medicationsJson` with ≥1 entry or explicit `noActiveMedications: true`
`allergy_update`	Linked patient, allergies list (may be empty with explicit `noKnownAllergies: true`)
`mixed`	Linked patient, encounter context, and at least one of: ≥1 observation with `recordedAt`, or complete encounter summary (admission date + department + admission reason)

Concurrent draft editing: Redis assignment lock prevents double-assignment at intake (PATCH .../assign). Draft saves require the acting user to match enteredByUserId (or hold the Administrator role); otherwise 409 BATCH_NOT_ASSIGNED. v1 does not use field-level optimistic locking — within an assigned session, last write wins. Administrators may reassign via work-queue tools, which clears the Redis lock and updates enteredByUserId.

Plausibility validation: Reuse VigilCareClinical observation plausibility ranges at draft save time. Out-of-range values return 422 OBSERVATION_OUT_OF_PLAUSIBLE_RANGE with the same error shape — catch decimal errors (5.2 vs 52) before verification.

UI requirement (digitization workstation): Side-by-side layout — scan viewer (zoom, pan, rotate) on the left; structured entry form on the right. Field-level "verified" checkbox for verifier pass (stored in draft metadata, not live record).

Concepts practiced: Draft vs live data separation, optimistic UI with server validation, batch-type-driven validation rules.

3. Verification and Rejection — implemented (Phase 3)

Description: Verifier reviews entry against the scan. Can approve field-by-field or reject the entire batch with a mandatory reason. Verifier cannot be the entry clerk.

On verify pass: status → verified or awaiting_clinical_approval per site configuration. On verify fail: status → rejected (returns to entry queue).

Endpoints:

GET /api/v1/work-queue/verification — batches in pending_verification, sorted by submittedAt ASC
GET /api/v1/work-queue/clinical-approval — batches in awaiting_clinical_approval, sorted by submittedAt ASC
POST /api/v1/digitization-batches/:id/verify — body: { "fieldChecks": [{ "fieldName": "observations[0].value", "status": "ok", "note": null }], "passed": true }
POST /api/v1/digitization-batches/:id/reject — body: { "reason": "..." } → status rejected, notifies entry clerk

Reject is allowed from pending_verification (verifier; separation of duties applies) or awaiting_clinical_approval (clinical approver; separation of duties does not apply — approver may not have been the entry clerk by role design).

Concepts practiced: Separation of duties enforcement, work-queue patterns, structured rejection loops.

4. Approval and Promotion to VigilCareClinical — implemented (Phase 4)

Description: Final approval triggers an atomic promotion: draft records become live Patient / Encounter / Observation rows in VigilCareClinical (same database in integrated deployment, or HTTP calls to VigilCareClinical API in split deployment). Batch status → promoted.

Endpoints:

POST /api/v1/digitization-batches/:id/approve — requires verified or awaiting_clinical_approval; restricted to ClinicalApprover or Administrator; Idempotency-Key required; returns 202 with PROMOTION_DEFERRED on transient failure (batch stays approved)
POST /api/v1/digitization-batches/:id/promote — manual promotion of an approved batch (also used after deferral)
GET /api/v1/digitization-batches/:id/promotion-result — live IDs created: patientId, encounterId, observationIds[]

Promotion transaction sequence:

Begin database transaction (or saga with compensating actions in split deployment)
Create or update Patient in VigilCareClinical (assign MRN if new)
Create or match Encounter (open as active or discharged based on draft)
Insert each DraftObservation as live Observation with source: digitization_backfill or source: live_capture
Write outbox events for each observation (Kafka pipeline activates only if enableRetroactiveAlerts or track: live_capture)
Update batch status → promoted, set promotedAt, write DigitizationEvent
Commit

Default alert behavior:

track	enableRetroactiveAlerts	Alert pipeline
`backfill`	`false` (default)	Observations stored; no alert evaluation
`backfill`	`true`	Full VigilCareClinical alert path
`live_capture`	n/a	Full alert path immediately

Concepts practiced: Outbox pattern for promotion side effects, idempotent promotion, configurable clinical safety policy for historical data.

5. Corrections and Supersession — implemented (Phase 5)

Description: Approved records are not silently edited. A correction creates a new batch with supersedesBatchId pointing to the original. Correction goes through the full entry → verify → approve cycle. On promotion, erroneous live observations are marked superseded (append-only — not deleted).

Endpoints:

POST /api/v1/digitization-batches — body includes optional supersedesBatchId
GET /api/v1/patients/:id/digitization-history — all batches for a patient with promotion status

Concepts practiced: Immutable clinical audit trail, correction-as-new-batch pattern (same as national digital services land registry approach).

6. Live Capture (Track B) — implemented (Phase 6)

Description: Credentialed clinician enters vitals or labs at point of care on a tablet. No verification queue. Each submission creates an audit DigitizationBatch (see Track B audit model), writes draft observations for traceability, and promotes live observations synchronously with full alert evaluation.

Endpoints:

POST /api/v1/live-capture/encounters/:encounterId/observations — body: observation fields + clinicianAttestation: true + password re-confirm or PIN
POST /api/v1/live-capture/encounters — open encounter + initial vitals in one request (outpatient workflow)

Validation: Requires Role: Clinician. Creates batch + draft observations + live observations in one transaction; batch lands in promoted immediately. Returns VigilCareClinical observation IDs and any synchronous critical alerts generated.

Concepts practiced: Lighter gate for real-time care vs heavy gate for backfill; same underlying observation schema.

7. Patient Registry (Draft and Live) — partially implemented

Description: Search and link batches to patients. New patient registration flows through the draft pipeline on a patient_registration batch; there is no standalone draft-patient API in v1.

Implemented endpoints:

GET /api/v1/patients/search?q= — search live patients by MRN or name (min 2 characters)
GET /api/v1/patients/:id/digitization-history — all batches for a patient with correction chain and audit trails

Deferred (not in v1):

POST /api/v1/patients/draft — create draft-only patient without a batch
GET /api/v1/patients/:id/summary — live patient + pending draft batches + digitization coverage stats

Digitization coverage stat (future): approvedBatchCount / estimatedTotalBatches — optional manual estimatedChartSections per patient for progress tracking.

8. Work Queues and Operational Dashboard — implemented (Phase 8)

Description: Supervisors monitor backlog, assignment, and throughput.

Endpoints:

GET /api/v1/work-queue/entry — batches awaiting or in entry (UPLOADED, IN_ENTRY, REJECTED)
GET /api/v1/work-queue/verification — batches in PENDING_VERIFICATION, sorted by submission time
GET /api/v1/work-queue/clinical-approval — batches in AWAITING_CLINICAL_APPROVAL
GET /api/v1/work-queue/overview — counts by status, average time-in-queue, 24h reject rate, oldest pending verification (administrator only)
GET /api/v1/digitization-batches/:id/events — cursor-paginated audit trail with actor username and full name

Promotion retry (Phase 8): When POST .../approve fails due to transient infrastructure errors, the batch stays APPROVED and returns 202 with PROMOTION_DEFERRED. PromotionRetryService retries with exponential backoff; operators may also call POST .../promote manually.

Metrics (Prometheus) — implemented at GET /metrics:

digitization_batches_by_status (gauge)
digitization_promotion_duration_seconds (histogram)
digitization_rejection_total (counter)
digitization_queue_age_seconds (gauge — oldest pending verification)

Docker Compose exposes Prometheus on port 9095 and Grafana on 3013.

9. Authentication and Audit — implemented (Phases 1, 8, post-phase hardening)

Description: JWT auth with role claims. Every state transition writes a DigitizationEvent. Document access logged. Auth endpoints rate-limited (10 requests per 5 minutes). Administrators manage users via CRUD endpoints.

Endpoints:

POST /api/v1/auth/login — returns access token (15 min), refresh token (7 days), and user profile
POST /api/v1/auth/refresh — rotates refresh token and issues new access token
POST /api/v1/auth/logout — revokes refresh token server-side
GET /api/v1/auth/me
POST /api/v1/users — create user (administrator)
PATCH /api/v1/users/:id — update user (administrator)
POST /api/v1/users/:id/reset-password — reset password (administrator)
POST /api/v1/users/me/change-password — self-service password change

Audit requirements:

Auth events (USER_LOGOUT, TOKEN_REFRESHED) persisted in auth_audit_events
Who viewed a scan and when (document_accessed events on batch detail and document download)
Who changed which draft field (field-level diff in event metadata on save)
Who approved promotion and which live record IDs were created

Health probes (post-phase hardening): GET /health/live, GET /health/ready (PostgreSQL, Redis, MinIO), GET /health/startup.

10. Cover Sheet System — implemented (Phase 10)

Description: Printable barcode/QR cover sheets encode batch type, track, optional patient, and optional entry-clerk pre-assignment. Intake clerks attach cover sheets to paper chart sections before bulk scanning; the workstation reads the code and auto-creates batches without manual classification.

Endpoints:

POST /api/v1/cover-sheets/generate — create 1–100 cover sheets with unique VCR-CS-{hex} codes
GET /api/v1/cover-sheets/lookup/{code} — resolve a cover sheet for intake auto-fill
GET /api/v1/cover-sheets — list cover sheets with isUsed and patientId filters
POST /api/v1/cover-sheets/{id}/pdf — printable single cover sheet PDF with QR code
POST /api/v1/cover-sheets/batch-pdf — multi-page PDF for batch printing

Cover sheets are single-use; redeemed on batch creation (409 COVER_SHEET_ALREADY_USED on reuse).

11. HL7 FHIR R4 Read API — implemented (Phase 11)

Description: Read-only FHIR R4 endpoints expose promoted clinical data (Patient, Encounter, Observation) for external EHR and interoperability consumers. Maps VigilCare observation codes to LOINC; search bundles include pagination links.

Endpoints:

GET /fhir/metadata — anonymous CapabilityStatement
GET /fhir/Patient/{id}, GET /fhir/Patient?name=, GET /fhir/Patient?identifier= (MRN)
GET /fhir/Encounter/{id}, GET /fhir/Encounter?patient=
GET /fhir/Observation/{id}, GET /fhir/Observation?patient=&code=&category=&date=
GET /fhir/Patient/{id}/$everything — composite bundle

SMART on FHIR authorization and FHIR write operations remain out of scope.

12. Batch-Type Field Requirements — implemented (Phase 12)

Description: fieldRequirements metadata on GET /digitization-batches/:id and GET /digitization-batches/:id/draft tells the workstation which form sections to render per batch type (patient demographics, encounter context, encounter summary fields, observations, allergies, medications). Entry and verification forms consume this metadata instead of hardcoding batch-type rules. Backend ValidateCompleteness() remains the enforcement layer.

13. Optional OCR-Assisted Pre-Fill — implemented (Phase 13; disabled by default)

Description: When Ocr:Enabled is true, OcrProcessingService polls uploaded batches, extracts text via Azure Document Intelligence or self-hosted Tesseract, and pre-fills draft fields with per-field confidence scores. Entry clerks review and correct; verification and approval workflows are unchanged.

Configuration: Ocr:Enabled (default false), Ocr:Provider (azure or tesseract), Ocr:ConfidenceThreshold, Ocr:PollIntervalSeconds.

Privacy: Cloud provider requires a signed BAA; local Tesseract keeps PHI on-network.

Draft response: ocrConfidence map on draft payload; UI shows confidence borders and an "OCR Pre-filled" banner when OCR was used.

Digitization Workstation UI — implemented (Phases 7, 10, 11, 12, 13)

Vue 3 SPA at vigilcare-records-web/ (dev server port 3028, proxies /api and /fhir → API on 5217). Role-based routing with JWT refresh.

View	User	Purpose
Intake	Intake clerk	Upload, barcode cover sheet lookup, assign patient, assign entry clerk
Cover sheets	Intake clerk, administrator	Generate, list, and print QR cover sheets
Entry	Data entry clerk	Side-by-side scan + form with auto-save, backend-driven field visibility, optional OCR confidence indicators
Verification	Verifier	Side-by-side with field checkboxes, approve/reject
Clinical approval	Clinical approver	Scan review, approve/reject, retroactive alert toggle
Live capture	Clinician	Bedside vitals with attestation and password confirm
Patient history	All roles	Digitization timeline with correction chain and audit trail
Queue dashboard	Administrator	Backlog metrics from work-queue overview
FHIR Explorer	Administrator	Browse and search FHIR resources, inspect JSON, load Patient `$everything`

Scan viewer loads documents via authenticated GET /digitization-batches/:id/document blob URLs (avoids cross-origin MinIO iframe issues). Not a full EMR UI — clinical alerting views remain in VigilCareClinical's ward dashboard.

See digitization-workstation-guide.md for operator workflows and clinical scenarios.

Data Storage

Store	Purpose
PostgreSQL	Draft tables, batch metadata, digitization events, user/role data. Shares database with VigilCareClinical in integrated deployment.
MinIO	Scanned PDFs and images; content-addressed keys `scans/{year}/{month}/{batchId}/{sha256}.pdf`
Redis	Batch assignment locks (`SET batch:assign:{id} NX EX 3600`). Work-queue counters are derived from PostgreSQL queries in v1 (no Redis counter cache required)

Integration Contract with VigilCareClinical

v1 deployment: integrated PostgreSQL — promotion writes directly to VigilCareClinical tables in a single transaction. Split deployment (Records service calling VigilCareClinical REST + PromotionRetryService) remains supported as an alternate topology; see Phase 8.

On promotion, VigilCare Records writes to the same tables VigilCareClinical owns:

Draft entity	Live entity	Notes
`DraftPatient`	`patients`	MRN generated via existing `GenerateMrnAsync` logic
`DraftEncounter`	`encounters`	Status from draft; `roomBed`, `admissionReason` mapped
`DraftObservation`	`observations`	Same `observationCode`, `value`, `unit`, `recordedAt`; adds `metadata.source`

Observation codes must match VigilCareClinical's catalog: HEART_RATE, TEMP_C, BP_SYSTOLIC, BP_DIASTOLIC, RESP_RATE, SPO2, POTASSIUM_MEQ_L, GLUCOSE_MG_DL, WBC_K_UL, LACTATE_MMOL_L, etc.

If VigilCareClinical is unreachable in split deployment, batch remains approved and a background promotion retry job runs with exponential backoff. Batch does not revert to draft.

Acceptance Criteria

Criterion	Verification	Status
Separation of duties	Entry clerk cannot verify or approve own batch — `409`	Done
Draft isolation	Draft observations never appear in VigilCareClinical alert queries or ward dashboard	Done
Promotion atomicity	Partial promotion (patient created, observations failed) never committed	Done
Idempotent approval	Duplicate `Idempotency-Key` on approve returns same result, no duplicate observations	Done
Rejection loop	Rejected batch returns to entry; resubmit reaches verification again	Done
Backfill alert suppression	Default backfill promotion creates observations with zero alerts	Done
Live capture alert path	Track B critical potassium triggers synchronous alert in VigilCareClinical	Done
Audit completeness	Every status transition has a `DigitizationEvent` with actor and timestamp	Done
Document immutability	Scan object in MinIO not modified or deleted on reject/correct	Done
Plausibility at draft	Value 520 for potassium rejected at draft save, not at promotion	Done
Clinical approval routing	`vitals_sheet` verify-pass → `awaiting_clinical_approval` when site config requires it	Done
Track B audit batch	Live capture creates `DigitizationBatch` in `promoted` with draft observations and events	Done
Draft assignment guard	Unassigned clerk receives `409 BATCH_NOT_ASSIGNED` on draft save	Done
Cross-patient duplicate scan	Same SHA for two different patients allowed; same patient within 24h rejected	Done
Supervisor metrics	Work-queue overview and Prometheus gauges reflect live batch counts	Done
Promotion retry	Transient promotion failure defers to `APPROVED` with automatic retry	Done
E2E verification script	`./scripts/run-vigilcare-records-verification-p9.sh` passes against running API	Done
Extended demo seed data	Startup seed includes patients/batches across all statuses for dashboard demos	Done
Cover sheet intake	Barcode cover sheet auto-creates batch with correct type/track/patient	Done
FHIR read API	`GET /fhir/Patient`, `/Encounter`, `/Observation` return valid FHIR R4 JSON	Done
Field requirements metadata	Entry/verification forms render sections from `fieldRequirements` on batch/draft responses	Done
OCR pre-fill (when enabled)	Uploaded batch gets draft pre-fill with confidence map; disabled by default	Done
Health checks	`/health/ready` reports PostgreSQL, Redis, MinIO status	Done
Batch cancellation	Administrator can cancel batches in `uploaded`, `in_entry`, or `rejected`	Done

Build Order

Phase	Focus	Status
1	Schema, auth, roles, batch CRUD, MinIO upload, status machine	Done
2	Draft entry API (patient, encounter, observations), submit-for-verification	Done
3	Verification, rejection, separation of duties, work queues	Done
4	Promotion service → VigilCareClinical live tables, outbox integration, idempotency	Done
5	Corrections / supersession, patient digitization history	Done
6	Track B live capture with clinician attestation	Done
7	Digitization workstation UI (entry + verification side-by-side)	Done
8	Prometheus metrics, supervisor dashboard, promotion retry job	Done
9	Extended seed data, E2E verification script, clinical scenario documentation	Done
10	Barcode/QR cover sheet system for high-volume intake	Done
11	HL7 FHIR R4 read-only API and FHIR Explorer UI	Done
12	Backend-driven batch-type field requirements	Done
13	Optional OCR-assisted draft pre-fill	Done

Step-by-Step Guide

Complete phases in order. Promotion (Phase 4) must not be built until the draft state machine and separation of duties are correct — debugging promotion bugs alongside workflow bugs is painful.

Phase 1 — Schema, Upload, and Status Machine

What to do:

Create digitization_batches, draft_patients, draft_encounters, draft_observations, digitization_events tables.
Implement batch status machine with explicit transition matrix; illegal transitions → 409.
Wire MinIO upload with SHA-256 computation and presigned GET URLs.
Seed two users per role (entry, verifier, clinician) for separation-of-duties testing.
Implement JWT auth with role claims.

Why: The status machine is the backbone. Getting transitions wrong means drafts leak into live data or approved batches get re-edited. Test every illegal transition before building entry forms.

Phase 2 — Draft Entry

What to do:

Implement draft CRUD endpoints and batch-type validation on submit.
Port plausibility validator from VigilCareClinical (shared library or duplicated with comment linking source).
Write integration tests: incomplete vitals batch cannot submit; plausible observations save; implausible rejected.

Why: Plausibility at draft save prevents the most common digitization error — decimal misplacement — from ever reaching verification.

Phase 3 — Verification and Rejection

What to do:

Implement verification and rejection endpoints with separation-of-duties checks.
Build work-queue endpoints sorted by submittedAt.
Test: entry clerk A submits → verifier A attempts verify → 409; verifier B succeeds.

Why: Separation of duties is a clinical trust requirement, not a nice-to-have. Enforce in the service layer from day one.

Phase 4 — Promotion

What to do:

Implement approval endpoint and promotion transaction against VigilCareClinical tables.
Wire outbox events for observations where alerting is enabled.
Implement enableRetroactiveAlerts flag — default false.
Test full path: upload → entry → verify → approve → observations in live table → zero alerts for backfill default.
Test idempotency: approve twice with same key → one set of live rows.

Why: This phase connects Records to Clinical. Run against a VigilCareClinical Phase 2 environment where synchronous critical alerting exists, and verify Track B vs Track A behavior explicitly.

Phase 5 — Corrections

What to do:

Add supersedesBatchId and supersession logic on promotion.
Mark superseded live observations inactive (soft flag, not delete).
Test: wrong potassium promoted → correction batch → new value live, old value superseded in audit.

Phase 6 — Live Capture (Track B)

What to do:

Implement clinician attestation endpoint bypassing verification.
Test critical value entered via live capture → alert fires before response returns.

Phases 7–9 — UI, Observability, Documentation

Phase 7 (done): Vue 3 workstation UI with intake, entry, verification, and supervisor dashboard views. Role-based routing, split-pane scan viewer, presigned URL refresh.

Phase 8 (done): Prometheus metrics, GET /work-queue/overview, cursor-paginated GET /digitization-batches/:id/events, promotion deferral (202) and PromotionRetryService, Docker Prometheus/Grafana stack. Verification: ./scripts/run-vigilcare-records-phase-8-verification.sh.

Phase 9 (done): Extended DataSeeder.cs with 10 demo batches across all statuses/types/tracks, docs/digitization-workstation-guide.md, ./scripts/run-vigilcare-records-verification-p9.sh, project README.

Phase 10 — Cover Sheet System

What to do:

Add CoverSheet entity and cover sheet CRUD/generate/lookup endpoints.
Wire coverSheetCode on batch upload to auto-populate type, track, patient, and redeem the sheet.
Generate printable PDFs with QR codes; add /cover-sheets and barcode-assisted intake in the Vue UI.

Verification: ./scripts/run-vigilcare-records-phase-10-verification.sh

Phase 11 — FHIR R4 Read API

What to do:

Install Firely SDK; map promoted clinical data to FHIR Patient, Encounter, Observation resources.
Implement read and search endpoints with LOINC mapping and pagination bundles.
Add administrator FHIR Explorer view at /fhir-explorer.

Verification: ./scripts/run-vigilcare-records-phase-11-verification.sh

Phase 12 — Batch-Type Field Requirements

What to do:

Add BatchTypeFieldRequirements record with static mapping per batch type.
Include fieldRequirements on DraftPayloadResponse and BatchDetailResponse.
Update EntryForm.vue and VerificationForm.vue to consume metadata instead of hardcoded batch-type switches.

Phase 13 — Optional OCR Pre-Fill

What to do:

Add IOcrService with Azure Document Intelligence and Tesseract providers behind Ocr:Enabled config.
Implement OcrProcessingService background polling and OcrDraftPreFiller.
Return ocrConfidence on draft payload; show confidence indicators in entry and verification forms.

Verification: ./scripts/run-vigilcare-records-phase-13-verification.sh (automated); manual Azure/Tesseract end-to-end when credentials are configured.

Deployment Notes (Small Island Context)

Single-site tenant: One hospital or health district per deployment. No cross-island federation in v1.
Offline intake (optional extension): Scan and draft entry on a local server; promotion queued until uplink to central VigilCareClinical returns. Aligns with VigilCareClinical climate-resilience Phases 20–24 — Records gateway can share the same ward-first sync pattern.
Staffing reality: Same person may hold entry and intake roles, but never entry and verifier on the same batch. System enforces this even when staff roster is small.
Paper originals: Scanned document is the working copy; physical chart remains legal original until jurisdiction defines otherwise. README must state this explicitly.

Success Metrics (Operational)

Metric	Target (6 months post go-live)
Charts with ≥1 approved batch	80% of active patients
Average verification turnaround	< 24 hours
Rejection rate	< 15% (indicates entry quality or scan quality issues if higher)
Live capture share of new observations	Trending up month-over-month
VigilCareClinical alerts from live capture	≥1 demonstrated critical-value workflow per site

Architecture Decisions

Decision	Choice (v1)	Rationale
Integrated DB vs split API	Integrated PostgreSQL, shared instance, separate schemas	Atomic promotion in one transaction; simpler to build and demo; split path documented for production hardening
Clinical approver routing	Site config map (see Site configuration); default requires physician queue for encounter/vitals/labs/medications/mixed	High-stakes chart sections get an extra gate; registration and allergy-only updates stay verifier → approver
Retroactive alerts	`enableRetroactiveAlerts: false` default on backfill	Prevents alert storms from historical critical values; opt-in per batch for facilities that accept the risk
MRN issuance	Locally generated MRN via VigilCareClinical `GenerateMrnAsync`	National health ID integration deferred per deployment

Open Questions (per deployment)

Retroactive alerts policy: Will the Ministry of Health allow backfilled critical values to trigger pages, or is storage-only the mandated default?
MRN issuance: National health ID integration vs locally generated MRN when a national registry becomes available?
Split deployment: When does the site require Records and Clinical in separate services (saga + retry) vs integrated database?

References

README.md — API reference, quick start, verification scripts, data models
digitization-workstation-guide.md — clinical scenarios (backfill, live capture, corrections)
plans/ — phase-by-phase implementation guides (Phases 1–13)
vigilcare-records-gap-analysis.md — post-phase hardening tracker and remaining open items
vigilcare-clinical-api-prd.md — downstream alerting and observation ingest
Completed/VigilCareClinicalAPI/VigilCare-Partner-Brief.md — clinical positioning and scope boundaries
Completed/national-digital-services-architecture.md — scan-and-verify pattern for paper-to-digital government services

52 KiB Raw Blame History Unescape Escape

PRD: VigilCare Records — Paper Chart Digitization & Approval Platform

Implementation Status

Overview

Goals

Non-Goals

Problem Statement

Relationship to VigilCareClinical

API Conventions

Roles and Permissions

Verification vs approval (Track A)

Site configuration

Domain Model

DigitizationBatch

DraftPatient / DraftPatientUpdate

DraftEncounter

DraftObservation

DigitizationEvent

ScannedDocument

Batch Status State Machine

Track B audit model

Features

1. Document Upload and Batch Creation — implemented (Phase 1)

2. Draft Data Entry — implemented (Phase 2)

3. Verification and Rejection — implemented (Phase 3)

4. Approval and Promotion to VigilCareClinical — implemented (Phase 4)

5. Corrections and Supersession — implemented (Phase 5)

6. Live Capture (Track B) — implemented (Phase 6)

7. Patient Registry (Draft and Live) — partially implemented

8. Work Queues and Operational Dashboard — implemented (Phase 8)

9. Authentication and Audit — implemented (Phases 1, 8, post-phase hardening)

10. Cover Sheet System — implemented (Phase 10)

11. HL7 FHIR R4 Read API — implemented (Phase 11)

12. Batch-Type Field Requirements — implemented (Phase 12)

13. Optional OCR-Assisted Pre-Fill — implemented (Phase 13; disabled by default)

Digitization Workstation UI — implemented (Phases 7, 10, 11, 12, 13)

Data Storage

Integration Contract with VigilCareClinical

Acceptance Criteria

Build Order

Step-by-Step Guide

Phase 1 — Schema, Upload, and Status Machine

Phase 2 — Draft Entry

Phase 3 — Verification and Rejection

Phase 4 — Promotion

Phase 5 — Corrections

Phase 6 — Live Capture (Track B)

Phases 7–9 — UI, Observability, Documentation

Phase 10 — Cover Sheet System

Phase 11 — FHIR R4 Read API

Phase 12 — Batch-Type Field Requirements

Phase 13 — Optional OCR Pre-Fill

Deployment Notes (Small Island Context)

Success Metrics (Operational)

Architecture Decisions

Open Questions (per deployment)

References

52 KiB

Raw Blame History