diff --git a/README.md b/README.md index abb8990..dc2faf4 100644 --- a/README.md +++ b/README.md @@ -141,6 +141,7 @@ VigilCareRecordsAPI/ │ ├── AuthController.cs # JWT login, token refresh, logout, authenticated user profile │ ├── DigitizationBatchesController.cs # Batch CRUD, document upload, assignment │ ├── DraftController.cs # Draft data entry: patient, encounter, observations, submit +│ ├── PatientsController.cs # Patient digitization history with correction chain │ ├── VerificationController.cs # Batch verification and rejection with separation of duties │ └── WorkQueueController.cs # Work queues: verification, entry, clinical approval ├── Domain/ @@ -156,6 +157,8 @@ VigilCareRecordsAPI/ │ │ │ ├── DraftEncounter.cs # Encounter context: admission, department, room/bed │ │ │ └── DraftObservation.cs # Single measurement: code, value, unit, recordedAt │ │ ├── DigitizationBatch.cs # Central workflow entity with status machine +│ │ ├── LiveEncounter.cs # Live encounter mirror for supersession / history queries +│ │ ├── LiveObservation.cs # Live observation with supersession flags (is_superseded, superseded_by_batch_id) │ │ ├── ScannedDocument.cs # MinIO object reference with SHA-256 integrity hash │ │ ├── DigitizationEvent.cs # Append-only audit trail for every status transition │ │ ├── User.cs # Username, BCrypt hash, full name, role, active flag @@ -170,12 +173,13 @@ VigilCareRecordsAPI/ │ ├── BloodType.cs # A+, A-, B+, B-, AB+, AB-, O+, O- │ └── Department.cs # Clinical departments ├── Services/ -│ ├── Interfaces/ # IAuthService, IBatchService, IDraftService, IVerificationService, IWorkQueueService, IDocumentStorageService, IPromotionService, IIdempotencyService, IMrnGenerator +│ ├── Interfaces/ # IAuthService, IBatchService, IDraftService, IVerificationService, IWorkQueueService, IDocumentStorageService, IPromotionService, IIdempotencyService, IMrnGenerator, IDigitizationHistoryService │ ├── AuthService.cs # Login (BCrypt verify), JWT generation, refresh rotation, logout, audit logging -│ ├── BatchService.cs # Batch CRUD, status machine transitions, Redis assignment lock, duplicate detection -│ ├── DraftService.cs # Draft CRUD with plausibility validation, batch-type completeness on submit +│ ├── BatchService.cs # Batch CRUD, status machine transitions, Redis assignment lock, duplicate detection, supersession validation on create +│ ├── DraftService.cs # Draft CRUD with plausibility validation, batch-type completeness on submit (relaxed for correction batches) │ ├── VerificationService.cs # Verify/reject with separation of duties, site-config clinical approval routing -│ ├── PromotionService.cs # Atomic approve + promote: draft → live tables in single transaction +│ ├── PromotionService.cs # Atomic approve + promote: draft → live tables in single transaction; supersede original observations on correction promotion +│ ├── DigitizationHistoryService.cs # Patient history with correction chain, observation counts, audit trails │ ├── IdempotencyService.cs # Idempotency-Key record storage + replay (24h TTL) │ ├── MrnGenerator.cs # PostgreSQL sequence-backed MRN generation (VCR-000001) │ ├── WorkQueueService.cs # Verification, entry, and clinical approval work queues @@ -187,7 +191,7 @@ VigilCareRecordsAPI/ │ └── SiteConfigOptions.cs # ClinicalApprovalRequired map per batch type ├── Models/Records/ │ ├── Auth/ # LoginRequest, LoginResponse, RefreshRequest, LogoutRequest, TokenResponse, UserProfileResponse -│ ├── Batch/ # CreateBatchForm, CreateBatchRequest, AssignBatchRequest, BatchDetailResponse, BatchListResponse, DraftPayloadResponse, VerifyBatchRequest, RejectBatchRequest, FieldCheck +│ ├── Batch/ # CreateBatchForm, CreateBatchRequest, AssignBatchRequest, BatchDetailResponse, SupersessionInfo, PatientDigitizationHistoryResponse, DigitizationHistoryEntry, ... │ ├── Promotion/ # ApproveRequest, PromotionResultResponse │ ├── DraftPatient/ # DraftPatientDto, UpsertDraftPatientRequest │ ├── DraftEncounter/ # DraftEncounterDto, UpsertDraftEncounterRequest @@ -199,10 +203,12 @@ VigilCareRecordsAPI/ │ ├── Configurations/ │ │ ├── Clinical/ # PatientConfiguration, EncounterConfiguration, ObservationConfiguration, OutboxEventConfiguration, IdempotencyRecordConfiguration │ │ ├── Draft/ # DraftPatientConfiguration, DraftEncounterConfiguration, DraftObservationConfiguration +│ │ ├── LiveEncounterConfiguration.cs # live_encounters table mapping +│ │ ├── LiveObservationConfiguration.cs # live_observations with supersession columns and partial index │ │ └── ... # DigitizationBatch, DigitizationEvent, ScannedDocument, User, RefreshToken, AuthAuditEvent configs │ ├── Seed/ │ │ └── DataSeeder.cs # Seeds 12 demo users (two per role) on startup -│ └── Migrations/ # InitialCreate, InitialSchema, AddRefreshTokensAndAuthAudit, AddIdempotencyRecords, AddClinicalSchemaAndPromotion, AddMrnSequence +│ └── Migrations/ # InitialCreate through AddLiveObservationAndLiveEncounter ├── Common/ │ ├── ApiResponse.cs # { success, statusCode, data, error } envelope │ └── Exceptions/ @@ -223,6 +229,7 @@ tests/ ├── DraftEntryTests.cs # Draft CRUD, plausibility validation, submit-for-verification completeness ├── VerificationTests.cs # Verification, rejection, separation of duties, clinical approval routing ├── PromotionTests.cs # Approval, atomic promotion, idempotency, separation of duties, retroactive alerts, patient dedup + ├── CorrectionSupersessionTests.cs # Correction batch supersession, validation guards, patient digitization history ├── Fixtures/ │ ├── ApiFixture.cs # WebApplicationFactory with PostgreSQL, Redis, MinIO containers │ └── DatabaseCollection.cs # Shared test collection @@ -230,13 +237,15 @@ tests/ ├── AuthHelper.cs # JWT token generation for test users ├── BatchSeedHelper.cs # Creates seeded batches at various lifecycle stages ├── BatchPipelineHelper.cs # End-to-end batch pipeline: upload → entry → verify → ready for approval + ├── CorrectionPipelineHelper.cs # Lab correction pipeline: promote original → promote correction batch └── DbResetHelper.cs # Database cleanup between tests scripts/ ├── run-vigilcare-records-verification.sh # Phase 1 — schema, auth, roles, batch CRUD, MinIO, status machine ├── run-vigilcare-records-phase-2-verification.sh # Phase 2 — draft entry, plausibility, submit-for-verification ├── run-vigilcare-records-phase-3-verification.sh # Phase 3 — verification, rejection, separation of duties, work queues -└── run-vigilcare-records-phase-4-verification.sh # Phase 4 — approval, atomic promotion, idempotency, patient dedup, outbox events +├── run-vigilcare-records-phase-4-verification.sh # Phase 4 — approval, atomic promotion, idempotency, patient dedup, outbox events +└── run-vigilcare-records-phase-5-verification.sh # Phase 5 — corrections, supersession, digitization history, audit trail docs/ ├── plans/ # Phase 1–9 implementation and verification guides @@ -284,9 +293,15 @@ docs/ ▼ ┌──────────────┐ │ PROMOTED │ (terminal — live records exist) - └──────────────┘ + └──────┬───────┘ + │ correction batch promoted (Phase 5) + ▼ + original observations marked superseded; + correction observations become active ``` +**Correction flow (Phase 5):** A `PROMOTED` batch cannot be edited in place. To fix an erroneous live value, intake uploads a new batch with `supersedesBatchId` pointing at the promoted batch. The correction goes through entry → verification → approval like any other batch. On promotion, the original batch's `live_observations` rows are soft-flagged (`is_superseded = true`, `superseded_by_batch_id` set) — never deleted. + **Allowed transitions:** | From | To | @@ -338,6 +353,10 @@ Redis serves one purpose in this project: preventing double-assignment of batche VigilCare Records draft tables and VigilCareClinical live tables share one PostgreSQL instance (separate logical concerns). Promotion runs in a single local transaction — no distributed saga required. Split deployment with HTTP + saga retry is documented as a future deployment option. +### Corrections — Append-Only Supersession + +Approved live observations are never mutated or deleted. When a transcription error is discovered after promotion, a correction batch (`supersedesBatchId`) goes through the full human workflow. On promotion, `PromotionService` marks the original `live_observations` as superseded and inserts the corrected values as new active rows. Clinical queries filter `is_superseded = false` by default; audit queries retain the full chain. No API endpoint exists to PATCH live observations directly. + --- ## Getting Started @@ -387,6 +406,7 @@ Integration tests use `WebApplicationFactory` with PostgreSQL, Redis, and MinIO | `DraftEntryTests` | 2 | Draft CRUD, plausibility validation, submit-for-verification completeness checks | | `VerificationTests` | 3 | Verification, rejection, separation of duties enforcement, clinical approval routing | | `PromotionTests` | 4 | Approval, atomic promotion to live tables, idempotency replay, separation of duties, retroactive alert policy, patient deduplication, MRN generation | +| `CorrectionSupersessionTests` | 5 | Correction batch supersession, validation guards (non-promoted, already superseded), patient digitization history, unknown patient 404 | ### Verification Scripts @@ -397,6 +417,7 @@ With the API running (`dotnet run`) and Docker Compose up: ./scripts/run-vigilcare-records-phase-2-verification.sh # Phase 2 — draft entry, plausibility, submit ./scripts/run-vigilcare-records-phase-3-verification.sh # Phase 3 — verification, rejection, separation of duties ./scripts/run-vigilcare-records-phase-4-verification.sh # Phase 4 — approval, promotion, idempotency, patient dedup +./scripts/run-vigilcare-records-phase-5-verification.sh # Phase 5 — corrections, supersession, digitization history ``` --- @@ -485,6 +506,7 @@ Error response: | `batchType` | string | yes | `PATIENT_REGISTRATION`, `ENCOUNTER_SUMMARY`, `VITALS_SHEET`, `LAB_RESULTS`, `MEDICATION_LIST`, `ALLERGY_UPDATE`, `MIXED` | | `track` | string | no | `BACKFILL` (default) or `LIVE_CAPTURE` | | `patientId` | Guid | no | Link to existing patient (enables duplicate detection) | +| `supersedesBatchId` | Guid | no | Links a correction batch to the promoted batch it will supersede on promotion | **Status codes:** @@ -492,7 +514,9 @@ Error response: |---|---| | 201 | Batch created | | 400 | Empty file or invalid MIME type | -| 409 | Duplicate document (same SHA-256 for same patient within 24 hours) | +| 404 | Superseded batch not found (`SUPERSEDED_BATCH_NOT_FOUND`) | +| 409 | Duplicate document (same SHA-256 for same patient within 24 hours); batch already superseded (`BATCH_ALREADY_SUPERSEDED`) | +| 422 | Superseded batch not in `PROMOTED` status (`SUPERSEDED_BATCH_NOT_PROMOTED`) | **PATCH `/digitization-batches/{id}/assign` body:** @@ -529,7 +553,7 @@ Error response: | `PATIENT_REGISTRATION` | Full name, date of birth, sex | | `ENCOUNTER_SUMMARY` | Linked patient; encounter with admission date, department, admission reason | | `VITALS_SHEET` | Linked patient, encounter context, at least one observation with `recordedAt` | -| `LAB_RESULTS` | Linked patient, encounter, at least one lab observation code, `recordedAt` | +| `LAB_RESULTS` | Linked patient, encounter, at least one lab observation code, `recordedAt` (correction batches with `supersedesBatchId` require observations only — patient and encounter are inherited) | | `MEDICATION_LIST` | Linked patient; `medicationsJson` with at least one entry or explicit `noActiveMedications: true` | | `ALLERGY_UPDATE` | Linked patient, allergies list (may be empty with explicit `noKnownAllergies: true`) | | `MIXED` | Linked patient, encounter context, and at least one of: observation with `recordedAt`, or complete encounter summary | @@ -620,6 +644,30 @@ Error response: All work queue endpoints support pagination via `?page=1&pageSize=20`. +### Patient Digitization History + +| Method | Path | Auth | Description | +|---|---|---|---| +| GET | `/patients/{patientId}/digitization-history` | Any authenticated | Full digitization history for a patient: batches, correction chain, observation counts, audit trails | + +**Response summary fields:** + +| Field | Type | Description | +|---|---|---| +| `patientId` | Guid | Patient ID | +| `totalBatches` | int | All batches linked to this patient | +| `promotedBatches` | int | Batches in `PROMOTED` status | +| `supersededBatches` | int | Promoted batches that have been superseded by a correction | +| `pendingBatches` | int | Batches not yet promoted or rejected | +| `entries` | array | Per-batch detail with `isCorrection`, `hasBeenSuperseded`, `supersededByBatchId`, live/superseded observation counts, and `auditTrail` | + +**Status codes:** + +| Code | Meaning | +|---|---| +| 200 | History returned | +| 404 | No digitization batches for patient (`PATIENT_HISTORY_NOT_FOUND`) | + --- ## Data Models @@ -711,7 +759,7 @@ uploadedAt DateTimeOffset ``` id Guid PK batchId Guid FK → DigitizationBatch -eventType string uploaded | entry_started | submitted_for_verification | rejected | verified | awaiting_clinical_approval | approved | promoted | ... +eventType string uploaded | entry_started | submitted_for_verification | rejected | verified | awaiting_clinical_approval | approved | promoted | correction_uploaded | correction_promoted | superseded | ... actorUserId Guid FK → User occurredAt DateTimeOffset metadataJson string? optional JSON (field checks, rejection reason, assigned-to) @@ -749,6 +797,26 @@ createdAt DateTimeOffset updatedAt DateTimeOffset ``` +### LiveObservation + +Append-only mirror of promoted observations used for supersession tracking and digitization history queries. Rows are never deleted; corrections mark prior rows as superseded. + +``` +id Guid PK +encounterId Guid FK → LiveEncounter +patientId Guid? FK → Patient +sourceBatchId Guid FK → DigitizationBatch +observationCode string e.g. K, Na +value decimal +unit string +recordedAt DateTimeOffset +note string? +isSuperseded bool default false — set true when a correction batch promotes +supersededByBatchId Guid? correction batch that replaced this observation +supersededAt DateTimeOffset? when supersession occurred +createdAt DateTimeOffset +``` + ### Observation (Clinical — Live) ``` @@ -854,7 +922,7 @@ Response shape: ## Implemented Phases -Four phases from the project roadmap are implemented and verified. Integration tests (`dotnet test`) and per-phase verification scripts cover Phases 1–4. +Five phases from the project roadmap are implemented and verified. Integration tests (`dotnet test`) and per-phase verification scripts cover Phases 1–5. | Phase | Feature | Status | |---|---|---| @@ -862,7 +930,7 @@ Four phases from the project roadmap are implemented and verified. Integration t | 2 | Draft data entry API (patient demographics, encounter context, observations), plausibility validation on observation values, batch-type completeness validation on submit-for-verification, automatic `UPLOADED → IN_ENTRY` transition on first save, assignment guard (`BATCH_NOT_ASSIGNED`), `DraftEntryTests` integration tests | Done | | 3 | Verification with field-level checks (`ok`, `warning`, `error` per field), rejection with mandatory reason (min 10 chars), separation of duties enforcement (`SEPARATION_OF_DUTIES_VIOLATION`), site-configurable clinical approval routing per batch type, work queue endpoints (verification, entry, clinical approval) with role-based access, `VerificationTests` integration tests | Done | | 4 | Approval with three-person separation of duties (entry ≠ verifier ≠ approver), atomic promotion to live `patients`/`encounters`/`observations` tables in a single PostgreSQL transaction, PostgreSQL sequence-backed MRN generation (`VCR-NNNNNN`), patient dedup by name + DOB, encounter matching by patient + department + active status, transactional outbox (`observation.created` events), retroactive alert policy per batch, `Idempotency-Key` header with 24h TTL for safe retries, `PromotionTests` integration tests, Phase 4 verification script | Done | -| 5 | Corrections and supersession — new batch replaces old, superseded observations soft-flagged | Planned | +| 5 | Correction batches via `supersedesBatchId`, supersession validation on create (`422`/`404`/`409`), append-only `live_observations` supersession flags (`is_superseded`, `superseded_by_batch_id`, `superseded_at`), correction promotion reuses original encounter and linked patient, `correction_uploaded`/`correction_promoted`/`superseded` audit events, `GET /patients/:id/digitization-history` with correction chain and per-batch audit trails, `CorrectionSupersessionTests` integration tests, Phase 5 verification script | Done | | 6 | Track B live capture with clinician attestation, synchronous alert evaluation | Planned | | 7 | Digitization workstation UI (Vue 3 side-by-side scan viewer + entry form) | Planned | | 8 | Prometheus metrics, supervisor dashboard, promotion retry job | Planned |