feature: Prometheus Metrics, Supervisor Dashboard, and Promotion Retry Job

2026-06-27 13:29:04 +08:00
parent e22d33b654
commit 04bdb7e85c
31 changed files with 4482 additions and 259 deletions
@@ -2,7 +2,7 @@

 A clinical records digitization platform built with ASP.NET Core 8, PostgreSQL, MinIO, Redis, and a Vue 3 workstation UI. The domain models the scan-to-approved lifecycle at the center of any paper chart digitization system: scanned document upload, human data entry, dual-human verification with separation of duties, site-configurable clinical approval routing, atomic promotion to live clinical tables, and governed correction via supersession.

-**Implementation status:** Phases 1–6 are complete (API core through Track B live capture). Phase 7 delivers the digitization workstation UI (`vigilcare-records-web`). Phase 8 is partially implemented — work queue overview endpoint and supervisor dashboard UI are in place; Prometheus metrics and promotion retry remain planned. See [Implemented Phases](#implemented-phases) for the full breakdown.
+**Implementation status:** Phases 1–8 are complete. Phase 7 delivers the digitization workstation UI (`vigilcare-records-web`). Phase 8 adds Prometheus metrics, supervisor work-queue overview, cursor-paginated batch audit trails, promotion retry with exponential backoff, and a Docker monitoring stack (Prometheus on **9095**, Grafana on **3013**). Phase 9 (E2E verification and clinical scenario docs) is partially implemented. See [Implemented Phases](#implemented-phases) for the full breakdown.

 ## Domain Model — How It Maps to a Real Clinical System

@@ -51,12 +51,13 @@ Append-only audit log entry for every state transition, field-level correction,
 - **Submit for Verification** — validates completeness per batch type (e.g. vitals sheet requires linked patient, encounter context, and at least one observation with `recordedAt`); transitions `IN_ENTRY → PENDING_VERIFICATION`; returns `422` with missing fields if incomplete
 - **Verification and Rejection** — verifier reviews entry against the scan with field-level checks (`fieldName`, `status: ok|warning|error`, optional `note`); verify pass transitions to `VERIFIED` or `AWAITING_CLINICAL_APPROVAL` based on site configuration for the batch type; verify fail transitions to `REJECTED` with mandatory reason; **separation of duties** enforced: entry clerk cannot verify their own batch (`409 SEPARATION_OF_DUTIES_VIOLATION`)
 - **Clinical Approval Routing** — site-configurable per batch type (`SiteConfig.ClinicalApprovalRequired`); high-stakes batch types (encounter summary, vitals sheet, lab results, medication list, mixed) route to `AWAITING_CLINICAL_APPROVAL` after verification for physician sign-off; low-stakes types (patient registration, allergy update) go directly to `VERIFIED`
- **Approval and Promotion** — `POST /digitization-batches/:id/approve` atomically promotes draft data to live VigilCareClinical tables (`patients`, `encounters`, `observations`) in a single PostgreSQL transaction; generates MRN via PostgreSQL sequence (`VCR-000001`); patient deduplication by name + DOB; encounter matching by patient + department + active status; writes `observation.created` outbox events for downstream consumers; separation of duties enforced (approver cannot be entry clerk or verifier); `Idempotency-Key` header required for safe retries with 24-hour TTL; retroactive alert policy (`enableRetroactiveAlerts`) controls whether backfill observations emit outbox events
+- **Approval and Promotion** — `POST /digitization-batches/:id/approve` (on `ApprovalController` only) atomically promotes draft data to live VigilCareClinical tables (`patients`, `encounters`, `observations`) in a single PostgreSQL transaction; generates MRN via PostgreSQL sequence (`VCR-000001`); patient deduplication by name + DOB; encounter matching by patient + department + active status; writes `observation.created` outbox events for downstream consumers; separation of duties enforced (approver cannot be entry clerk or verifier); `Idempotency-Key` header required for safe retries with 24-hour TTL; retroactive alert policy (`enableRetroactiveAlerts`) controls whether backfill observations emit outbox events; on transient infrastructure failure returns **202** with `PROMOTION_DEFERRED` — batch stays `APPROVED` and `PromotionRetryService` retries via `POST /digitization-batches/:id/promote`
 - **Promotion Result Query** — `GET /digitization-batches/:id/promotion-result` returns live entity IDs (patient, MRN, encounter, observations) created during promotion
 - **Corrections and Supersession** — approved live observations are never silently edited; a correction uploads a new batch with `supersedesBatchId`, goes through the full entry → verify → approve cycle, and on promotion marks the original batch's `live_observations` rows as `is_superseded` (append-only — never deleted); `422 SUPERSEDED_BATCH_NOT_PROMOTED`, `409 BATCH_ALREADY_SUPERSEDED`, and `404 SUPERSEDED_BATCH_NOT_FOUND` guard invalid supersession targets; correction batches inherit patient context and reuse the original promotion encounter
 - **Patient Digitization History** — `GET /patients/:id/digitization-history` returns all batches for a patient with correction chain metadata (`isCorrection`, `hasBeenSuperseded`, `supersededByBatchId`), live vs superseded observation counts, summary totals, and per-batch audit trails; `404 PATIENT_HISTORY_NOT_FOUND` when no batches exist for the patient
 - **Patient Registry Search** — `GET /patients/search?q=` searches live patients by MRN or full name (minimum 2 characters); used by the intake workstation to link uploads to existing patients
 - **Work Queues** — `GET /work-queue/verification` (batches in `PENDING_VERIFICATION`, sorted by submission time ASC); `GET /work-queue/entry` (batches in `UPLOADED`, `IN_ENTRY`, or `REJECTED`); `GET /work-queue/clinical-approval` (batches in `AWAITING_CLINICAL_APPROVAL`); `GET /work-queue/overview` (aggregate status counts, average queue age, reject rate, oldest pending verification — administrator only); role-restricted access
+- **Batch Audit Trail API** — `GET /digitization-batches/:id/events` returns cursor-paginated digitization events with actor username and full name; accessible by administrator, verifier, and clinical approver roles
 - **User Directory** — `GET /users?role=` lists active users for batch assignment (intake clerks assign entry clerks via the workstation UI)
 - **Digitization Workstation UI** — Vue 3 SPA at `vigilcare-records-web`: role-based routing (intake, entry, verification, supervisor dashboard), split-pane scan viewer with zoom/pan/rotate, draft entry with auto-save, field-level verification checkboxes, presigned URL refresh for long sessions, JWT refresh interceptor
 - **Live Capture (Track B)** — `POST /live-capture/encounters/{encounterId}/observations` and `POST /live-capture/encounters` for credentialed clinicians entering vitals at bedside; clinician attestation + password re-confirm replaces the dual-human verification gate; observations promote synchronously to live tables in a single transaction with `source = live_capture`; critical threshold evaluation runs before the response returns, with inline `criticalAlert` per observation, committed `ClinicalAlert` rows, and `observation.recorded` / `alert.generated` outbox events; `422 ATTESTATION_REQUIRED`, `422 PASSWORD_CONFIRM_INVALID`, `422 EMPTY_OBSERVATIONS`, `409 ENCOUNTER_NOT_ACTIVE`, `409 ACTIVE_ENCOUNTER_EXISTS`
@@ -65,7 +66,7 @@ Append-only audit log entry for every state transition, field-level correction,
 - **Role-Based Access** — six roles (`INTAKE_CLERK`, `DATA_ENTRY_CLERK`, `VERIFIER`, `CLINICAL_APPROVER`, `CLINICIAN`, `ADMINISTRATOR`) with role-based endpoint authorization; twelve seeded demo users (two per role)
 - **Auth Audit Events** — append-only `auth_audit_events` table records login, logout, token refresh, and failed login attempts with user ID, IP address, and timestamp
 - **Standard Envelope** — all responses use `{ success, statusCode, data, error }` wrapper; validation errors use the same shape with stable error codes
- **Observability** — Serilog structured logging with Seq sink; correlation IDs via `CorrelationIdMiddleware`; `ExceptionHandlerMiddleware` for consistent error responses
+- **Observability** — Serilog structured logging with Seq sink; correlation IDs via `CorrelationIdMiddleware`; `ExceptionHandlerMiddleware` for consistent error responses; Prometheus metrics at `GET /metrics` (HTTP request histograms, .NET runtime stats, custom gauges for batch counts by status and queue age, promotion duration histogram, rejection counter by reason category); `MetricsCollectorService` refreshes DB-backed gauges every 30 seconds; Prometheus scrapes the API via `prometheus.yml`; Grafana available at `http://localhost:3013` (dashboard panels configured manually)
 - **Swagger UI** — OpenAPI spec via Swashbuckle (Development only) at `http://localhost:5217/swagger`

 ---
@@ -89,6 +90,9 @@ HTTP request
      ├── IdempotencyService (Idempotency-Key storage + replay for safe promotion retries)
      ├── MrnGenerator (PostgreSQL sequence-backed MRN generation: VCR-000001)
      ├── WorkQueueService (verification, entry, clinical approval queues, supervisor overview metrics)
+      ├── BatchEventService (cursor-paginated batch audit trail)
+      ├── MetricsCollectorService (periodic DB gauge refresh for Prometheus)
+      ├── PromotionRetryService (exponential backoff retry for deferred promotions)
      ├── PatientRegistryService (live patient search by MRN or name)
      ├── UserDirectoryService (active user listing for batch assignment)
      ├── AttestationService (clinician role + password re-confirm for live capture)
@@ -134,6 +138,7 @@ HTTP request
 | Authentication | JWT Bearer (Microsoft.AspNetCore.Authentication.JwtBearer) |
 | Password hashing | BCrypt.Net-Next |
 | Logging | Serilog + Seq sink |
+| Metrics | Prometheus (`prometheus-net`) + Grafana |
 | Docs | Swagger / OpenAPI (Swashbuckle) |
 | Testing | xUnit + FluentAssertions + WebApplicationFactory |

@@ -147,9 +152,9 @@ VigilCareRecords/
 │   ├── Program.cs                                  # Service registration, middleware, seed on startup
 │   ├── appsettings.json                            # Connection strings, Redis, MinIO, Seq, JWT, site config
 │   ├── Controllers/
-│   │   ├── ApprovalController.cs                   # Batch approval and promotion to live clinical tables
+│   │   ├── ApprovalController.cs                   # Batch approval, promotion deferral (202), promotion to live tables
 │   │   ├── AuthController.cs                       # JWT login, token refresh, logout, authenticated user profile
-│   │   ├── DigitizationBatchesController.cs        # Batch CRUD, document upload, assignment
+│   │   ├── DigitizationBatchesController.cs        # Batch CRUD, document upload, assignment, events, promote
 │   │   ├── DraftController.cs                      # Draft data entry: patient, encounter, observations, submit
 │   │   ├── PatientsController.cs                   # Patient search and digitization history
 │   │   ├── UsersController.cs                      # User directory for batch assignment
@@ -173,7 +178,9 @@ VigilCareRecords/
 │   ├── vite.config.ts                            # Dev server on port 3028; proxies /api → localhost:5217
 │   └── tailwind.config.js                        # Clinical color palette and layout component classes
 ├── tests/
-│   └── VigilCareRecordsAPI.Tests/                # Integration tests (Phases 1–6)
+│   └── VigilCareRecordsAPI.Tests/                # Integration tests (Phases 1–8)
+├── prometheus.yml                                # Prometheus scrape config (API on host.docker.internal:5217)
+├── docker-compose.yml                            # PostgreSQL, Redis, Seq, MinIO, Prometheus, Grafana
 ├── scripts/
 │   ├── run-vigilcare-records-verification.sh       # Phase 1
 │   ├── run-vigilcare-records-phase-2-verification.sh
@@ -181,6 +188,7 @@ VigilCareRecords/
 │   ├── run-vigilcare-records-phase-4-verification.sh
 │   ├── run-vigilcare-records-phase-5-verification.sh
 │   ├── run-vigilcare-records-phase-6-verification.sh
+│   ├── run-vigilcare-records-phase-8-verification.sh  # Prometheus metrics, overview, events, retry
 │   └── run-vigilcare-records-verification-p9.sh    # E2E workflow + work queue overview smoke test
 └── docs/
    ├── plans/                                    # Phase 1–9 implementation guides
@@ -315,6 +323,8 @@ docker compose up -d
 | Redis 7 | 6383 | No auth |
 | Seq | 5346 | UI at `http://localhost:5346`, login: `admin` / `seqadmin` |
 | MinIO | 9012 (S3 API), 9013 (console) | login: `minioadmin` / `minioadmin` |
+| Prometheus | 9095 | Scrapes API at `host.docker.internal:5217/metrics`; UI at `http://localhost:9095` |
+| Grafana | 3013 | UI at `http://localhost:3013`; add Prometheus data source `http://prometheus:9090` |

 ### Install and Run

@@ -385,6 +395,7 @@ With the API running (`dotnet run`) and Docker Compose up:
 ./scripts/run-vigilcare-records-phase-4-verification.sh # Phase 4 — approval, promotion, idempotency, patient dedup
 ./scripts/run-vigilcare-records-phase-5-verification.sh # Phase 5 — corrections, supersession, digitization history
 ./scripts/run-vigilcare-records-phase-6-verification.sh # Phase 6 — live capture, attestation, critical alerts
+./scripts/run-vigilcare-records-phase-8-verification.sh # Phase 8 — Prometheus metrics, work-queue overview, batch events, promotion retry
 ./scripts/run-vigilcare-records-verification-p9.sh      # E2E workflow + work queue overview smoke test
 ```

@@ -1018,7 +1029,7 @@ Response shape:

 ## Implemented Phases

-Phases 1–6 are fully implemented and verified via integration tests and per-phase scripts. Phase 7 (workstation UI) and parts of Phase 8 (supervisor overview) are implemented.
+Phases 1–8 are fully implemented and verified via integration tests and per-phase scripts. Phase 9 (E2E verification and clinical scenario documentation) is partially implemented.

 | Phase | Feature | Status |
 |---|---|---|
@@ -1029,5 +1040,5 @@ Phases 1–6 are fully implemented and verified via integration tests and per-ph
 | 5 | Correction batches via `supersedesBatchId`, supersession validation on create (`422`/`404`/`409`), append-only `live_observations` supersession flags (`is_superseded`, `superseded_by_batch_id`, `superseded_at`), correction promotion reuses original encounter and linked patient, `correction_uploaded`/`correction_promoted`/`superseded` audit events, `GET /patients/:id/digitization-history` with correction chain and per-batch audit trails, `CorrectionSupersessionTests` integration tests, Phase 5 verification script | Done |
 | 6 | Track B live capture: `LiveCaptureController` with clinician-only endpoints, `AttestationService` (role + password re-confirm), synchronous promotion via `LiveCaptureService`, Redis-backed critical threshold evaluation, inline critical alerts + committed `ClinicalAlert` rows, `observation.recorded` and `alert.generated` outbox events, open-encounter + vitals outpatient workflow, `LiveCaptureIntegrationTests`, Phase 6 verification script | Done |
 | 7 | Digitization workstation UI (`vigilcare-records-web`): Vue 3 + Pinia + Tailwind, role-based routing, split-pane scan viewer, draft entry with auto-save, verification checkboxes, patient search, assign-clerk dialog, presigned URL refresh, supervisor dashboard shell | Done |
-| 8 | `GET /work-queue/overview`, supervisor dashboard UI, patient search API, user directory API | Partial — Prometheus metrics and promotion retry job planned |
+| 8 | Prometheus metrics (`GET /metrics`), custom digitization gauges/histograms/counters, `MetricsCollectorService`, supervisor `GET /work-queue/overview`, cursor-paginated `GET /digitization-batches/:id/events`, promotion deferral (202 / `PROMOTION_DEFERRED`), `PromotionRetryService` with exponential backoff, Docker Prometheus (9095) + Grafana (3013), Phase 8 verification script | Done |
 | 9 | E2E verification script (`run-vigilcare-records-verification-p9.sh`), clinical scenario documentation | Partial |
@@ -0,0 +1,131 @@
+using Microsoft.EntityFrameworkCore;
+
+/// <summary>
+/// Background service that periodically queries the database to update
+/// gauge metrics for Prometheus. Runs every 30 seconds.
+///
+/// Metrics updated:
+/// - digitization_batches_by_status: count per status
+/// - digitization_queue_age_seconds: age of oldest PendingVerification batch
+///
+/// Design decisions:
+/// - Uses IServiceScopeFactory (not injected AppDbContext) because
+///   BackgroundService is a singleton and AppDbContext is scoped.
+///   Each collection cycle creates a fresh scope.
+/// - 30-second interval balances freshness against DB load. Prometheus
+///   typically scrapes every 15-60 seconds, so 30 seconds ensures the
+///   gauge is never more than one scrape interval stale.
+/// - Explicit zero-setting for empty statuses prevents stale gauge values
+///   from persisting after all batches of a status are processed.
+/// - Uses UpdatedAt (not CreatedAt) for queue age because UpdatedAt
+///   reflects when the batch entered PendingVerification.
+/// </summary>
+public class MetricsCollectorService : BackgroundService
+{
+    private readonly IServiceScopeFactory _scopeFactory;
+    private readonly ILogger<MetricsCollectorService> _logger;
+    private static readonly TimeSpan CollectionInterval = TimeSpan.FromSeconds(30);
+
+    /// <summary>
+    /// All batch statuses that should be reported as gauge values.
+    /// If a status has zero batches, the gauge is set to 0 (not omitted).
+    /// </summary>
+    private static readonly BatchStatus[] AllStatuses =
+    {
+        BatchStatus.Uploaded,
+        BatchStatus.InEntry,
+        BatchStatus.PendingVerification,
+        BatchStatus.Rejected,
+        BatchStatus.Verified,
+        BatchStatus.AwaitingClinicalApproval,
+        BatchStatus.Approved,
+        BatchStatus.Promoted
+    };
+
+    public MetricsCollectorService(
+        IServiceScopeFactory scopeFactory,
+        ILogger<MetricsCollectorService> logger)
+    {
+        _scopeFactory = scopeFactory;
+        _logger = logger;
+    }
+
+    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
+    {
+        _logger.LogInformation(
+            "MetricsCollectorService started. Collection interval: {Interval}s",
+            CollectionInterval.TotalSeconds);
+
+        while (!stoppingToken.IsCancellationRequested)
+        {
+            try
+            {
+                await CollectMetricsAsync(stoppingToken);
+            }
+            catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested)
+            {
+                // Graceful shutdown — do not log as error
+                break;
+            }
+            catch (Exception ex)
+            {
+                _logger.LogError(ex,
+                    "MetricsCollectorService failed to collect metrics");
+                // Continue running — transient DB errors should not kill the collector
+            }
+
+            await Task.Delay(CollectionInterval, stoppingToken);
+        }
+
+        _logger.LogInformation("MetricsCollectorService stopped");
+    }
+
+    private async Task CollectMetricsAsync(CancellationToken ct)
+    {
+        using var scope = _scopeFactory.CreateScope();
+        var db = scope.ServiceProvider.GetRequiredService<AppDbContext>();
+
+        // --- Batch counts by status ---
+        // Single query: GROUP BY status, returns dictionary
+        var statusCounts = await db.DigitizationBatches
+            .AsNoTracking()
+            .GroupBy(b => b.Status)
+            .Select(g => new { Status = g.Key, Count = g.Count() })
+            .ToDictionaryAsync(x => x.Status, x => x.Count, ct);
+
+        // Set gauge for every status — zero out statuses with no batches
+        foreach (var status in AllStatuses)
+        {
+            var count = statusCounts.GetValueOrDefault(status, 0);
+            DiagnosticsMetrics.BatchesByStatus
+                .WithLabels(status.ToDbString())
+                .Set(count);
+        }
+
+        // --- Queue age: oldest batch in PendingVerification ---
+        var oldestPendingUpdatedAt = await db.DigitizationBatches
+            .AsNoTracking()
+            .Where(b => b.Status == BatchStatus.PendingVerification)
+            .OrderBy(b => b.UpdatedAt)
+            .Select(b => (DateTimeOffset?)b.UpdatedAt)
+            .FirstOrDefaultAsync(ct);
+
+        if (oldestPendingUpdatedAt.HasValue)
+        {
+            var ageSeconds = (DateTimeOffset.UtcNow - oldestPendingUpdatedAt.Value).TotalSeconds;
+            DiagnosticsMetrics.QueueAgeSeconds.Set(ageSeconds);
+        }
+        else
+        {
+            // No batches in PendingVerification — queue is empty
+            DiagnosticsMetrics.QueueAgeSeconds.Set(0);
+        }
+
+        _logger.LogDebug(
+            "Metrics collected: {StatusCount} status groups, queue age {QueueAge}s",
+            statusCounts.Count,
+            oldestPendingUpdatedAt.HasValue
+                ? (DateTimeOffset.UtcNow - oldestPendingUpdatedAt.Value).TotalSeconds
+                : 0);
+    }
+}
@@ -0,0 +1,274 @@
+using System.Text.Json;
+using Microsoft.EntityFrameworkCore;
+using Microsoft.Extensions.Options;
+
+/// <summary>
+/// Background service that retries failed promotion attempts with exponential backoff.
+///
+/// When VigilCareClinical is unreachable in a split deployment, promotion fails and
+/// the batch remains in Approved status. This service:
+/// 1. Finds batches in Approved status that have a failed PromotionAttempt
+///    with a NextRetryAt in the past.
+/// 2. Retries promotion via IPromotionService.
+/// 3. On failure, records a new PromotionAttempt with exponentially increasing NextRetryAt.
+/// 4. After MaxRetryAttempts, sets NextRetryAt to null (manual intervention required)
+///    and logs a critical warning.
+///
+/// The batch NEVER reverts to draft — it stays Approved until promotion succeeds
+/// or an administrator manually intervenes.
+/// </summary>
+public class PromotionRetryService : BackgroundService
+{
+    private readonly IServiceScopeFactory _scopeFactory;
+    private readonly PromotionRetryOptions _options;
+    private readonly ILogger<PromotionRetryService> _logger;
+
+    public PromotionRetryService(
+        IServiceScopeFactory scopeFactory,
+        IOptions<PromotionRetryOptions> options,
+        ILogger<PromotionRetryService> logger)
+    {
+        _scopeFactory = scopeFactory;
+        _options = options.Value;
+        _logger = logger;
+    }
+
+    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
+    {
+        _logger.LogInformation(
+            "PromotionRetryService started. Poll interval: {PollInterval}s, " +
+            "max retries: {MaxRetries}, initial delay: {InitialDelay}s, " +
+            "max delay: {MaxDelay}s, backoff multiplier: {Multiplier}",
+            _options.PollIntervalSeconds,
+            _options.MaxRetryAttempts,
+            _options.InitialDelaySeconds,
+            _options.MaxDelaySeconds,
+            _options.BackoffMultiplier);
+
+        while (!stoppingToken.IsCancellationRequested)
+        {
+            try
+            {
+                await ProcessPendingRetriesAsync(stoppingToken);
+            }
+            catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested)
+            {
+                break;
+            }
+            catch (Exception ex)
+            {
+                _logger.LogError(ex,
+                    "PromotionRetryService encountered an error during processing");
+            }
+
+            await Task.Delay(
+                TimeSpan.FromSeconds(_options.PollIntervalSeconds), stoppingToken);
+        }
+
+        _logger.LogInformation("PromotionRetryService stopped");
+    }
+
+    private async Task ProcessPendingRetriesAsync(CancellationToken ct)
+    {
+        using var scope = _scopeFactory.CreateScope();
+        var db = scope.ServiceProvider.GetRequiredService<AppDbContext>();
+        var promotionService = scope.ServiceProvider.GetRequiredService<IPromotionService>();
+        var now = DateTimeOffset.UtcNow;
+
+        // Find batches that need retry:
+        // 1. Batch is in Approved status (not yet promoted)
+        // 2. Has a failed PromotionAttempt with NextRetryAt <= now
+        var pendingRetries = await db.PromotionAttempts
+            .Include(a => a.Batch)
+            .Where(a => !a.Succeeded
+                     && a.NextRetryAt != null
+                     && a.NextRetryAt <= now
+                     && a.Batch.Status == BatchStatus.Approved)
+            .OrderBy(a => a.NextRetryAt)
+            .Take(10) // Process up to 10 retries per poll cycle
+            .ToListAsync(ct);
+
+        if (pendingRetries.Count == 0)
+            return;
+
+        _logger.LogInformation(
+            "PromotionRetryService found {Count} batches pending retry",
+            pendingRetries.Count);
+
+        foreach (var attempt in pendingRetries)
+        {
+            await RetryPromotionAsync(db, promotionService, attempt, ct);
+        }
+    }
+
+    private async Task RetryPromotionAsync(
+        AppDbContext db,
+        IPromotionService promotionService,
+        PromotionAttempt lastAttempt,
+        CancellationToken ct)
+    {
+        var batchId = lastAttempt.BatchId;
+        var nextAttemptNumber = lastAttempt.AttemptNumber + 1;
+
+        _logger.LogInformation(
+            "Retrying promotion for batch {BatchId}, attempt {Attempt}/{MaxAttempts}",
+            batchId, nextAttemptNumber, _options.MaxRetryAttempts);
+
+        try
+        {
+            // Verify batch is still in Approved status
+            var batch = await db.DigitizationBatches.FindAsync(
+                new object[] { batchId }, ct);
+
+            if (batch is null || batch.Status != BatchStatus.Approved)
+            {
+                _logger.LogWarning(
+                    "Batch {BatchId} is no longer in Approved status, skipping retry",
+                    batchId);
+                return;
+            }
+
+            var approverUserId = batch.ApprovedByUserId
+                ?? throw new InvalidOperationException(
+                    $"Batch {batchId} is in Approved status but has no ApprovedByUserId");
+
+            // Attempt promotion with a unique idempotency key per attempt
+            await promotionService.PromoteAsync(batchId, approverUserId);
+
+            // --- Success path ---
+            var successAttempt = new PromotionAttempt
+            {
+                Id = Guid.NewGuid(),
+                BatchId = batchId,
+                AttemptNumber = nextAttemptNumber,
+                Succeeded = true,
+                ErrorMessage = null,
+                AttemptedAt = DateTimeOffset.UtcNow,
+                NextRetryAt = null
+            };
+
+            db.PromotionAttempts.Add(successAttempt);
+
+            // Clear the previous attempt's NextRetryAt so it won't be picked up again
+            lastAttempt.NextRetryAt = null;
+
+            // Audit event
+            db.DigitizationEvents.Add(new DigitizationEvent
+            {
+                Id = Guid.NewGuid(),
+                BatchId = batchId,
+                EventType = DigitizationEventType.PromotionRetrySucceeded,
+                ActorUserId = approverUserId,
+                OccurredAt = DateTimeOffset.UtcNow,
+                MetadataJson = JsonSerializer.Serialize(new
+                {
+                    attemptNumber = nextAttemptNumber,
+                    retriedAt = DateTimeOffset.UtcNow
+                })
+            });
+
+            await db.SaveChangesAsync(ct);
+
+            _logger.LogInformation(
+                "Batch {BatchId} successfully promoted on retry attempt {Attempt}",
+                batchId, nextAttemptNumber);
+        }
+        catch (OperationCanceledException) when (ct.IsCancellationRequested)
+        {
+            throw; // Let the outer loop handle shutdown
+        }
+        catch (Exception ex)
+        {
+            _logger.LogWarning(ex,
+                "Promotion retry failed for batch {BatchId}, attempt {Attempt}",
+                batchId, nextAttemptNumber);
+
+            // Calculate next retry delay with exponential backoff
+            var delay = CalculateBackoffDelay(nextAttemptNumber);
+            DateTimeOffset? nextRetryAt = null;
+
+            if (nextAttemptNumber < _options.MaxRetryAttempts)
+            {
+                nextRetryAt = DateTimeOffset.UtcNow.Add(delay);
+
+                _logger.LogInformation(
+                    "Scheduling retry for batch {BatchId} at {NextRetryAt} " +
+                    "(delay: {DelaySec:F0}s, attempt {Attempt}/{Max})",
+                    batchId, nextRetryAt, delay.TotalSeconds,
+                    nextAttemptNumber, _options.MaxRetryAttempts);
+            }
+            else
+            {
+                // Exhausted all retries — manual intervention required
+                _logger.LogCritical(
+                    "Batch {BatchId} has exhausted all {Max} retry attempts. " +
+                    "Manual intervention required. Last error: {Error}",
+                    batchId, _options.MaxRetryAttempts, ex.Message);
+            }
+
+            // Record failed attempt
+            var failedAttempt = new PromotionAttempt
+            {
+                Id = Guid.NewGuid(),
+                BatchId = batchId,
+                AttemptNumber = nextAttemptNumber,
+                Succeeded = false,
+                ErrorMessage = ex.Message.Length > 2000
+                    ? ex.Message[..2000]
+                    : ex.Message,
+                AttemptedAt = DateTimeOffset.UtcNow,
+                NextRetryAt = nextRetryAt
+            };
+
+            db.PromotionAttempts.Add(failedAttempt);
+
+            // Clear the previous attempt's NextRetryAt
+            lastAttempt.NextRetryAt = null;
+
+            // Audit event
+            var batch = await db.DigitizationBatches.FindAsync(
+                new object[] { batchId }, ct);
+            var actorId = batch?.ApprovedByUserId ?? Guid.Empty;
+
+            db.DigitizationEvents.Add(new DigitizationEvent
+            {
+                Id = Guid.NewGuid(),
+                BatchId = batchId,
+                EventType = nextRetryAt.HasValue
+                    ? DigitizationEventType.PromotionRetryFailed
+                    : DigitizationEventType.PromotionRetryExhausted,
+                ActorUserId = actorId,
+                OccurredAt = DateTimeOffset.UtcNow,
+                MetadataJson = JsonSerializer.Serialize(new
+                {
+                    attemptNumber = nextAttemptNumber,
+                    error = ex.Message,
+                    nextRetryAt = nextRetryAt?.ToString("o"),
+                    exhausted = !nextRetryAt.HasValue
+                })
+            });
+
+            await db.SaveChangesAsync(ct);
+        }
+    }
+
+    /// <summary>
+    /// Calculates exponential backoff delay for a given attempt number.
+    /// Formula: min(initialDelay * multiplier^(attempt-1), maxDelay)
+    /// With jitter: adds up to 10% random jitter to prevent thundering herd
+    /// when multiple replicas retry simultaneously.
+    /// </summary>
+    private TimeSpan CalculateBackoffDelay(int attemptNumber)
+    {
+        var baseDelay = _options.InitialDelaySeconds
+            * Math.Pow(_options.BackoffMultiplier, attemptNumber - 1);
+
+        var cappedDelay = Math.Min(baseDelay, _options.MaxDelaySeconds);
+
+        // Add up to 10% jitter
+        var jitter = cappedDelay * 0.1 * Random.Shared.NextDouble();
+        var finalDelay = cappedDelay + jitter;
+
+        return TimeSpan.FromSeconds(finalDelay);
+    }
+}
@@ -0,0 +1,38 @@
+/// <summary>
+/// Configuration for the promotion retry background job.
+/// All values configurable via appsettings.json under "PromotionRetry".
+/// </summary>
+public class PromotionRetryOptions
+{
+    public const string Section = "PromotionRetry";
+
+    /// <summary>
+    /// How often the retry service checks for stuck batches (in seconds).
+    /// Default: 60 seconds.
+    /// </summary>
+    public int PollIntervalSeconds { get; set; } = 60;
+
+    /// <summary>
+    /// Initial delay before the first retry attempt (in seconds).
+    /// Default: 30 seconds.
+    /// </summary>
+    public int InitialDelaySeconds { get; set; } = 30;
+
+    /// <summary>
+    /// Maximum delay between retries (in seconds). Exponential backoff
+    /// caps at this value. Default: 900 seconds (15 minutes).
+    /// </summary>
+    public int MaxDelaySeconds { get; set; } = 900;
+
+    /// <summary>
+    /// Maximum number of retry attempts before the batch is flagged
+    /// for manual intervention. Default: 10.
+    /// </summary>
+    public int MaxRetryAttempts { get; set; } = 10;
+
+    /// <summary>
+    /// Backoff multiplier. Each retry delay is multiplied by this value.
+    /// Default: 2.0 (doubles each time).
+    /// </summary>
+    public double BackoffMultiplier { get; set; } = 2.0;
+}
@@ -1,7 +1,9 @@

 using System.Security.Claims;
+using System.Text.Json;
 using Microsoft.AspNetCore.Authorization;
 using Microsoft.AspNetCore.Mvc;
+using Microsoft.Extensions.Options;

 /// <summary>
 /// Approval and promotion of verified digitization batches to VigilCareClinical live tables.
@@ -13,11 +15,19 @@ using Microsoft.AspNetCore.Mvc;
 public class ApprovalController : ControllerBase
 {
    private readonly IPromotionService _promotion;
+    private readonly AppDbContext _db;
+    private readonly PromotionRetryOptions _retryOptions;
    private readonly ILogger<ApprovalController> _logger;

-    public ApprovalController(IPromotionService promotion, ILogger<ApprovalController> logger)
+    public ApprovalController(
+        IPromotionService promotion,
+        AppDbContext db,
+        IOptions<PromotionRetryOptions> retryOptions,
+        ILogger<ApprovalController> logger)
    {
        _promotion = promotion;
+        _db = db;
+        _retryOptions = retryOptions.Value;
        _logger = logger;
    }

@@ -28,6 +38,9 @@ public class ApprovalController : ControllerBase
    /// Requires the Idempotency-Key header for safe retries. If the same key is resubmitted,
    /// the original response is returned without re-executing the promotion.
    ///
+    /// If promotion fails due to infrastructure issues, the batch transitions to Approved status
+    /// and automatic retry is scheduled. Returns 202 Accepted with PROMOTION_DEFERRED.
+    ///
    /// Separation of duties: the approver cannot be the entry clerk or verifier of the same batch.
    /// </summary>
    /// <param name="id">The batch ID to approve.</param>
@@ -36,6 +49,7 @@ public class ApprovalController : ControllerBase
    [HttpPost("{id:guid}/approve")]
    [Authorize(Roles = "CLINICAL_APPROVER,ADMINISTRATOR")]
    [ProducesResponseType(typeof(ApiResponse<PromotionResultResponse>), StatusCodes.Status200OK)]
+    [ProducesResponseType(typeof(ApiResponse<object>), StatusCodes.Status202Accepted)]
    [ProducesResponseType(typeof(ApiResponse<object>), StatusCodes.Status400BadRequest)]
    [ProducesResponseType(typeof(ApiResponse<object>), StatusCodes.Status404NotFound)]
    [ProducesResponseType(typeof(ApiResponse<object>), StatusCodes.Status409Conflict)]
@@ -61,10 +75,22 @@ public class ApprovalController : ControllerBase
            "Approve request for batch {BatchId} by user {UserId} with idempotency key {Key}",
            id, approverUserId, idempotencyKey);

-        var result = await _promotion.ApproveAndPromoteAsync(
-            id, approverUserId, enableRetroactiveAlerts, idempotencyKey);
+        try
+        {
+            var result = await _promotion.ApproveAndPromoteAsync(
+                id, approverUserId, enableRetroactiveAlerts, idempotencyKey);

-        return Ok(ApiResponse<PromotionResultResponse>.Ok(result));
+            return Ok(ApiResponse<PromotionResultResponse>.Ok(result));
+        }
+        catch (Exception ex) when (ex is not NotFoundException
+                                && ex is not ConflictException
+                                && ex is not ValidationException)
+        {
+            _logger.LogWarning(ex,
+                "Promotion failed for batch {BatchId}, scheduling for retry", id);
+
+            return await DeferPromotionAsync(id, approverUserId, enableRetroactiveAlerts, ex);
+        }
    }

    /// <summary>
@@ -83,4 +109,81 @@ public class ApprovalController : ControllerBase
        var result = await _promotion.GetPromotionResultAsync(id);
        return Ok(ApiResponse<PromotionResultResponse>.Ok(result));
    }
+
+    private async Task<IActionResult> DeferPromotionAsync(
+        Guid batchId,
+        Guid approverUserId,
+        bool enableRetroactiveAlerts,
+        Exception ex)
+    {
+        var batch = await _db.DigitizationBatches.FindAsync(batchId);
+        if (batch is null)
+            throw new NotFoundException("Batch not found.", "BATCH_NOT_FOUND");
+
+        if (batch.Status is not (BatchStatus.Verified or BatchStatus.AwaitingClinicalApproval))
+        {
+            throw new ConflictException(
+                $"Cannot defer promotion for batch in '{batch.Status.ToDbString()}' status.",
+                "ILLEGAL_STATUS_TRANSITION");
+        }
+
+        var now = DateTimeOffset.UtcNow;
+        batch.Status = BatchStatus.Approved;
+        batch.ApprovedByUserId = approverUserId;
+        batch.EnableRetroactiveAlerts = enableRetroactiveAlerts;
+        batch.UpdatedAt = now;
+
+        _db.DigitizationEvents.Add(new DigitizationEvent
+        {
+            Id = Guid.NewGuid(),
+            BatchId = batchId,
+            EventType = DigitizationEventType.Approved,
+            ActorUserId = approverUserId,
+            OccurredAt = now,
+            MetadataJson = JsonSerializer.Serialize(new
+            {
+                enableRetroactiveAlerts,
+                promotionDeferred = true
+            })
+        });
+
+        var nextRetryAt = now.Add(TimeSpan.FromSeconds(_retryOptions.InitialDelaySeconds));
+        var attempt = new PromotionAttempt
+        {
+            Id = Guid.NewGuid(),
+            BatchId = batchId,
+            AttemptNumber = 1,
+            Succeeded = false,
+            ErrorMessage = ex.Message.Length > 2000
+                ? ex.Message[..2000]
+                : ex.Message,
+            AttemptedAt = now,
+            NextRetryAt = nextRetryAt
+        };
+
+        _db.PromotionAttempts.Add(attempt);
+
+        _db.DigitizationEvents.Add(new DigitizationEvent
+        {
+            Id = Guid.NewGuid(),
+            BatchId = batchId,
+            EventType = DigitizationEventType.PromotionFailed,
+            ActorUserId = approverUserId,
+            OccurredAt = now,
+            MetadataJson = JsonSerializer.Serialize(new
+            {
+                error = ex.Message,
+                nextRetryAt = nextRetryAt.ToString("o"),
+                scheduledForRetry = true
+            })
+        });
+
+        await _db.SaveChangesAsync();
+
+        return StatusCode(202, ApiResponse<object>.Fail(
+            202,
+            "Batch approved but promotion deferred due to infrastructure issue. " +
+            "Automatic retry has been scheduled.",
+            "PROMOTION_DEFERRED"));
+    }
 }
@@ -15,6 +15,7 @@ public class DigitizationBatchesController : ControllerBase
    private readonly IBatchService _batches;
    private readonly IDocumentStorageService _storage;
    private readonly IPromotionService _promotion;
+    private readonly IBatchEventService _batchEventService;

    private static readonly HashSet<string> _allowedMimeTypes = new()
    {
@@ -24,11 +25,13 @@ public class DigitizationBatchesController : ControllerBase
    public DigitizationBatchesController(
        IBatchService batches,
        IDocumentStorageService storage,
-        IPromotionService promotion)
+        IPromotionService promotion,
+        IBatchEventService batchEventService)
    {
        _batches = batches;
        _storage = storage;
        _promotion = promotion;
+        _batchEventService = batchEventService;
    }

    /// <summary>
@@ -142,4 +145,38 @@ public class DigitizationBatchesController : ControllerBase
        var result = await _promotion.PromoteAsync(id, actorUserId);
        return Ok(ApiResponse<PromotionResult>.Ok(result));
    }
+
+    /// <summary>
+    /// Returns cursor-paginated audit trail events for a batch.
+    /// Events are ordered chronologically (oldest first).
+    /// Pass the "after" parameter with the cursor from the previous page to paginate.
+    /// </summary>
+    /// <param name="id">Batch ID.</param>
+    /// <param name="after">Cursor: ISO-8601 timestamp from the previous page's nextCursor field.</param>
+    /// <param name="pageSize">Number of events per page. Default 50, max 200.</param>
+    [HttpGet("{id:guid}/events")]
+    [Authorize(Roles = "ADMINISTRATOR,VERIFIER,CLINICAL_APPROVER")]
+    [ProducesResponseType(typeof(ApiResponse<CursorPagedResult<BatchEventResponse>>), StatusCodes.Status200OK)]
+    [ProducesResponseType(typeof(ApiResponse<object>), StatusCodes.Status400BadRequest)]
+    [ProducesResponseType(typeof(ApiResponse<object>), StatusCodes.Status404NotFound)]
+    public async Task<IActionResult> GetEvents(
+        Guid id,
+        [FromQuery] string? after = null,
+        [FromQuery] int pageSize = 50)
+    {
+        DateTimeOffset? afterCursor = null;
+        if (!string.IsNullOrWhiteSpace(after))
+        {
+            if (!DateTimeOffset.TryParse(after, out var parsed))
+                return BadRequest(ApiResponse<object>.Fail(
+                    400,
+                    "Invalid cursor format. Expected ISO-8601 timestamp.",
+                    "INVALID_CURSOR"));
+
+            afterCursor = parsed;
+        }
+
+        var result = await _batchEventService.GetEventsAsync(id, afterCursor, pageSize);
+        return Ok(ApiResponse<CursorPagedResult<BatchEventResponse>>.Ok(result));
+    }
 }
@@ -22,6 +22,7 @@ public class AppDbContext : DbContext
    public DbSet<ClinicalAlert> ClinicalAlerts => Set<ClinicalAlert>();
    public DbSet<LiveEncounter> LiveEncounters => Set<LiveEncounter>();
    public DbSet<LiveObservation> LiveObservations => Set<LiveObservation>();
+    public DbSet<PromotionAttempt> PromotionAttempts => Set<PromotionAttempt>();

    protected override void OnModelCreating(ModelBuilder modelBuilder)
    {
@@ -0,0 +1,31 @@
+using Microsoft.EntityFrameworkCore;
+using Microsoft.EntityFrameworkCore.Metadata.Builders;
+
+public class PromotionAttemptConfiguration : IEntityTypeConfiguration<PromotionAttempt>
+{
+    public void Configure(EntityTypeBuilder<PromotionAttempt> builder)
+    {
+        builder.ToTable("promotion_attempts");
+        builder.HasKey(a => a.Id);
+        builder.Property(a => a.Id).HasColumnName("id").HasDefaultValueSql("gen_random_uuid()");
+        builder.Property(a => a.BatchId).HasColumnName("batch_id").IsRequired();
+        builder.Property(a => a.AttemptNumber).HasColumnName("attempt_number").IsRequired();
+        builder.Property(a => a.Succeeded).HasColumnName("succeeded").HasDefaultValue(false);
+        builder.Property(a => a.ErrorMessage).HasColumnName("error_message").HasMaxLength(2000);
+        builder.Property(a => a.AttemptedAt).HasColumnName("attempted_at").HasDefaultValueSql("NOW()");
+        builder.Property(a => a.NextRetryAt).HasColumnName("next_retry_at");
+
+        builder.HasOne(a => a.Batch)
+            .WithMany()
+            .HasForeignKey(a => a.BatchId)
+            .OnDelete(DeleteBehavior.Restrict);
+
+        builder.HasIndex(a => new { a.BatchId, a.AttemptNumber })
+            .IsUnique()
+            .HasDatabaseName("ix_promotion_attempts_batch_attempt");
+
+        builder.HasIndex(a => a.NextRetryAt)
+            .HasFilter("succeeded = false AND next_retry_at IS NOT NULL")
+            .HasDatabaseName("ix_promotion_attempts_pending_retry");
+    }
+}
@@ -0,0 +1,57 @@
+using System;
+using Microsoft.EntityFrameworkCore.Migrations;
+
+#nullable disable
+
+namespace VigilCareRecordsAPI.Data.Migrations
+{
+    /// <inheritdoc />
+    public partial class AddPromotionAttempts : Migration
+    {
+        /// <inheritdoc />
+        protected override void Up(MigrationBuilder migrationBuilder)
+        {
+            migrationBuilder.CreateTable(
+                name: "promotion_attempts",
+                columns: table => new
+                {
+                    id = table.Column<Guid>(type: "uuid", nullable: false, defaultValueSql: "gen_random_uuid()"),
+                    batch_id = table.Column<Guid>(type: "uuid", nullable: false),
+                    attempt_number = table.Column<int>(type: "integer", nullable: false),
+                    succeeded = table.Column<bool>(type: "boolean", nullable: false, defaultValue: false),
+                    error_message = table.Column<string>(type: "character varying(2000)", maxLength: 2000, nullable: true),
+                    attempted_at = table.Column<DateTimeOffset>(type: "timestamp with time zone", nullable: false, defaultValueSql: "NOW()"),
+                    next_retry_at = table.Column<DateTimeOffset>(type: "timestamp with time zone", nullable: true)
+                },
+                constraints: table =>
+                {
+                    table.PrimaryKey("PK_promotion_attempts", x => x.id);
+                    table.ForeignKey(
+                        name: "FK_promotion_attempts_digitization_batches_batch_id",
+                        column: x => x.batch_id,
+                        principalTable: "digitization_batches",
+                        principalColumn: "id",
+                        onDelete: ReferentialAction.Restrict);
+                });
+
+            migrationBuilder.CreateIndex(
+                name: "ix_promotion_attempts_batch_attempt",
+                table: "promotion_attempts",
+                columns: new[] { "batch_id", "attempt_number" },
+                unique: true);
+
+            migrationBuilder.CreateIndex(
+                name: "ix_promotion_attempts_pending_retry",
+                table: "promotion_attempts",
+                column: "next_retry_at",
+                filter: "succeeded = false AND next_retry_at IS NOT NULL");
+        }
+
+        /// <inheritdoc />
+        protected override void Down(MigrationBuilder migrationBuilder)
+        {
+            migrationBuilder.DropTable(
+                name: "promotion_attempts");
+        }
+    }
+}
@@ -1055,6 +1055,56 @@ namespace VigilCareRecordsAPI.Data.Migrations
                        });
                });

+            modelBuilder.Entity("PromotionAttempt", b =>
+                {
+                    b.Property<Guid>("Id")
+                        .ValueGeneratedOnAdd()
+                        .HasColumnType("uuid")
+                        .HasColumnName("id")
+                        .HasDefaultValueSql("gen_random_uuid()");
+
+                    b.Property<int>("AttemptNumber")
+                        .HasColumnType("integer")
+                        .HasColumnName("attempt_number");
+
+                    b.Property<DateTimeOffset>("AttemptedAt")
+                        .ValueGeneratedOnAdd()
+                        .HasColumnType("timestamp with time zone")
+                        .HasColumnName("attempted_at")
+                        .HasDefaultValueSql("NOW()");
+
+                    b.Property<Guid>("BatchId")
+                        .HasColumnType("uuid")
+                        .HasColumnName("batch_id");
+
+                    b.Property<string>("ErrorMessage")
+                        .HasMaxLength(2000)
+                        .HasColumnType("character varying(2000)")
+                        .HasColumnName("error_message");
+
+                    b.Property<DateTimeOffset?>("NextRetryAt")
+                        .HasColumnType("timestamp with time zone")
+                        .HasColumnName("next_retry_at");
+
+                    b.Property<bool>("Succeeded")
+                        .ValueGeneratedOnAdd()
+                        .HasColumnType("boolean")
+                        .HasDefaultValue(false)
+                        .HasColumnName("succeeded");
+
+                    b.HasKey("Id");
+
+                    b.HasIndex("NextRetryAt")
+                        .HasDatabaseName("ix_promotion_attempts_pending_retry")
+                        .HasFilter("succeeded = false AND next_retry_at IS NOT NULL");
+
+                    b.HasIndex("BatchId", "AttemptNumber")
+                        .IsUnique()
+                        .HasDatabaseName("ix_promotion_attempts_batch_attempt");
+
+                    b.ToTable("promotion_attempts", (string)null);
+                });
+
            modelBuilder.Entity("RefreshToken", b =>
                {
                    b.Property<Guid>("Id")
@@ -1345,6 +1395,17 @@ namespace VigilCareRecordsAPI.Data.Migrations
                    b.Navigation("Patient");
                });

+            modelBuilder.Entity("PromotionAttempt", b =>
+                {
+                    b.HasOne("DigitizationBatch", "Batch")
+                        .WithMany()
+                        .HasForeignKey("BatchId")
+                        .OnDelete(DeleteBehavior.Restrict)
+                        .IsRequired();
+
+                    b.Navigation("Batch");
+                });
+
            modelBuilder.Entity("RefreshToken", b =>
                {
                    b.HasOne("RefreshToken", "ReplacedByToken")
@@ -0,0 +1,89 @@
+using Prometheus;
+
+/// <summary>
+/// Application-level Prometheus metrics for the digitization pipeline.
+/// All metrics are static singletons — safe for concurrent use across
+/// all services and background workers.
+///
+/// prometheus-net throws InvalidOperationException if a metric with the
+/// same name but different label configuration is registered twice.
+/// Static fields guarantee each metric is created exactly once.
+/// </summary>
+public static class DiagnosticsMetrics
+{
+    /// <summary>
+    /// Gauge: count of digitization batches per status.
+    /// Updated periodically by MetricsCollectorService.
+    /// Labels: status (UPLOADED, IN_ENTRY, PENDING_VERIFICATION, etc.)
+    ///
+    /// This is a gauge (not a counter) because statuses change — a batch
+    /// moves from UPLOADED to IN_ENTRY, decrementing one label and
+    /// incrementing another. The gauge is set to the current count
+    /// each collection cycle.
+    /// </summary>
+    public static readonly Gauge BatchesByStatus = Metrics.CreateGauge(
+        "digitization_batches_by_status",
+        "Number of digitization batches grouped by current status.",
+        new GaugeConfiguration
+        {
+            LabelNames = new[] { "status" }
+        });
+
+    /// <summary>
+    /// Histogram: how long a promotion operation takes in seconds.
+    /// Recorded in PromotionService when a batch transitions to Promoted.
+    /// Buckets tuned for typical promotion durations (50ms to 30s).
+    ///
+    /// The p50/p95/p99 can be derived from the bucket boundaries in
+    /// Grafana using histogram_quantile().
+    /// </summary>
+    public static readonly Histogram PromotionDuration = Metrics.CreateHistogram(
+        "digitization_promotion_duration_seconds",
+        "Duration of batch promotion operations in seconds.",
+        new HistogramConfiguration
+        {
+            Buckets = new[] { 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0, 30.0 }
+        });
+
+    /// <summary>
+    /// Counter: total number of batch rejections.
+    /// Incremented in VerificationService on every rejection.
+    /// Labels: reason_category (verification_failed, clinical_rejected)
+    ///
+    /// Supervisors need to distinguish between verification-stage rejections
+    /// (data entry errors) and clinical-stage rejections (clinical judgment
+    /// issues). The label enables separate alerting thresholds.
+    /// </summary>
+    public static readonly Counter RejectionTotal = Metrics.CreateCounter(
+        "digitization_rejection_total",
+        "Total number of digitization batch rejections.",
+        new CounterConfiguration
+        {
+            LabelNames = new[] { "reason_category" }
+        });
+
+    private static readonly string[] RejectionReasonCategories =
+    {
+        "verification_failed",
+        "clinical_rejected"
+    };
+
+    static DiagnosticsMetrics()
+    {
+        // Expose all reason_category label combinations at 0 before any rejections occur.
+        foreach (var category in RejectionReasonCategories)
+            RejectionTotal.WithLabels(category).Inc(0);
+    }
+
+    /// <summary>
+    /// Gauge: age in seconds of the oldest batch in PendingVerification status.
+    /// Updated periodically by MetricsCollectorService.
+    /// A high value indicates the verification queue is backed up.
+    ///
+    /// This is a gauge because it reflects a point-in-time measurement —
+    /// the age of the oldest pending batch right now.
+    /// </summary>
+    public static readonly Gauge QueueAgeSeconds = Metrics.CreateGauge(
+        "digitization_queue_age_seconds",
+        "Age in seconds of the oldest batch in pending_verification status.");
+}
@@ -0,0 +1,17 @@
+/// <summary>
+/// Tracks individual promotion attempts for a batch. Used by the
+/// PromotionRetryService to determine retry timing and attempt count.
+/// Each row represents one attempt (successful or failed).
+/// </summary>
+public class PromotionAttempt
+{
+    public Guid Id { get; set; }
+    public Guid BatchId { get; set; }
+    public int AttemptNumber { get; set; }
+    public bool Succeeded { get; set; }
+    public string? ErrorMessage { get; set; }
+    public DateTimeOffset AttemptedAt { get; set; }
+    public DateTimeOffset? NextRetryAt { get; set; }
+
+    public DigitizationBatch Batch { get; set; } = null!;
+}
@@ -0,0 +1,15 @@
+/// <summary>
+/// A single audit trail event for a digitization batch.
+/// Includes the actor's username and full name for display
+/// without requiring a separate user lookup.
+/// </summary>
+public record BatchEventResponse(
+    Guid Id,
+    Guid BatchId,
+    string EventType,
+    Guid ActorUserId,
+    string ActorUsername,
+    string ActorFullName,
+    DateTimeOffset OccurredAt,
+    string? MetadataJson
+);
@@ -0,0 +1,15 @@
+/// <summary>
+/// Cursor-paginated result set. The cursor is the OccurredAt timestamp
+/// of the last item in this page. Pass it as the "after" query parameter
+/// to get the next page.
+///
+/// This uses the N+1 fetch pattern: fetch pageSize+1 rows, return pageSize,
+/// and use the existence of the extra row to determine HasMore without
+/// a separate COUNT query.
+/// </summary>
+public record CursorPagedResult<T>(
+    IReadOnlyList<T> Items,
+    int PageSize,
+    string? NextCursor,
+    bool HasMore
+);
@@ -3,8 +3,28 @@
 /// Returned by GET /api/v1/work-queue/overview.
 /// </summary>
 public record WorkQueueOverviewResponse(
+    /// <summary>
+    /// Count of batches per status. Key is the DB status string
+    /// (e.g. "UPLOADED", "IN_ENTRY", "PENDING_VERIFICATION").
+    /// All 8 statuses are always present, even if count is 0.
+    /// </summary>
    Dictionary<string, int> StatusCounts,
+
+    /// <summary>
+    /// Average time in minutes that batches currently in PendingVerification
+    /// have been waiting. Zero if no batches are pending.
+    /// </summary>
    double AverageTimeInQueueMinutes,
+
+    /// <summary>
+    /// Rejection rate as a decimal (0.0 to 1.0). Calculated as
+    /// rejections / (rejections + verifications) over the last 24 hours.
+    /// </summary>
    double RejectRate,
+
+    /// <summary>
+    /// Age in minutes of the oldest batch in PendingVerification status.
+    /// Zero if no batches are pending.
+    /// </summary>
    double OldestPendingVerificationMinutes
 );
@@ -3,6 +3,7 @@ using Microsoft.AspNetCore.Authentication.JwtBearer;
 using Microsoft.EntityFrameworkCore;
 using Microsoft.IdentityModel.Tokens;
 using Minio;
+using Prometheus;
 using Serilog;
 using StackExchange.Redis;

@@ -37,6 +38,9 @@ try
    builder.Services.Configure<SiteConfigOptions>(
    builder.Configuration.GetSection(SiteConfigOptions.Section));

+    builder.Services.Configure<PromotionRetryOptions>(
+    builder.Configuration.GetSection(PromotionRetryOptions.Section));
+
    // JWT Authentication
    var jwtOptions = builder.Configuration.GetSection(JwtOptions.Section).Get<JwtOptions>()!;
    builder.Services.AddAuthentication(JwtBearerDefaults.AuthenticationScheme)
@@ -70,6 +74,10 @@ try
    builder.Services.AddScoped<IDigitizationHistoryService, DigitizationHistoryService>();
    builder.Services.AddScoped<IAttestationService, AttestationService>();
    builder.Services.AddScoped<ILiveCaptureService, LiveCaptureService>();
+    builder.Services.AddScoped<IBatchEventService, BatchEventService>();
+
+    builder.Services.AddHostedService<MetricsCollectorService>();
+    builder.Services.AddHostedService<PromotionRetryService>();

    builder.Services.AddControllers();
    builder.Services.AddEndpointsApiExplorer();
@@ -90,9 +98,11 @@ try
        });
    }

+    app.UseHttpMetrics(); 
    app.UseAuthentication();
    app.UseAuthorization();
    app.MapControllers();
+    app.MapMetrics(); 

    if (!app.Environment.IsEnvironment("Testing"))
    {
@@ -14,7 +14,7 @@
      "dotnetRunMessages": true,
      "launchBrowser": true,
      "launchUrl": "swagger",
-      "applicationUrl": "http://localhost:5217",
+      "applicationUrl": "http://0.0.0.0:5217",
      "environmentVariables": {
        "ASPNETCORE_ENVIRONMENT": "Development"
      }
@@ -24,7 +24,7 @@
      "dotnetRunMessages": true,
      "launchBrowser": true,
      "launchUrl": "swagger",
-      "applicationUrl": "https://localhost:7223;http://localhost:5217",
+      "applicationUrl": "https://localhost:7223;http://0.0.0.0:5217",
      "environmentVariables": {
        "ASPNETCORE_ENVIRONMENT": "Development"
      }
@@ -0,0 +1,78 @@
+using Microsoft.EntityFrameworkCore;
+
+/// <summary>
+/// Provides cursor-paginated access to the audit trail of digitization events
+/// for a given batch. Events are ordered by occurred_at ascending with id
+/// as a tie-breaker for events at the same timestamp.
+/// </summary>
+public class BatchEventService : IBatchEventService
+{
+    private readonly AppDbContext _db;
+    private const int MaxPageSize = 200;
+    private const int DefaultPageSize = 50;
+
+    public BatchEventService(AppDbContext db)
+    {
+        _db = db;
+    }
+
+    public async Task<CursorPagedResult<BatchEventResponse>> GetEventsAsync(
+        Guid batchId, DateTimeOffset? after, int pageSize)
+    {
+        // Validate batch exists
+        var batchExists = await _db.DigitizationBatches
+            .AnyAsync(b => b.Id == batchId);
+
+        if (!batchExists)
+            throw new NotFoundException("Batch not found.", "BATCH_NOT_FOUND");
+
+        // Clamp page size
+        pageSize = Math.Clamp(pageSize, 1, MaxPageSize);
+
+        // Build query
+        var query = _db.DigitizationEvents
+            .AsNoTracking()
+            .Include(e => e.Actor)
+            .Where(e => e.BatchId == batchId);
+
+        // Apply cursor filter — only events strictly after the cursor timestamp
+        if (after.HasValue)
+        {
+            query = query.Where(e => e.OccurredAt > after.Value);
+        }
+
+        // Fetch pageSize+1 rows to determine HasMore without a COUNT query
+        var events = await query
+            .OrderBy(e => e.OccurredAt)
+            .ThenBy(e => e.Id) // tie-breaker for events at the same timestamp
+            .Take(pageSize + 1)
+            .Select(e => new BatchEventResponse(
+                e.Id,
+                e.BatchId,
+                e.EventType.ToDbString(),
+                e.ActorUserId,
+                e.Actor.Username,
+                e.Actor.FullName,
+                e.OccurredAt,
+                e.MetadataJson))
+            .ToListAsync();
+
+        var hasMore = events.Count > pageSize;
+        var page = hasMore ? events.Take(pageSize).ToList() : events;
+
+        // Build next cursor from the last item's OccurredAt
+        string? nextCursor = null;
+        if (hasMore && page.Count > 0)
+        {
+            var lastEvent = page[^1];
+            // ISO-8601 round-trip format preserves full precision
+            nextCursor = lastEvent.OccurredAt.ToString("o");
+        }
+
+        return new CursorPagedResult<BatchEventResponse>(
+            Items: page,
+            PageSize: pageSize,
+            NextCursor: nextCursor,
+            HasMore: hasMore);
+    }
+}
@@ -0,0 +1,12 @@
+public interface IBatchEventService
+{
+    /// <summary>
+    /// Returns cursor-paginated audit trail events for a batch.
+    /// Events are ordered by OccurredAt ascending (oldest first).
+    /// </summary>
+    /// <param name="batchId">The batch to query events for.</param>
+    /// <param name="after">Cursor: ISO-8601 timestamp. Only events after this timestamp are returned.</param>
+    /// <param name="pageSize">Number of events per page. Default 50, max 200.</param>
+    Task<CursorPagedResult<BatchEventResponse>> GetEventsAsync(
+        Guid batchId, DateTimeOffset? after, int pageSize);
+}
@@ -1,3 +1,4 @@
+using System.Diagnostics;
 using System.Text.Json;
 using Microsoft.EntityFrameworkCore;

@@ -39,167 +40,181 @@ public class PromotionService : IPromotionService
            }
        }

-        // --- Load batch with all draft data ---
-        var batch = await _db.DigitizationBatches
-            .Include(b => b.DraftPatient)
-            .Include(b => b.DraftEncounter)
-            .Include(b => b.DraftObservations)
-            .FirstOrDefaultAsync(b => b.Id == batchId);
-
-        if (batch is null)
-            throw new NotFoundException("Batch not found.", "BATCH_NOT_FOUND");
-
-        // --- Status validation ---
-        if (batch.Status != BatchStatus.Verified && batch.Status != BatchStatus.AwaitingClinicalApproval)
-            throw new ConflictException(
-                $"Batch must be in 'verified' or 'awaiting_clinical_approval' status to approve. Current status: '{batch.Status.ToDbString()}'.",
-                "ILLEGAL_STATUS_TRANSITION");
-
-        // --- Separation of duties: approver cannot be the entry clerk ---
-        if (batch.EnteredByUserId == approverUserId)
-            throw new ConflictException(
-                "Separation of duties: the entry clerk cannot approve their own batch.",
-                "SEPARATION_OF_DUTIES_VIOLATION");
-
-        // --- Also cannot be the verifier ---
-        if (batch.VerifiedByUserId == approverUserId)
-            throw new ConflictException(
-                "Separation of duties: the verifier cannot also approve the same batch.",
-                "SEPARATION_OF_DUTIES_VIOLATION");
-
-        // --- Validate draft data completeness (corrections reuse the linked live patient) ---
-        if (!batch.SupersedesBatchId.HasValue)
-        {
-            if (batch.DraftPatient is null)
-                throw new ValidationException("Batch has no draft patient data.", "MISSING_DRAFT_PATIENT");
-
-            if (string.IsNullOrWhiteSpace(batch.DraftPatient.FullName))
-                throw new ValidationException("Draft patient must have a full name.", "INVALID_DRAFT_PATIENT");
-        }
-        else if (!batch.PatientId.HasValue)
-        {
-            throw new ValidationException(
-                "Correction batch has no linked patient.",
-                "MISSING_PATIENT");
-        }
-
-        // --- Begin atomic transaction ---
-        await using var transaction = await _db.Database.BeginTransactionAsync();
+        // --- Start timing the promotion ---
+        var stopwatch = Stopwatch.StartNew();

        try
        {
-            var now = DateTimeOffset.UtcNow;
+            // --- Load batch with all draft data ---
+            var batch = await _db.DigitizationBatches
+                .Include(b => b.DraftPatient)
+                .Include(b => b.DraftEncounter)
+                .Include(b => b.DraftObservations)
+                .FirstOrDefaultAsync(b => b.Id == batchId);

-            // === Step 1: Create or update Patient ===
-            Patient patient;
-            if (batch.SupersedesBatchId.HasValue)
+            if (batch is null)
+                throw new NotFoundException("Batch not found.", "BATCH_NOT_FOUND");
+
+            // --- Status validation ---
+            if (batch.Status != BatchStatus.Verified && batch.Status != BatchStatus.AwaitingClinicalApproval)
+                throw new ConflictException(
+                    $"Batch must be in 'verified' or 'awaiting_clinical_approval' status to approve. Current status: '{batch.Status.ToDbString()}'.",
+                    "ILLEGAL_STATUS_TRANSITION");
+
+            // --- Separation of duties: approver cannot be the entry clerk ---
+            if (batch.EnteredByUserId == approverUserId)
+                throw new ConflictException(
+                    "Separation of duties: the entry clerk cannot approve their own batch.",
+                    "SEPARATION_OF_DUTIES_VIOLATION");
+
+            // --- Also cannot be the verifier ---
+            if (batch.VerifiedByUserId == approverUserId)
+                throw new ConflictException(
+                    "Separation of duties: the verifier cannot also approve the same batch.",
+                    "SEPARATION_OF_DUTIES_VIOLATION");
+
+            // --- Validate draft data completeness (corrections reuse the linked live patient) ---
+            if (!batch.SupersedesBatchId.HasValue)
            {
-                patient = await _db.Patients.FirstOrDefaultAsync(p => p.Id == batch.PatientId!.Value)
-                    ?? throw new NotFoundException(
-                        $"Patient {batch.PatientId.Value} not found.",
-                        "PATIENT_NOT_FOUND");
+                if (batch.DraftPatient is null)
+                    throw new ValidationException("Batch has no draft patient data.", "MISSING_DRAFT_PATIENT");
+
+                if (string.IsNullOrWhiteSpace(batch.DraftPatient.FullName))
+                    throw new ValidationException("Draft patient must have a full name.", "INVALID_DRAFT_PATIENT");
            }
-            else
+            else if (!batch.PatientId.HasValue)
            {
-                patient = await CreateOrUpdatePatientAsync(batch.DraftPatient!, now);
+                throw new ValidationException(
+                    "Correction batch has no linked patient.",
+                    "MISSING_PATIENT");
            }

-            // === Step 2: Create or match Encounter (reuse original encounter for corrections) ===
-            var encounter = await ResolveClinicalEncounterAsync(batch, patient.Id, now);
-            await EnsureLiveEncounterAsync(encounter, now);
+            // --- Begin atomic transaction ---
+            await using var transaction = await _db.Database.BeginTransactionAsync();

-            // === Step 3: Insert each DraftObservation as live Observation ===
-            var (observationIds, outboxCount) = await PromoteObservationsAsync(
-                batch, patient.Id, encounter.Id, enableRetroactiveAlerts, now);
-
-            // === Step 3b: Mirror to live_observations for supersession / digitization history ===
-            PromoteLiveObservationsAsync(batch, patient.Id, encounter.Id, now);
-
-            SupersessionResult? supersessionResult = null;
-            if (batch.SupersedesBatchId.HasValue)
+            try
            {
-                supersessionResult = await SupersedeOriginalBatchAsync(
-                    batch.SupersedesBatchId.Value, batchId, approverUserId, now);
-            }
+                var now = DateTimeOffset.UtcNow;

-            // === Step 4: Update batch status to Promoted ===
-            batch.Status = BatchStatus.Promoted;
-            batch.ApprovedByUserId = approverUserId;
-            batch.PatientId = patient.Id;
-            batch.PromotedAt = now;
-            batch.PromotionEncounterId = encounter.Id;
-            batch.EnableRetroactiveAlerts = enableRetroactiveAlerts;
-            batch.UpdatedAt = now;
-
-            // === Step 5: Write DigitizationEvent ===
-            var promotionMetadata = new Dictionary<string, object>
-            {
-                ["patientId"] = patient.Id,
-                ["mrn"] = patient.Mrn,
-                ["encounterId"] = encounter.Id,
-                ["observationCount"] = observationIds.Length,
-                ["outboxEventsWritten"] = outboxCount,
-                ["enableRetroactiveAlerts"] = enableRetroactiveAlerts,
-                ["track"] = batch.Track.ToDbString(),
-                ["isCorrection"] = batch.SupersedesBatchId.HasValue
-            };
-
-            if (supersessionResult is not null)
-            {
-                promotionMetadata["supersession"] = new
+                // === Step 1: Create or update Patient ===
+                Patient patient;
+                if (batch.SupersedesBatchId.HasValue)
                {
-                    originalBatchId = supersessionResult.OriginalBatchId,
-                    observationsSuperseded = supersessionResult.ObservationsSuperseded,
-                    observationsReplaced = supersessionResult.ObservationsReplaced
+                    patient = await _db.Patients.FirstOrDefaultAsync(p => p.Id == batch.PatientId!.Value)
+                        ?? throw new NotFoundException(
+                            $"Patient {batch.PatientId.Value} not found.",
+                            "PATIENT_NOT_FOUND");
+                }
+                else
+                {
+                    patient = await CreateOrUpdatePatientAsync(batch.DraftPatient!, now);
+                }
+
+                // === Step 2: Create or match Encounter (reuse original encounter for corrections) ===
+                var encounter = await ResolveClinicalEncounterAsync(batch, patient.Id, now);
+                await EnsureLiveEncounterAsync(encounter, now);
+
+                // === Step 3: Insert each DraftObservation as live Observation ===
+                var (observationIds, outboxCount) = await PromoteObservationsAsync(
+                    batch, patient.Id, encounter.Id, enableRetroactiveAlerts, now);
+
+                // === Step 3b: Mirror to live_observations for supersession / digitization history ===
+                PromoteLiveObservationsAsync(batch, patient.Id, encounter.Id, now);
+
+                SupersessionResult? supersessionResult = null;
+                if (batch.SupersedesBatchId.HasValue)
+                {
+                    supersessionResult = await SupersedeOriginalBatchAsync(
+                        batch.SupersedesBatchId.Value, batchId, approverUserId, now);
+                }
+
+                // === Step 4: Update batch status to Promoted ===
+                batch.Status = BatchStatus.Promoted;
+                batch.ApprovedByUserId = approverUserId;
+                batch.PatientId = patient.Id;
+                batch.PromotedAt = now;
+                batch.PromotionEncounterId = encounter.Id;
+                batch.EnableRetroactiveAlerts = enableRetroactiveAlerts;
+                batch.UpdatedAt = now;
+
+                // === Step 5: Write DigitizationEvent ===
+                var promotionMetadata = new Dictionary<string, object>
+                {
+                    ["patientId"] = patient.Id,
+                    ["mrn"] = patient.Mrn,
+                    ["encounterId"] = encounter.Id,
+                    ["observationCount"] = observationIds.Length,
+                    ["outboxEventsWritten"] = outboxCount,
+                    ["enableRetroactiveAlerts"] = enableRetroactiveAlerts,
+                    ["track"] = batch.Track.ToDbString(),
+                    ["isCorrection"] = batch.SupersedesBatchId.HasValue
                };
+
+                if (supersessionResult is not null)
+                {
+                    promotionMetadata["supersession"] = new
+                    {
+                        originalBatchId = supersessionResult.OriginalBatchId,
+                        observationsSuperseded = supersessionResult.ObservationsSuperseded,
+                        observationsReplaced = supersessionResult.ObservationsReplaced
+                    };
+                }
+
+                _db.DigitizationEvents.Add(new DigitizationEvent
+                {
+                    Id = Guid.NewGuid(),
+                    BatchId = batchId,
+                    EventType = batch.SupersedesBatchId.HasValue
+                        ? DigitizationEventType.CorrectionPromoted
+                        : DigitizationEventType.Promoted,
+                    ActorUserId = approverUserId,
+                    OccurredAt = now,
+                    MetadataJson = JsonSerializer.Serialize(promotionMetadata)
+                });
+
+                // === Step 6: Store idempotency record (within same transaction) ===
+                var result = new PromotionResultResponse(
+                    BatchId: batchId,
+                    Status: BatchStatus.Promoted.ToDbString(),
+                    PatientId: patient.Id,
+                    Mrn: patient.Mrn,
+                    EncounterId: encounter.Id,
+                    ObservationIds: observationIds,
+                    PromotedAt: now,
+                    OutboxEventsWritten: outboxCount
+                );
+
+                if (!string.IsNullOrWhiteSpace(idempotencyKey))
+                {
+                    await _idempotency.SaveAsync(
+                        idempotencyKey, "batch_promote", batchId,
+                        200, result, TimeSpan.FromHours(24));
+                }
+
+                // === Step 7: Commit ===
+                await _db.SaveChangesAsync();
+                await transaction.CommitAsync();
+
+                stopwatch.Stop();
+                DiagnosticsMetrics.PromotionDuration.Observe(stopwatch.Elapsed.TotalSeconds);
+
+                _logger.LogInformation(
+                    "Batch {BatchId} promoted in {ElapsedMs}ms: Patient {PatientId} (MRN {Mrn}), " +
+                    "Encounter {EncounterId}, {ObsCount} observations, {OutboxCount} outbox events",
+                    batchId, stopwatch.ElapsedMilliseconds, patient.Id, patient.Mrn, encounter.Id,
+                    observationIds.Length, outboxCount);
+
+                return result;
            }
-
-            _db.DigitizationEvents.Add(new DigitizationEvent
+            catch
            {
-                Id = Guid.NewGuid(),
-                BatchId = batchId,
-                EventType = batch.SupersedesBatchId.HasValue
-                    ? DigitizationEventType.CorrectionPromoted
-                    : DigitizationEventType.Promoted,
-                ActorUserId = approverUserId,
-                OccurredAt = now,
-                MetadataJson = JsonSerializer.Serialize(promotionMetadata)
-            });
-
-            // === Step 6: Store idempotency record (within same transaction) ===
-            var result = new PromotionResultResponse(
-                BatchId: batchId,
-                Status: BatchStatus.Promoted.ToDbString(),
-                PatientId: patient.Id,
-                Mrn: patient.Mrn,
-                EncounterId: encounter.Id,
-                ObservationIds: observationIds,
-                PromotedAt: now,
-                OutboxEventsWritten: outboxCount
-            );
-
-            if (!string.IsNullOrWhiteSpace(idempotencyKey))
-            {
-                await _idempotency.SaveAsync(
-                    idempotencyKey, "batch_promote", batchId,
-                    200, result, TimeSpan.FromHours(24));
+                await transaction.RollbackAsync();
+                throw;
            }
-
-            // === Step 7: Commit ===
-            await _db.SaveChangesAsync();
-            await transaction.CommitAsync();
-
-            _logger.LogInformation(
-                "Batch {BatchId} promoted: Patient {PatientId} (MRN {Mrn}), " +
-                "Encounter {EncounterId}, {ObsCount} observations, {OutboxCount} outbox events",
-                batchId, patient.Id, patient.Mrn, encounter.Id,
-                observationIds.Length, outboxCount);
-
-            return result;
        }
        catch
        {
-            await transaction.RollbackAsync();
+            stopwatch.Stop();
            throw;
        }
    }
@@ -513,117 +528,131 @@ public class PromotionService : IPromotionService

    public async Task<PromotionResult> PromoteAsync(Guid batchId, Guid actorUserId)
    {
-        await using var transaction = await _db.Database.BeginTransactionAsync();
+        var stopwatch = Stopwatch.StartNew();

        try
        {
-            // Load the batch with all draft data
-            var batch = await _db.DigitizationBatches
-                .Include(b => b.DraftPatient)
-                .Include(b => b.DraftEncounter)
-                .Include(b => b.DraftObservations)
-                .FirstOrDefaultAsync(b => b.Id == batchId);
+            await using var transaction = await _db.Database.BeginTransactionAsync();

-            if (batch is null)
-                throw new NotFoundException("Batch not found.", "BATCH_NOT_FOUND");
-
-            if (batch.Status != BatchStatus.Approved)
-                throw new ConflictException(
-                    $"Only approved batches can be promoted. Batch is in '{batch.Status.ToDbString()}' status.",
-                    "ILLEGAL_STATUS_TRANSITION");
-
-            // Resolve or create the live encounter
-            var encounterId = await ResolveEncounterAsync(batch);
-
-            // Promote draft observations to live observations
-            var now = DateTimeOffset.UtcNow;
-            var liveObservations = batch.DraftObservations.Select(draft => new LiveObservation
+            try
            {
-                Id = Guid.NewGuid(),
-                EncounterId = encounterId,
-                PatientId = batch.PatientId,
-                SourceBatchId = batch.Id,
-                ObservationCode = draft.ObservationCode,
-                Value = draft.Value,
-                Unit = draft.Unit,
-                RecordedAt = draft.RecordedAt,
-                Note = draft.Note,
-                CreatedAt = now,
-                IsSuperseded = false,
-                SupersededByBatchId = null,
-                SupersededAt = null
-            }).ToList();
+                // Load the batch with all draft data
+                var batch = await _db.DigitizationBatches
+                    .Include(b => b.DraftPatient)
+                    .Include(b => b.DraftEncounter)
+                    .Include(b => b.DraftObservations)
+                    .FirstOrDefaultAsync(b => b.Id == batchId);

-            _db.LiveObservations.AddRange(liveObservations);
+                if (batch is null)
+                    throw new NotFoundException("Batch not found.", "BATCH_NOT_FOUND");

-            // Handle supersession if this is a correction batch
-            SupersessionResult? supersessionResult = null;
-            if (batch.SupersedesBatchId.HasValue)
-            {
-                supersessionResult = await SupersedeOriginalBatchAsync(
-                    batch.SupersedesBatchId.Value,
-                    batch.Id,
-                    actorUserId,
-                    now);
-            }
+                if (batch.Status != BatchStatus.Approved)
+                    throw new ConflictException(
+                        $"Only approved batches can be promoted. Batch is in '{batch.Status.ToDbString()}' status.",
+                        "ILLEGAL_STATUS_TRANSITION");

-            // Update batch status to Promoted
-            batch.Status = BatchStatus.Promoted;
-            batch.PromotedAt = now;
-            batch.PromotionEncounterId = encounterId;
-            batch.UpdatedAt = now;
+                // Resolve or create the live encounter
+                var encounterId = await ResolveEncounterAsync(batch);

-            // Record promotion event on the correction batch
-            var promotionMetadata = new Dictionary<string, object>
-            {
-                ["encounterId"] = encounterId,
-                ["observationsPromoted"] = liveObservations.Count,
-                ["isCorrection"] = batch.SupersedesBatchId.HasValue
-            };
-
-            if (supersessionResult is not null)
-            {
-                promotionMetadata["supersession"] = new
+                // Promote draft observations to live observations
+                var now = DateTimeOffset.UtcNow;
+                var liveObservations = batch.DraftObservations.Select(draft => new LiveObservation
                {
-                    originalBatchId = supersessionResult.OriginalBatchId,
-                    observationsSuperseded = supersessionResult.ObservationsSuperseded,
-                    observationsReplaced = supersessionResult.ObservationsReplaced
+                    Id = Guid.NewGuid(),
+                    EncounterId = encounterId,
+                    PatientId = batch.PatientId,
+                    SourceBatchId = batch.Id,
+                    ObservationCode = draft.ObservationCode,
+                    Value = draft.Value,
+                    Unit = draft.Unit,
+                    RecordedAt = draft.RecordedAt,
+                    Note = draft.Note,
+                    CreatedAt = now,
+                    IsSuperseded = false,
+                    SupersededByBatchId = null,
+                    SupersededAt = null
+                }).ToList();
+
+                _db.LiveObservations.AddRange(liveObservations);
+
+                // Handle supersession if this is a correction batch
+                SupersessionResult? supersessionResult = null;
+                if (batch.SupersedesBatchId.HasValue)
+                {
+                    supersessionResult = await SupersedeOriginalBatchAsync(
+                        batch.SupersedesBatchId.Value,
+                        batch.Id,
+                        actorUserId,
+                        now);
+                }
+
+                // Update batch status to Promoted
+                batch.Status = BatchStatus.Promoted;
+                batch.PromotedAt = now;
+                batch.PromotionEncounterId = encounterId;
+                batch.UpdatedAt = now;
+
+                // Record promotion event on the correction batch
+                var promotionMetadata = new Dictionary<string, object>
+                {
+                    ["encounterId"] = encounterId,
+                    ["observationsPromoted"] = liveObservations.Count,
+                    ["isCorrection"] = batch.SupersedesBatchId.HasValue
                };
+
+                if (supersessionResult is not null)
+                {
+                    promotionMetadata["supersession"] = new
+                    {
+                        originalBatchId = supersessionResult.OriginalBatchId,
+                        observationsSuperseded = supersessionResult.ObservationsSuperseded,
+                        observationsReplaced = supersessionResult.ObservationsReplaced
+                    };
+                }
+
+                _db.DigitizationEvents.Add(new DigitizationEvent
+                {
+                    Id = Guid.NewGuid(),
+                    BatchId = batch.Id,
+                    EventType = batch.SupersedesBatchId.HasValue
+                        ? DigitizationEventType.CorrectionPromoted
+                        : DigitizationEventType.Promoted,
+                    ActorUserId = actorUserId,
+                    OccurredAt = now,
+                    MetadataJson = JsonSerializer.Serialize(promotionMetadata)
+                });
+
+                await _db.SaveChangesAsync();
+                await transaction.CommitAsync();
+
+                stopwatch.Stop();
+                DiagnosticsMetrics.PromotionDuration.Observe(stopwatch.Elapsed.TotalSeconds);
+
+                _logger.LogInformation(
+                    "Batch {BatchId} promoted in {ElapsedMs}ms (correction={IsCorrection}, " +
+                    "observations={ObservationCount}, superseded={SupersededCount})",
+                    batchId,
+                    stopwatch.ElapsedMilliseconds,
+                    batch.SupersedesBatchId.HasValue,
+                    liveObservations.Count,
+                    supersessionResult?.ObservationsSuperseded ?? 0);
+
+                return new PromotionResult(
+                    batch.Id,
+                    encounterId,
+                    liveObservations.Count,
+                    batch.SupersedesBatchId.HasValue,
+                    supersessionResult);
            }
-
-            _db.DigitizationEvents.Add(new DigitizationEvent
+            catch
            {
-                Id = Guid.NewGuid(),
-                BatchId = batch.Id,
-                EventType = batch.SupersedesBatchId.HasValue
-                    ? DigitizationEventType.CorrectionPromoted
-                    : DigitizationEventType.Promoted,
-                ActorUserId = actorUserId,
-                OccurredAt = now,
-                MetadataJson = JsonSerializer.Serialize(promotionMetadata)
-            });
-
-            await _db.SaveChangesAsync();
-            await transaction.CommitAsync();
-
-            _logger.LogInformation(
-                "Batch {BatchId} promoted (correction={IsCorrection}, " +
-                "observations={ObservationCount}, superseded={SupersededCount})",
-                batchId,
-                batch.SupersedesBatchId.HasValue,
-                liveObservations.Count,
-                supersessionResult?.ObservationsSuperseded ?? 0);
-
-            return new PromotionResult(
-                batch.Id,
-                encounterId,
-                liveObservations.Count,
-                batch.SupersedesBatchId.HasValue,
-                supersessionResult);
+                await transaction.RollbackAsync();
+                throw;
+            }
        }
        catch
        {
-            await transaction.RollbackAsync();
+            stopwatch.Stop();
            throw;
        }
    }
@@ -138,6 +138,10 @@ public class VerificationService : IVerificationService

            await _db.SaveChangesAsync();

+            DiagnosticsMetrics.RejectionTotal
+                .WithLabels("verification_failed")
+                .Inc();
+
            _logger.LogInformation(
                "Batch {BatchId} verification failed by {VerifierUserId}, rejected",
                batchId, verifierUserId);
@@ -206,6 +210,13 @@ public class VerificationService : IVerificationService

        await _db.SaveChangesAsync();

+        var category = previousStatus == BatchStatus.AwaitingClinicalApproval
+            ? "clinical_rejected"
+            : "verification_failed";
+        DiagnosticsMetrics.RejectionTotal
+            .WithLabels(category)
+            .Inc();
+
        _logger.LogInformation(
            "Batch {BatchId} rejected by {ActorUserId} from {PreviousStatus}: {Reason}",
            batchId, actorUserId, previousStatus.ToDbString(), request.Reason);
@@ -125,6 +125,7 @@ public class WorkQueueService : IWorkQueueService
            .AsNoTracking()
            .Where(e => e.OccurredAt >= cutoff)
            .Where(e => e.EventType == DigitizationEventType.Rejected
+                     || e.EventType == DigitizationEventType.VerificationFailed
                     || e.EventType == DigitizationEventType.Verified
                     || e.EventType == DigitizationEventType.VerifiedPendingClinical)
            .GroupBy(e => e.EventType)
@@ -132,7 +133,8 @@ public class WorkQueueService : IWorkQueueService
            .ToListAsync();

        var rejections = recentEvents
-            .Where(e => e.EventType == DigitizationEventType.Rejected)
+            .Where(e => e.EventType == DigitizationEventType.Rejected
+                     || e.EventType == DigitizationEventType.VerificationFailed)
            .Sum(e => e.Count);

        var verifications = recentEvents
@@ -18,6 +18,8 @@
    </PackageReference>
    <PackageReference Include="Minio" Version="6.0.3" />
    <PackageReference Include="Npgsql.EntityFrameworkCore.PostgreSQL" Version="8.0.4" />
+    <PackageReference Include="prometheus-net" Version="8.2.1" />
+    <PackageReference Include="prometheus-net.AspNetCore" Version="8.2.1" />
    <PackageReference Include="Serilog.AspNetCore" Version="8.0.2" />
    <PackageReference Include="Serilog.Sinks.Console" Version="5.0.1" />
    <PackageReference Include="Serilog.Sinks.Seq" Version="9.1.0" />
@@ -60,5 +60,12 @@
      "ALLERGY_UPDATE": false,
      "MIXED": true
    }
+  },
+  "PromotionRetry": {
+    "PollIntervalSeconds": 60,
+    "InitialDelaySeconds": 30,
+    "MaxDelaySeconds": 900,
+    "MaxRetryAttempts": 10,
+    "BackoffMultiplier": 2.0
  }
 }
@@ -0,0 +1,859 @@
+# VigilCare Clinical Platform — Gap Analysis
+
+Comprehensive gap analysis of the VigilCareClinical system covering data integrity, API surface, infrastructure reliability, security posture, observability, and test coverage. Items are ordered by **impact on correctness and patient safety first**, then **operational reliability**, then **API completeness**, then **observability and polish**.
+
+Each item includes **why** it matters and **how** to fix it at an implementation-ready level.
+
+---
+
+## Priority legend
+
+| Tier | Meaning |
+|------|---------|
+| **P0** | Data integrity or clinical correctness bug; fix before expanding clinical workflows |
+| **P1** | Silent misconfiguration, transaction safety hole, or data loss risk; fix soon after P0 |
+| **P2** | Blocks common admin/integration workflows or degrades operational reliability |
+| **P3** | Security hardening, compliance, or defense-in-depth; no immediate patient-safety impact |
+| **P4** | API completeness, consistency, and developer experience |
+| **P5** | Observability and test coverage; does not change clinical outcomes but makes incidents diagnosable |
+
+---
+
+# Part A — Data Integrity & Correctness
+
+---
+
+## P0 — MRN generation race condition
+
+### Problem
+
+`PatientService.RegisterAsync` generates MRNs via `MRN-{count+1:D6}` where `count` is a `SELECT COUNT(*)`. Two concurrent registrations can read the same count and generate duplicate MRNs. The unique index on `Patient.Mrn` catches this at the database level, but the exception surfaces as an unhandled `DbUpdateException`, not a controlled retry or user-friendly error.
+
+### Why fix
+
+MRN is the primary patient identifier across clinical systems. Duplicate MRN attempts that surface as 500 errors during FHIR bulk-import or concurrent admissions will halt ingest pipelines and require manual intervention.
+
+### How to fix
+
+1. **Replace count-based generation** with a PostgreSQL sequence: `CREATE SEQUENCE mrn_seq START WITH 1 INCREMENT BY 1`.
+2. In `PatientService.RegisterAsync`, call `SELECT nextval('mrn_seq')` to get the next MRN atomically.
+3. Format as `MRN-{sequence:D6}`.
+4. Extract MRN prefix/format to `PatientOptions` for configurability.
+5. Handle `DbUpdateException` with unique violation check as a fallback (retry once with next sequence value).
+
+**Files:** `PatientService.cs:197-200`, new migration for `mrn_seq`, optional `PatientOptions.cs`.
+
+**Dependency:** None.
+
+---
+
+## P0 — Sepsis bundle creation race condition (TOCTOU)
+
+### Problem
+
+`SepsisBundleService.CreateAsync` checks `AnyAsync(b => b.EncounterId == encounterId && b.ComplianceStatus == InProgress)` before inserting a new bundle. Two SOFA_SEPSIS alerts arriving simultaneously for the same encounter can both pass this check and create duplicate bundles, resulting in duplicate sepsis bundle elements and compliance tracking.
+
+### Why fix
+
+Duplicate bundles for the same sepsis episode create conflicting compliance timelines, confuse clinician dashboards, and may trigger duplicate paging/escalation workflows. In a clinical setting this means duplicate nurse pages for the same patient.
+
+### How to fix
+
+1. Replace `AnyAsync` check with an **idempotent INSERT** pattern matching the approach used for alert creation:
+   ```sql
+   INSERT INTO sepsis_bundles (...)
+   SELECT ... WHERE NOT EXISTS (
+     SELECT 1 FROM sepsis_bundles
+     WHERE encounter_id = @encounterId AND compliance_status = 'IN_PROGRESS'
+   )
+   ```
+2. Check `rowsAffected == 0` to detect concurrent creation; return existing bundle instead of creating a new one.
+3. Wrap bundle + elements creation in a single transaction with `SERIALIZABLE` isolation or use `FOR UPDATE` on the encounter row.
+
+**Files:** `SepsisBundleService.cs:24-29`, `SepsisBundleConfiguration.cs` (add unique filtered index on `(encounter_id) WHERE compliance_status = 'IN_PROGRESS'`).
+
+**Dependency:** None.
+
+---
+
+## P0 — Trend alert matching uses fragile LIKE pattern
+
+### Problem
+
+`TrendDetector.TryCreateAlertAsync` uses `LIKE '%{observationCode}%'` to check for existing open trend alerts. The pattern `%HEART_RATE%` could match a hypothetical `HEART_RATE_VARIABILITY` alert, and `%TEMP%` could match `TEMP_C` and `TEMP_F`. This bypasses deduplication and creates spurious alerts, or worse, suppresses alerts for the wrong vital sign.
+
+### Why fix
+
+Trend alerts fire for the 5 most critical vitals (HR, RR, SBP, Temp, SpO2). False suppression means a rapid deterioration goes unnotified; false creation means alert fatigue on a clinical floor.
+
+### How to fix
+
+1. Change the deduplication query to use **exact match** on a structured field rather than LIKE on the `Details` text column.
+2. Option A: Add an `ObservationCode` column to `ClinicalAlert` (nullable, indexed) and match on it directly.
+3. Option B: Use `Details LIKE 'Rapid deterioration: {observationCode} %'` with a prefix match instead of substring.
+4. Prefer **Option A** — it also benefits analytics queries that currently parse alert details text.
+
+**Files:** `TrendDetector.cs:102-113`, `ClinicalAlert.cs` (optional new column), `ClinicalAlertConfiguration.cs`, migration.
+
+**Dependency:** None.
+
+---
+
+## P1 — Order result → sepsis bundle update lacks spanning transaction
+
+### Problem
+
+`OrderService.RecordResultAsync` updates the order status to `Resulted`, then calls `SepsisBundleService.OnOrderResultedAsync` as a separate operation. If the bundle update fails (e.g., database timeout), the order is marked as resulted but the bundle element remains `Pending`. The bundle may then be incorrectly marked `NonCompliant` by `SepsisBundleMonitorService` even though the order was completed on time.
+
+### Why fix
+
+Sepsis bundle compliance is a CMS/Joint Commission quality metric. A false `NonCompliant` due to a transient failure triggers incorrect escalation and skews compliance reporting.
+
+### How to fix
+
+1. Wrap both operations in a single `IDbContextTransaction`:
+   ```csharp
+   using var tx = await _db.Database.BeginTransactionAsync();
+   // update order status
+   // call bundle service
+   await tx.CommitAsync();
+   ```
+2. If `OnOrderResultedAsync` fails, the entire transaction rolls back — order stays in previous state for retry.
+3. Add explicit error logging when bundle element is not found for an order (currently silent no-op at `SepsisBundleService:120`).
+
+**Files:** `OrderService.cs:100-120`, `SepsisBundleService.cs:114-162`.
+
+**Dependency:** None.
+
+---
+
+## P1 — FHIR bundle processing has no rollback on partial failure
+
+### Problem
+
+`FhirBundleProcessor` processes transaction bundles by iterating entries and calling individual service methods (patient upsert, encounter upsert, observation ingest). If entry 3 of 5 fails, entries 1-2 are already persisted. FHIR R4 transaction semantics require **all-or-nothing**: either all entries succeed or none do.
+
+### Why fix
+
+EHR integration engines (Mirth, Rhapsody) send transaction bundles expecting atomic semantics. Partial writes create orphaned records — an encounter without its patient, observations without their encounter — that break referential integrity assumptions downstream.
+
+### How to fix
+
+1. Wrap the entire bundle processing loop in a single `IDbContextTransaction`.
+2. On any entry failure, roll back the transaction and return a FHIR `OperationOutcome` with per-entry diagnostics.
+3. Collect outbox events during processing but only write them after successful commit.
+4. Add a `batch` mode (non-atomic, per-entry results) as a separate code path if needed.
+
+**Files:** `FhirBundleProcessor.cs:60-80`, `FhirIngestController.cs:186-192`.
+
+**Dependency:** None.
+
+---
+
+# Part B — Infrastructure & Reliability
+
+---
+
+## P1 — Kafka replication factor hardcoded to 1
+
+### Problem
+
+`KafkaTopicProvisioner` creates all topics with `ReplicationFactor = 1`. A single broker failure loses all unconsumed messages on those topics — including `alert.generated`, `observation.recorded`, and `sepsis.bundle.created`.
+
+### Why fix
+
+Clinical alert delivery is safety-critical. Losing `alert.generated` messages means nurses are not paged for critical vitals. Losing `observation.recorded` means scoring services miss data points, potentially delaying sepsis detection.
+
+### How to fix
+
+1. Make replication factor configurable via `KafkaTopicOptions.ReplicationFactor` (default 3 for production, 1 for dev/test).
+2. Add `MinInSyncReplicas` to topic config (recommended: 2 with RF=3).
+3. Validate on startup: if `ReplicationFactor > broker count`, log a warning and fall back to broker count.
+4. Update docker-compose with a comment noting RF=1 is dev-only.
+
+**Files:** `KafkaTopicProvisioner.cs:38`, `KafkaTopicOptions.cs`, `appsettings.json`, `appsettings.Development.json`.
+
+**Dependency:** None.
+
+---
+
+## P2 — No health check endpoints
+
+### Problem
+
+The API has no `/health` or `/ready` endpoints. There is no startup probe, no liveness check, and no readiness check for any dependency (PostgreSQL, Redis, Kafka, RabbitMQ, Elasticsearch, MinIO).
+
+### Why fix
+
+Without health checks: Kubernetes/container orchestrators cannot detect unhealthy instances and route traffic away. Load balancers send requests to instances with dead database connections. Monitoring systems cannot distinguish "service down" from "service unhealthy."
+
+### How to fix
+
+1. Add `Microsoft.Extensions.Diagnostics.HealthChecks` and provider packages:
+   - `AspNetCore.HealthChecks.NpgSql` (PostgreSQL)
+   - `AspNetCore.HealthChecks.Redis` (Redis)
+   - `AspNetCore.HealthChecks.Kafka` (Kafka)
+   - `AspNetCore.HealthChecks.RabbitMQ` (RabbitMQ)
+   - `AspNetCore.HealthChecks.Elasticsearch` (Elasticsearch)
+2. Register health checks in `Program.cs` with tags: `startup`, `liveness`, `readiness`.
+3. Map endpoints:
+   - `GET /health/live` — liveness (is the process alive?)
+   - `GET /health/ready` — readiness (are all dependencies reachable?)
+   - `GET /health/startup` — startup (has initial provisioning completed?)
+4. Expose health check results to Prometheus via `AspNetCore.HealthChecks.Publisher.Prometheus`.
+
+**Files:** `Program.cs`, `VigilCareClinicalAPI.csproj` (new packages), optional `HealthChecksConfiguration.cs`.
+
+**Dependency:** None.
+
+---
+
+## P2 — Kafka consumer poison pill causes infinite retry
+
+### Problem
+
+All 7 Kafka consumer services (SepsisEngine, News2Scoring, GcsScoring, TrendAnalyzer, WarningAlert, SofaScoring, EsIndexer) share the same error handling pattern: on exception, log error, delay 2000ms, retry. A malformed message (corrupt JSON, unknown observation code causing unhandled exception) will block the consumer indefinitely — no other messages on that partition are processed.
+
+### Why fix
+
+A single bad observation record from a misconfigured device or FHIR integration halts all downstream scoring for that partition. NEWS2, SOFA, qSOFA, and trend alerts stop computing for all patients whose observations land on the blocked partition.
+
+### How to fix
+
+1. Add a **retry counter** per message (track in memory or via Kafka headers).
+2. After `MaxRetries` (configurable, default 3), log at Error level with full message payload and **commit the offset** to skip the poison pill.
+3. Optionally publish to a dead-letter topic (`{topic}.dlq`) for manual replay.
+4. Add a Prometheus counter `kafka_consumer_poison_pills_total{consumer_group, topic}`.
+
+**Files:** All consumer services in `BackgroundServices/`: `SepsisEngineService.cs`, `News2ScoringService.cs`, `GcsScoringService.cs`, `TrendAnalyzerService.cs`, `WarningAlertService.cs`, `SofaScoringService.cs`, `EsIndexerService.cs`. Extract shared retry logic to a `KafkaConsumerBase<T>` helper.
+
+**Dependency:** None.
+
+---
+
+## P2 — Outbox relay has no dead-letter or max retry limit
+
+### Problem
+
+`OutboxRelayService` retries failed publishes every 1000ms with no maximum retry count and no dead-letter mechanism. If Kafka is down for an extended period, the outbox table grows unbounded. When Kafka recovers, a flood of stale events may overwhelm consumers.
+
+### Why fix
+
+Extended Kafka outages are common during upgrades or broker failures. Unbounded outbox growth degrades PostgreSQL query performance (the unprocessed-events index grows). Stale clinical alerts published hours late may trigger incorrect escalations.
+
+### How to fix
+
+1. Add `MaxRetryCount` and `RetryBackoffMs` to outbox configuration.
+2. Add a `retry_count` and `last_error` column to `OutboxEvent`.
+3. After `MaxRetryCount` exceeded, mark event as `FAILED` (new status column or nullable `FailedAt` timestamp).
+4. Add backoff: `delay = min(RetryBackoffMs * 2^retryCount, MaxBackoffMs)`.
+5. Add `GET /api/v1/ops/outbox?status=failed` admin endpoint for manual inspection/replay.
+6. Prometheus metrics: `outbox_pending_total`, `outbox_failed_total`.
+
+**Files:** `OutboxRelayService.cs:58-139`, `OutboxEvent.cs`, `OutboxEventConfiguration.cs`, migration, `appsettings.json`.
+
+**Dependency:** None.
+
+---
+
+## P2 — ThresholdCacheLoader crashes startup on Redis failure
+
+### Problem
+
+`ThresholdCacheLoader` runs once at startup and loads all alert thresholds into Redis. If Redis is unavailable, the service throws an unhandled exception, which may crash the entire application depending on host configuration. There is no retry logic.
+
+### Why fix
+
+Redis restarts during deployment are common. A transient Redis blip at exactly the wrong moment prevents the entire clinical API from starting, even though Redis will be available seconds later.
+
+### How to fix
+
+1. Wrap the Redis write loop in a retry with exponential backoff (3 attempts, 2s/4s/8s).
+2. On final failure, log at Error level but **allow the application to start** — the observation ingest pipeline already has a Redis-miss fallback that loads thresholds from PostgreSQL.
+3. Optionally add a background retry that re-attempts cache population after 30 seconds.
+
+**Files:** `ThresholdCacheLoader.cs`.
+
+**Dependency:** None.
+
+---
+
+## P2 — DataLake writer partial commit inconsistency
+
+### Problem
+
+`DataLakeWriterService` flushes Parquet files per partition. If 5 of 6 partitions flush successfully but one fails, the service commits Kafka offsets for the 5 successful partitions and clears their buffers. The failed partition's buffer is also cleared (line 176) even though its data was not written to MinIO. Those events are lost — they won't be re-consumed because the surrounding offsets advanced.
+
+### Why fix
+
+Data lake completeness is essential for clinical analytics, research datasets, and regulatory reporting. Silently dropped observations create gaps in longitudinal patient records.
+
+### How to fix
+
+1. **Do not clear buffers on flush failure**: only clear the buffer for partitions that flushed successfully.
+2. **Do not commit offsets for failed partitions**: track per-partition flush success and only commit offsets for successful ones.
+3. Add a retry counter per partition buffer; after `MaxFlushRetries`, log at Error with partition/offset range and clear (accept data loss with explicit audit trail) or halt the consumer for that partition.
+4. Prometheus metric: `datalake_flush_failures_total{topic, partition}`.
+
+**Files:** `DataLakeWriterService.cs:144-176`, `DataLakeOptions.cs`.
+
+**Dependency:** None.
+
+---
+
+# Part C — API Completeness & Consistency
+
+---
+
+## P2 — Missing input validators for 4 request types
+
+### Problem
+
+Four request types used by controllers have no FluentValidation validator:
+1. `TransitionStatusRequest` (encounter status changes) — no validation of `DischargeDiagnosis` length.
+2. `RecordOrderResultRequest` (order results) — no validation of `ResultSummary` length or content.
+3. `FhirPatientUpsertRequest` — no validation of FHIR-mapped fields before database write.
+4. `FhirEncounterUpsertRequest` — no validation of department/type enum mappings.
+
+The existing 9 validators cover other request types thoroughly.
+
+### Why fix
+
+Unvalidated inputs can cause database constraint violations that surface as 500 errors instead of 422s. FHIR upsert requests from integration engines may contain malformed data that is difficult to debug without validation error messages.
+
+### How to fix
+
+1. Create `TransitionStatusRequestValidator`: validate `DischargeDiagnosis` max length (500), `NewStatus` is valid enum.
+2. Create `RecordOrderResultRequestValidator`: validate `ResultSummary` max length, non-empty.
+3. Create `FhirPatientUpsertRequestValidator`: validate identifier system/value presence, gender mapping.
+4. Create `FhirEncounterUpsertRequestValidator`: validate class mapping, department code mapping, period dates.
+5. Register all in DI (auto-registration via `FluentValidation.DependencyInjectionExtensions` if not already configured).
+
+**Files:** New files in `Validators/`, `Program.cs` (DI registration if needed).
+
+**Dependency:** None.
+
+---
+
+## P4 — No patient update endpoint
+
+### Problem
+
+`PatientsController` has `POST` (register) but no `PUT`/`PATCH`. Patient demographics (blood type, allergies, emergency contact, name corrections) cannot be updated without direct database access.
+
+### Why fix
+
+Patient data corrections are a daily workflow. Allergies discovered during an encounter, emergency contact changes, and name typos all require update capability. FHIR upsert handles external system updates, but internal admin workflows have no path.
+
+### How to fix
+
+1. Add `UpdatePatientRequest` record with optional fields: `firstName`, `lastName`, `dateOfBirth`, `gender`, `bloodType`, `allergies`, `emergencyContactName`, `emergencyContactPhone`.
+2. Add `UpdatePatientRequestValidator` (same rules as registration, all fields optional).
+3. Add `PatientService.UpdateAsync(Guid id, UpdatePatientRequest)` — load, apply non-null fields, save.
+4. Add `PATCH /api/v1/patients/{id}` with `[AuthorizePermission(PatientsWrite)]`.
+5. Emit `ClinicalAuditLog` entry with before/after JSON.
+
+**Files:** `PatientsController.cs`, `PatientService.cs`, new `UpdatePatientRequest.cs`, new `UpdatePatientRequestValidator.cs`.
+
+**Dependency:** None.
+
+---
+
+## P4 — Pagination inconsistencies across list endpoints
+
+### Problem
+
+List endpoints use three different pagination strategies:
+- **1-based page/pageSize** (most controllers): `page=1, pageSize=20`
+- **0-based page** (AnalyticsController.PatientSearch): `page=0`
+- **Cursor-based** (SOFA, NEWS2, Observations): varying default limits (20, 20, 50)
+
+`AlertThresholdsController.List()` has **no pagination at all** — returns every threshold in one response. No endpoint supports sorting parameters.
+
+### Why fix
+
+Inconsistent pagination confuses integrators and dashboard developers. Missing pagination on thresholds is fine today (small dataset) but will break if observation codes expand. Missing sort parameters force client-side sorting.
+
+### How to fix
+
+1. **Standardize page-based endpoints** to 1-based pagination with consistent defaults (`page=1, pageSize=20, maxPageSize=100`).
+2. Fix AnalyticsController.PatientSearch to use 1-based pagination (breaking change — document in release notes).
+3. **Standardize cursor-based endpoints** to a consistent default limit (20).
+4. Add pagination to `AlertThresholdsController.List()` (or document that the dataset is bounded and pagination is unnecessary).
+5. Add optional `sortBy` and `sortDirection` query parameters to list endpoints where ordering matters (alerts, observations, encounters).
+
+**Files:** `AnalyticsController.cs:103`, `AlertThresholdsController.cs:37-44`, `ObservationsController.cs:80`, all list endpoints for sort params.
+
+**Dependency:** None.
+
+---
+
+## P4 — Missing list/get-by-id endpoints
+
+### Problem
+
+Several resources lack expected REST endpoints:
+1. **SepsisBundles**: No list endpoint — only get-by-encounter. No way to query all active bundles across the hospital.
+2. **qSOFA**: Only "current" endpoint — no history, unlike NEWS2/SOFA/GCS which all have history endpoints.
+3. **AlertThresholds**: No get-by-id — only list-all and get-by-code.
+4. **ReconciliationAlerts**: No API surface at all — backend-only data quality checks.
+
+### Why fix
+
+Clinical dashboards need a hospital-wide view of active sepsis bundles for charge nurse/supervisor workflows. qSOFA history is needed for trend visualization. ReconciliationAlerts are invisible to operators without SQL access.
+
+### How to fix
+
+1. Add `GET /api/v1/sepsis-bundles?status=IN_PROGRESS&page=1&pageSize=20` — list with status filter.
+2. Add `GET /api/v1/encounters/{encounterId}/qsofa/history` — mirror NEWS2/SOFA history pattern with cursor pagination.
+3. Add `GET /api/v1/alert-thresholds/{id}` for admin detail views.
+4. Add `GET /api/v1/reconciliation-alerts?resolved=false&page=1&pageSize=20` with `checkType` filter.
+
+**Files:** `SepsisBundlesController.cs`, `QsofaController.cs`, `AlertThresholdsController.cs`, new `ReconciliationAlertsController.cs`, corresponding service methods.
+
+**Dependency:** None.
+
+---
+
+## P4 — No delete operations across entire API
+
+### Problem
+
+The API has zero DELETE endpoints. The system is entirely append-only/immutable. While this is appropriate for clinical records (observations, alerts, scores), it's problematic for configuration entities like alert thresholds and for test/dev workflows.
+
+### Why fix
+
+Administrators who create test thresholds or misconfigured entries cannot remove them. Draft/test patients created during onboarding clutter the production database. This is acceptable for clinical records but not for configuration data.
+
+### How to fix
+
+1. Add `DELETE /api/v1/alert-thresholds/{id}` with `[AuthorizePermission(ThresholdsWrite)]` — hard delete for configuration data.
+2. Document explicitly that clinical entities (patients, encounters, observations, alerts, scores) are **immutable by design** and do not support deletion (regulatory compliance).
+3. Optionally add a `Patient.Status = "inactive"` transition endpoint for marking test patients without deletion.
+
+**Files:** `AlertThresholdsController.cs`, `AlertThresholdService.cs`.
+
+**Dependency:** None.
+
+---
+
+## P4 — FHIR R4 compliance limited to inbound-only facade
+
+### Problem
+
+The FHIR implementation supports only `Create` interactions (POST). The `CapabilityStatement` correctly declares this, but there are no `Read`, `Search`, or `Update` operations. Only 4 resource types are supported (Patient, Encounter, Observation, MedicationAdministration). There is no FHIR search, no `_include`/`_revinclude`, no resource versioning (ETag/If-Match), and no batch bundle mode.
+
+### Why fix
+
+EHR integrations commonly need bidirectional data flow. Care coordination systems need to read patient data back in FHIR format. Audit systems query for encounters. Without read operations, downstream systems must use the proprietary REST API instead of standard FHIR.
+
+### How to fix (phased)
+
+**Phase 1 — Read operations:**
+1. Add `GET /fhir/Patient/{id}` and `GET /fhir/Patient?identifier={system}|{value}`.
+2. Add `GET /fhir/Encounter/{id}` and `GET /fhir/Encounter?patient={patientId}`.
+3. Map internal entities back to FHIR R4 resources using reverse mappers.
+4. Update `CapabilityStatement` to include `Read` and `SearchType` interactions.
+
+**Phase 2 — Search and versioning:**
+1. Add search parameters: `_lastUpdated`, `_count`, `_offset`.
+2. Add `ETag` headers based on `UpdatedAt` or row version.
+
+**Files:** `FhirIngestController.cs`, new `FhirReadController.cs`, `FhirMetadataController.cs`, new reverse mapper classes.
+
+**Dependency:** Product decision on FHIR read scope.
+
+---
+
+# Part D — Security & Hardening
+
+---
+
+## P3 — FHIR API key not rotatable and timing-attack vulnerable
+
+### Problem
+
+`FhirApiKeyOrJwtMiddleware` compares the `X-Api-Key` header against a config value using standard string equality (`== config["Fhir:ApiKey"]`). This is vulnerable to timing attacks. The API key is stored in `appsettings.json` in plaintext and cannot be rotated without redeploying the service.
+
+### Why fix
+
+FHIR endpoints receive PHI (Protected Health Information). A compromised API key grants full integration-role access to patient data. Timing attacks are low-probability but easily prevented.
+
+### How to fix
+
+1. Replace string equality with `CryptographicOperations.FixedTimeEquals()` for constant-time comparison.
+2. Support multiple active API keys (array in config) for zero-downtime rotation.
+3. Move API keys to environment variables or a secrets manager (Azure Key Vault, AWS Secrets Manager, HashiCorp Vault).
+4. Add `X-Api-Key` rotation documentation to the ops runbook.
+5. Optionally add per-key audit logging (which key was used).
+
+**Files:** `FhirApiKeyOrJwtMiddleware.cs:46`, `FhirOptions.cs`, `appsettings.json`.
+
+**Dependency:** None.
+
+---
+
+## P3 — JWT signing key not validated on startup
+
+### Problem
+
+`JwtOptions.SigningKey` is read from configuration and used to create a `SymmetricSecurityKey`. There is no validation that the key meets minimum length requirements (256 bits for HMAC-SHA256). A short or empty key causes a runtime exception on the first authentication attempt, not at startup.
+
+### Why fix
+
+Fail-fast on misconfiguration prevents deploying a service that accepts no requests. In development, a missing or weak key wastes debugging time on cryptic `SecurityTokenInvalidSignatureException` errors.
+
+### How to fix
+
+1. Add a startup validation check in `Program.cs` after binding `JwtOptions`:
+   ```csharp
+   if (string.IsNullOrEmpty(jwtOptions.SigningKey) || 
+       Encoding.UTF8.GetByteCount(jwtOptions.SigningKey) < 32)
+       throw new InvalidOperationException("JWT SigningKey must be at least 256 bits");
+   ```
+2. Optionally add `IValidateOptions<JwtOptions>` implementation for structured validation.
+
+**Files:** `Program.cs`, optionally `JwtOptions.cs`.
+
+**Dependency:** None.
+
+---
+
+## P3 — No audit of authorization failures
+
+### Problem
+
+`PermissionAuthorizationHandler` returns `context.Fail()` when a user lacks the required permission, but does not log the attempt or write a `ClinicalAuditLog` entry. Failed authorization attempts are invisible in both application logs and the audit trail.
+
+### Why fix
+
+Security audits and compliance reviews (HIPAA, SOC2) require evidence that unauthorized access attempts are logged. Without this, there is no way to detect credential compromise or privilege escalation attempts.
+
+### How to fix
+
+1. Inject `ILogger<PermissionAuthorizationHandler>` and log at Warning level on failure: `user={username}, role={role}, requiredPermission={permission}, endpoint={resource}`.
+2. Optionally write a `ClinicalAuditLog` entry with action `AuthorizationDenied` (new enum value) for persistent audit trail.
+3. Add a Prometheus counter: `authorization_failures_total{permission, role}`.
+
+**Files:** `PermissionAuthorizationHandler.cs`, `AuditAction.cs` (new enum value), `AuditService.cs`.
+
+**Dependency:** None.
+
+---
+
+## P3 — Elasticsearch security disabled in deployment
+
+### Problem
+
+`docker-compose.yml` sets `xpack.security.enabled=false` and `xpack.security.http.ssl.enabled=false` on the Elasticsearch container. The ES instance accepts unauthenticated requests from any container on the network. The `patient_encounters` index contains PHI (patient names, MRNs, encounter details).
+
+### Why fix
+
+Any compromised container on the Docker network can read/write/delete clinical data in Elasticsearch. Even in development, this creates a risk of accidental data exposure if the Docker network is bridged to a shared network.
+
+### How to fix
+
+1. Enable `xpack.security.enabled=true` in docker-compose.
+2. Set `ELASTIC_PASSWORD` via Docker secrets or `.env` file.
+3. Update `ElasticsearchOptions` to include `Username`, `Password`, and `UseTls` fields.
+4. Configure the .NET `ElasticClient` with basic auth credentials.
+5. Document that production deployments must use TLS + authentication.
+
+**Files:** `docker-compose.yml:67`, new `ElasticsearchOptions.cs` fields, `EsIndexerService.cs`, `ElasticIndexProvisioner.cs`.
+
+**Dependency:** None.
+
+---
+
+## ~~P3 — No token refresh or revocation mechanism~~ DONE
+
+Implemented: `RefreshToken` entity with DB-backed storage, `POST /api/v1/auth/refresh` (rotate refresh token + issue new access token), `POST /api/v1/auth/logout` (revoke refresh token server-side). Access token reduced to 15 min, refresh token 7 days. Frontend auto-refreshes before expiry, retries on 401, and redirects to login on refresh failure. Logout button in header, sidebar, and mobile nav. Audit logged as `USER_LOGOUT` and `TOKEN_REFRESHED`.
+
+---
+
+# Part E — Observability & Operations
+
+---
+
+## P5 — No request/response timing metrics
+
+### Problem
+
+The API has Prometheus metrics for clinical events (alerts, bundles, consumer lag) but no HTTP request timing histograms. There is no way to measure API latency, identify slow endpoints, or set SLOs.
+
+### Why fix
+
+Clinical dashboards and FHIR integrations depend on API responsiveness. Without latency metrics, there is no baseline for alerting on degradation, and performance regressions go undetected until users report them.
+
+### How to fix
+
+1. Add `prometheus-net.AspNetCore` middleware: `app.UseHttpMetrics()` in `Program.cs`.
+2. This automatically provides `http_request_duration_seconds` histogram with labels: `method`, `controller`, `action`, `status_code`.
+3. Add Grafana dashboard panels for p50/p95/p99 latency per endpoint.
+4. Set initial SLO targets (e.g., observation ingest p95 < 200ms).
+
+**Files:** `Program.cs:276` (add `app.UseHttpMetrics()` before `app.MapMetrics()`), `VigilCareClinicalAPI.csproj` (package).
+
+**Dependency:** None.
+
+---
+
+## P5 — Background service errors not metricked
+
+### Problem
+
+Kafka consumer services, outbox relay, bundle monitor, and RabbitMQ workers log errors but do not increment Prometheus counters on failure. The only background service metrics are `sepsis_bundle_compliance_total` and `kafka_consumer_lag`. There are no failure-rate metrics for any background service.
+
+### Why fix
+
+Log-based alerting requires parsing structured logs. Metrics-based alerting (Prometheus + Alertmanager) is standard in production Kubernetes deployments and enables rate-of-change alerts ("consumer errors spiking") that are impossible with log grep.
+
+### How to fix
+
+1. Add counters per background service:
+   - `kafka_consumer_errors_total{consumer_group, topic, error_type}`
+   - `outbox_relay_failures_total{reason}`
+   - `rabbitmq_worker_errors_total{queue, error_type}`
+   - `datalake_flush_failures_total{topic, partition}`
+2. Add processing duration histograms:
+   - `kafka_consumer_processing_seconds{consumer_group}`
+   - `outbox_relay_batch_seconds`
+3. Increment counters in existing catch blocks (minimal code change).
+
+**Files:** All background services, new `BackgroundServiceMetrics.cs` static class for metric definitions.
+
+**Dependency:** None.
+
+---
+
+## P5 — Thin test coverage for concurrent operations and background services
+
+### Problem
+
+Test coverage analysis reveals:
+- **No concurrent operation tests**: No tests for simultaneous alert creation, parallel observation ingest, or race conditions in deduplication logic.
+- **Thin background service tests**: Kafka consumer behavior, outbox relay failure recovery, and RabbitMQ worker retry logic are not directly tested.
+- **No performance tests**: No benchmarks for observation ingest throughput, scoring latency, or alert pipeline end-to-end timing.
+- **No chaos tests**: No fault injection for database/Redis/Kafka/RabbitMQ failures.
+
+Well-tested areas include: clinical scoring (qSOFA, SOFA, NEWS2, GCS), alert lifecycle, FHIR ingest, medication correlation, sepsis bundle tracking, and end-to-end scenarios.
+
+### Why fix
+
+The concurrent operation gaps directly correspond to P0 race conditions identified in this document (MRN generation, sepsis bundle creation). Without concurrent tests, fixes cannot be verified. Background service resilience is untested, meaning the Kafka poison pill and outbox retry gaps have no regression safety net.
+
+### How to fix
+
+1. **Concurrent operation tests** (priority — validates P0 fixes):
+   - Parallel patient registration with same demographics → verify unique MRN.
+   - Parallel SOFA_SEPSIS alerts for same encounter → verify single bundle.
+   - Parallel observation ingest with same idempotency key → verify single record.
+
+2. **Background service tests**:
+   - Test Kafka consumer with malformed message → verify skip after max retries.
+   - Test outbox relay with simulated Kafka failure → verify retry and eventual dead-letter.
+   - Test PagingWorker with acknowledged alert → verify no escalation.
+
+3. **Performance benchmarks** (optional, lower priority):
+   - Observation ingest throughput (target: 1000/sec per instance).
+   - Alert pipeline latency (observation → alert → page: target < 5s p95).
+
+**Files:** New test files in `VigilCareClinicalAPI.Tests/`: `ConcurrencyTests.cs`, `BackgroundServiceTests.cs`, optional `BenchmarkTests.cs`.
+
+**Dependency:** P0 fixes (concurrent tests validate the fixes).
+
+---
+
+# Part F — Hardcoded Values & Configuration Gaps
+
+---
+
+## P4 — Clinical parameters hardcoded instead of configurable
+
+### Problem
+
+Several clinically significant parameters are hardcoded:
+| Value | Location | Current |
+|-------|----------|---------|
+| Sepsis bundle deadline | `SepsisBundleService.cs:39` | 1 hour |
+| Bundle monitor scan interval | `SepsisBundleMonitorService.cs:5` | 5 minutes |
+| qSOFA criterion TTL | `QsofaDetector.cs:8` | 1800 seconds |
+| GCS/NEWS2 scoring TTL | `GcsDetector.cs:8`, `News2Detector.cs:8` | 14400 seconds |
+| MRN format pattern | `PatientService.cs:200` | `MRN-{count:D6}` |
+| Paging worker poll interval | `PagingWorkerService.cs` | 2 seconds |
+
+### Why fix
+
+Different hospitals and clinical settings have different protocols. CMS Sepsis SEP-1 requires a 3-hour bundle, not 1-hour. Facilities operating under different guidelines need to adjust these parameters without code changes.
+
+### How to fix
+
+1. Move sepsis bundle deadline to `SepsisOptions.BundleDeadlineHours` (default 1, CMS standard 3).
+2. Move bundle monitor scan interval to `SepsisOptions.MonitorScanIntervalMinutes`.
+3. Move qSOFA TTL to `QsofaOptions.CriterionTtlSeconds`.
+4. Move GCS/NEWS2 TTL to a shared `ScoringOptions.CalculationTtlSeconds`.
+5. Move MRN format to `PatientOptions.MrnPrefix` and `MrnDigits`.
+6. All via `IOptions<T>` pattern already established in the codebase.
+
+**Files:** Respective service files, new/updated options classes, `appsettings.json`.
+
+**Dependency:** None.
+
+---
+
+# Summary matrix
+
+| # | Issue | Priority | Part | Status |
+|---|-------|----------|------|--------|
+| 1 | MRN generation race condition | P0 | A | Open |
+| 2 | Sepsis bundle creation TOCTOU | P0 | A | Open |
+| 3 | Trend alert LIKE pattern | P0 | A | Open |
+| 4 | Order→Bundle transaction gap | P1 | A | Open |
+| 5 | FHIR bundle no rollback | P1 | A | Open |
+| 6 | Kafka replication factor = 1 | P1 | B | Open |
+| 7 | No health check endpoints | P2 | B | Open |
+| 8 | Kafka consumer poison pill | P2 | B | Open |
+| 9 | Outbox relay no dead-letter | P2 | B | Open |
+| 10 | ThresholdCacheLoader crash on Redis | P2 | B | Open |
+| 11 | DataLake partial commit | P2 | B | Open |
+| 12 | Missing input validators | P2 | C | Open |
+| 13 | No patient update endpoint | P4 | C | Open |
+| 14 | Pagination inconsistencies | P4 | C | Open |
+| 15 | Missing list/get endpoints | P4 | C | Open |
+| 16 | No delete operations | P4 | C | Open |
+| 17 | FHIR R4 read-only facade | P4 | C | Open |
+| 18 | API key timing attack + rotation | P3 | D | Open |
+| 19 | JWT key not validated on startup | P3 | D | Open |
+| 20 | No authorization failure audit | P3 | D | Open |
+| 21 | Elasticsearch security disabled | P3 | D | Open |
+| 22 | ~~No token refresh/revocation~~ | P3 | D | **Done** |
+| 23 | No request timing metrics | P5 | E | Open |
+| 24 | Background service error metrics | P5 | E | Open |
+| 25 | Thin concurrent/resilience tests | P5 | E | Open |
+| 26 | Clinical params hardcoded | P4 | F | Open |
+
+---
+
+## Suggested implementation sequence
+
+```mermaid
+flowchart TD
+    subgraph correctness [Part A — Correctness]
+        P0A[P0: MRN sequence]
+        P0B[P0: Bundle idempotent INSERT]
+        P0C[P0: Trend exact match]
+        P1A[P1: Order→Bundle transaction]
+        P1B[P1: FHIR bundle rollback]
+    end
+
+    subgraph infra [Part B — Infrastructure]
+        P1K[P1: Kafka replication factor]
+        P2H[P2: Health checks]
+        P2P[P2: Poison pill handling]
+        P2O[P2: Outbox dead-letter]
+        P2T[P2: ThresholdCacheLoader retry]
+        P2D[P2: DataLake partial commit]
+    end
+
+    subgraph security [Part D — Security]
+        P3K[P3: API key hardening]
+        P3J[P3: JWT validation]
+        P3A[P3: Auth failure audit]
+        P3E[P3: ES security]
+        P3R[P3: Token refresh]
+    end
+
+    subgraph api [Part C — API]
+        P2V[P2: Missing validators]
+        P4P[P4: Patient update]
+        P4G[P4: Pagination/sorting]
+        P4L[P4: Missing endpoints]
+        P4F[P4: FHIR read ops]
+    end
+
+    subgraph obs [Part E — Observability]
+        P5M[P5: Request metrics]
+        P5B[P5: Background metrics]
+        P5T[P5: Concurrent tests]
+    end
+
+    P0A --> P5T
+    P0B --> P5T
+    P0C --> P5T
+    P2P --> P5B
+    P2O --> P5B
+```
+
+### Sprint-sized batches
+
+| Batch | Items | Outcome |
+|-------|-------|---------|
+| **1 — Correctness** | P0 MRN sequence, P0 bundle idempotent INSERT, P0 trend exact match, P1 order→bundle tx, P1 FHIR rollback | Race conditions eliminated; clinical data integrity guaranteed |
+| **2 — Infrastructure resilience** | P1 Kafka RF, P2 health checks, P2 poison pill, P2 outbox dead-letter, P2 ThresholdCacheLoader, P2 DataLake commit | Production-ready infrastructure; no silent data loss |
+| **3 — Security hardening** | P3 API key, P3 JWT validation, P3 auth audit, P3 ES security, P3 token refresh | HIPAA/compliance baseline; audit trail for access |
+| **4 — API completeness** | P2 validators, P4 patient update, P4 pagination, P4 missing endpoints, P4 delete ops, P4 config extraction | Admin UI and integration teams unblocked |
+| **5 — Observability & testing** | P5 request metrics, P5 background metrics, P5 concurrent tests, P4 FHIR read | Incidents diagnosable; regression safety net for Batch 1 fixes |
+
+---
+
+## Testing strategy (cross-cutting)
+
+For each fix, add or extend tests in `VigilCareClinicalAPI.Tests/`:
+
+- **Concurrency tests** (Batch 1): Parallel patient registration, parallel bundle creation, parallel observation ingest with same idempotency key.
+- **Transaction rollback tests** (Batch 1): Order result failure rolls back bundle update; FHIR bundle entry failure rolls back all entries.
+- **Infrastructure resilience tests** (Batch 2): Consumer with poison pill message, outbox with simulated Kafka failure, startup with Redis unavailable.
+- **Security tests** (Batch 3): Timing-safe API key comparison, expired/revoked token rejection, authorization failure audit log entry.
+- **API contract tests** (Batch 4): New validators return 422 with correct error shapes, pagination parameters respected, new endpoints return expected status codes.
+- **Metrics verification tests** (Batch 5): Prometheus counter increments on consumer error, request histogram populated after API call.
+
+---
+
+## Out of scope (unless explicitly requested)
+
+- Full OpenTelemetry distributed tracing (P5 covers Prometheus metrics as interim).
+- Multi-tenancy or organization-scoped data isolation.
+- FHIR Subscription or WebSocket push for real-time updates.
+- HL7v2 ADT message support (current integration is FHIR-only).
+- Rate limiting on public-facing endpoints (API is internal-only today).
+- Database read replicas or CQRS pattern.
+- Kubernetes manifests, Helm charts, or CI/CD pipeline definitions.
+- SMART on FHIR authorization (OAuth2 scopes for EHR launch context).
+
+---
+
+## Success criteria
+
+When complete, the system should support:
+
+**Data Integrity (Part A)**
+- Concurrent patient registrations produce unique MRNs without 500 errors.
+- Concurrent SOFA_SEPSIS alerts for the same encounter create exactly one bundle.
+- Trend alerts match on exact observation code, not substring.
+- Order results and bundle compliance update atomically.
+- FHIR transaction bundles are all-or-nothing.
+
+**Infrastructure (Part B)**
+- Kafka topic loss requires losing 2+ brokers (RF=3).
+- Health checks report dependency status; orchestrators route around failures.
+- A malformed Kafka message is dead-lettered after 3 retries, not retried forever.
+- Outbox events have bounded retry with backoff and dead-letter.
+- Startup survives transient Redis outage.
+- Data lake writes are complete or explicitly failed — never silently dropped.
+
+**Security (Part D)**
+- FHIR API keys can be rotated without downtime.
+- JWT misconfiguration fails at startup, not at first request.
+- Authorization failures are logged and auditable.
+- Elasticsearch requires authentication.
+
+**API (Part C)**
+- All request types have input validation with 422 error responses.
+- Patient demographics are updatable via API.
+- Pagination is consistent (1-based, sortable) across all list endpoints.
+- Clinical dashboards have API access to sepsis bundles, qSOFA history, and reconciliation alerts.
+
+**Observability (Part E)**
+- HTTP request latency is measurable via Prometheus histograms.
+- Background service failures are countable and alertable.
+- Concurrent operation tests provide regression safety for P0 fixes.
@@ -46,10 +46,42 @@ services:
    networks:
      - vigilcare

+  prometheus:
+    image: prom/prometheus:v2.51.0
+    ports:
+      - "9095:9090"
+    volumes:
+      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
+      - prometheus_data:/prometheus
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+    command:
+      - "--config.file=/etc/prometheus/prometheus.yml"
+      - "--storage.tsdb.retention.time=30d"
+    networks:
+      - vigilcare
+
+  grafana:
+    image: grafana/grafana:10.4.0
+    ports:
+      - "3013:3000"
+    environment:
+      GF_SECURITY_ADMIN_USER: admin
+      GF_SECURITY_ADMIN_PASSWORD: admin
+      GF_USERS_ALLOW_SIGN_UP: "false"
+    volumes:
+      - grafana_data:/var/lib/grafana
+    depends_on:
+      - prometheus
+    networks:
+      - vigilcare
+
 volumes:
  pg_data:
  seq_data:
  minio_data:
+  prometheus_data:
+  grafana_data:

 networks:
  vigilcare:
@@ -0,0 +1,14 @@
+global:
+  scrape_interval: 15s
+  evaluation_interval: 15s
+
+scrape_configs:
+  - job_name: "vigilcare-records-api"
+    metrics_path: "/metrics"
+    static_configs:
+      - targets: ["host.docker.internal:5217"]
+        labels:
+          app: "vigilcare-records"
+          environment: "development"
+    scrape_interval: 10s
+    scrape_timeout: 5s
@@ -0,0 +1,800 @@
+#!/usr/bin/env bash
+# Runs Phase 8 verification checks from docs/plans/phase-8-plan.md.
+#
+# Covers Prometheus metrics, MetricsCollectorService gauges, rejection counter,
+# promotion duration histogram, work queue overview, cursor-paginated audit trail,
+# promotion retry schema, and the Prometheus/Grafana Docker stack.
+#
+# Prerequisites:
+#   docker compose up -d          (PostgreSQL, Redis, MinIO, Prometheus, Grafana)
+#   dotnet ef database update --project VigilCareRecordsAPI
+#   dotnet run --project VigilCareRecordsAPI
+#   Phase 1–7 seed data (intake1, entry1, verifier1, approver1, admin1)
+#
+# Environment overrides:
+#   VIGILCARE_API_URL                  default: http://localhost:5217
+#   VIGILCARE_PROMETHEUS_URL           default: http://localhost:9095
+#   VIGILCARE_GRAFANA_URL              default: http://localhost:3013
+#   VIGILCARE_COMPOSE_FILE             default: <repo>/docker-compose.yml
+#   VIGILCARE_PG_HOST                  default: localhost
+#   VIGILCARE_PG_PORT                  default: 5437
+#   VIGILCARE_PG_DB                    default: vigilcare_records
+#   VIGILCARE_PG_USER                  default: postgres
+#   VIGILCARE_PG_PASSWORD              default: password
+#   VIGILCARE_SKIP_DB_CHECKS           set to 1 to skip PostgreSQL assertions
+#   VIGILCARE_SKIP_MONITORING_CHECKS   set to 1 to skip Prometheus/Grafana checks
+#   VIGILCARE_METRICS_COLLECTOR_WAIT   default: 35 (seconds; MetricsCollectorService interval is 30s)
+#   VIGILCARE_RECORDED_AT              default: 2026-06-27T10:00:00Z
+
+set -uo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
+FIXTURE_PDF="$SCRIPT_DIR/fixtures/test-scan.pdf"
+
+API_URL="${VIGILCARE_API_URL:-http://localhost:5217}"
+PROMETHEUS_URL="${VIGILCARE_PROMETHEUS_URL:-http://localhost:9095}"
+GRAFANA_URL="${VIGILCARE_GRAFANA_URL:-http://localhost:3013}"
+COMPOSE_FILE="${VIGILCARE_COMPOSE_FILE:-$REPO_ROOT/docker-compose.yml}"
+COMPOSE=(docker compose -f "$COMPOSE_FILE")
+PG_HOST="${VIGILCARE_PG_HOST:-localhost}"
+PG_PORT="${VIGILCARE_PG_PORT:-5437}"
+PG_DB="${VIGILCARE_PG_DB:-vigilcare_records}"
+PG_USER="${VIGILCARE_PG_USER:-postgres}"
+PG_PASSWORD="${VIGILCARE_PG_PASSWORD:-password}"
+SKIP_DB_CHECKS="${VIGILCARE_SKIP_DB_CHECKS:-0}"
+SKIP_MONITORING_CHECKS="${VIGILCARE_SKIP_MONITORING_CHECKS:-0}"
+METRICS_COLLECTOR_WAIT="${VIGILCARE_METRICS_COLLECTOR_WAIT:-35}"
+RECORDED_AT="${VIGILCARE_RECORDED_AT:-2026-06-27T10:00:00Z}"
+RECORDED_AT_BP="${VIGILCARE_RECORDED_AT_BP:-2026-06-27T10:05:00Z}"
+
+PASS_COUNT=0
+FAIL_COUNT=0
+FAILED_TESTS=()
+
+ALL_BATCH_STATUSES=(
+  UPLOADED
+  IN_ENTRY
+  PENDING_VERIFICATION
+  REJECTED
+  VERIFIED
+  AWAITING_CLINICAL_APPROVAL
+  APPROVED
+  PROMOTED
+)
+
+log() {
+  printf '%s\n' "$*"
+}
+
+section() {
+  log ""
+  log "== $1 =="
+}
+
+pass() {
+  PASS_COUNT=$((PASS_COUNT + 1))
+  log "  PASS: $1"
+}
+
+fail() {
+  FAIL_COUNT=$((FAIL_COUNT + 1))
+  FAILED_TESTS+=("$1")
+  log "  FAIL: $1"
+}
+
+require_cmd() {
+  local cmd="$1"
+  if ! command -v "$cmd" >/dev/null 2>&1; then
+    log "ERROR: required command not found: $cmd"
+    exit 1
+  fi
+}
+
+ensure_fixture_pdf() {
+  if [[ -f "$FIXTURE_PDF" ]]; then
+    return 0
+  fi
+  mkdir -p "$(dirname "$FIXTURE_PDF")"
+  cat >"$FIXTURE_PDF" <<'EOF'
+%PDF-1.0
+1 0 obj<</Type/Catalog/Pages 2 0 R>>endobj 2 0 obj<</Type/Pages/Kids[3 0 R]/Count 1>>endobj 3 0 obj<</Type/Page/MediaBox[0 0 612 792]/Parent 2 0 R/Resources<<>>>>endobj
+xref
+0 4
+0000000000 65535 f 
+0000000009 00000 n 
+0000000058 00000 n 
+0000000115 00000 n 
+trailer<</Size 4/Root 1 0 R>>
+startxref
+206
+%%EOF
+EOF
+}
+
+new_uuid() {
+  if command -v uuidgen >/dev/null 2>&1; then
+    uuidgen
+  else
+    cat /proc/sys/kernel/random/uuid
+  fi
+}
+
+new_idempotency_key() {
+  printf 'phase8-%s' "$(new_uuid)"
+}
+
+compose_service_running() {
+  local service="$1"
+  "${COMPOSE[@]}" ps --status running --services 2>/dev/null | grep -qx "$service"
+}
+
+http_code() {
+  curl -sS -o /dev/null -w '%{http_code}' "$@"
+}
+
+json_post() {
+  local url="$1"
+  local body="$2"
+  local token="${3:-}"
+  if [[ -n "$token" ]]; then
+    curl -sS -X POST "$url" \
+      -H "Authorization: Bearer $token" \
+      -H 'Content-Type: application/json' \
+      -d "$body"
+  else
+    curl -sS -X POST "$url" \
+      -H 'Content-Type: application/json' \
+      -d "$body"
+  fi
+}
+
+json_put() {
+  local url="$1"
+  local body="$2"
+  local token="$3"
+  curl -sS -X PUT "$url" \
+    -H "Authorization: Bearer $token" \
+    -H 'Content-Type: application/json' \
+    -d "$body"
+}
+
+login() {
+  local username="$1"
+  local password="${2:-password}"
+  json_post "$API_URL/api/v1/auth/login" \
+    "{\"username\":\"$username\",\"password\":\"$password\"}"
+}
+
+extract_data_field() {
+  local json="$1"
+  local field="$2"
+  jq -er ".data.$field // empty" <<<"$json"
+}
+
+extract_error_code() {
+  local json="$1"
+  jq -er '.error.code // empty' <<<"$json" 2>/dev/null || true
+}
+
+assert_api_reachable() {
+  local code
+  code="$(http_code "$API_URL/swagger/index.html" || true)"
+  if [[ "$code" != "200" ]]; then
+    log "ERROR: API not reachable at $API_URL (HTTP $code)."
+    log "Start infrastructure with: docker compose up -d"
+    log "Apply migrations with: dotnet ef database update --project VigilCareRecordsAPI"
+    log "Start API with: dotnet run --project VigilCareRecordsAPI"
+    exit 1
+  fi
+}
+
+fetch_metrics() {
+  curl -sS "$API_URL/metrics"
+}
+
+metric_gauge_status() {
+  local status="$1"
+  local line value
+  line="$(fetch_metrics | grep -E "digitization_batches_by_status\\{status=\"${status}\"\\}" | head -1 || true)"
+  value="$(awk '{print $2}' <<<"$line")"
+  if [[ -z "$value" ]]; then
+    printf '0'
+  else
+    printf '%s' "$value"
+  fi
+}
+
+metric_counter_rejection() {
+  local category="$1"
+  local line value
+  line="$(fetch_metrics | grep -E "digitization_rejection_total\\{reason_category=\"${category}\"\\}" | head -1 || true)"
+  value="$(awk '{print $2}' <<<"$line")"
+  if [[ -z "$value" ]]; then
+    printf '0'
+  else
+    printf '%s' "$value"
+  fi
+}
+
+metric_histogram_count() {
+  local line value
+  line="$(fetch_metrics | grep -E '^digitization_promotion_duration_seconds_count ' | head -1 || true)"
+  value="$(awk '{print $2}' <<<"$line")"
+  if [[ -z "$value" ]]; then
+    printf '0'
+  else
+    printf '%s' "$value"
+  fi
+}
+
+psql_available() {
+  [[ "$SKIP_DB_CHECKS" == "1" ]] && return 1
+  compose_service_running postgres && return 0
+  command -v psql >/dev/null 2>&1 && return 0
+  return 1
+}
+
+psql_query() {
+  if [[ "$SKIP_DB_CHECKS" == "1" ]]; then
+    return 1
+  fi
+  if compose_service_running postgres; then
+    "${COMPOSE[@]}" exec -T postgres \
+      psql -U "$PG_USER" -d "$PG_DB" -Atqc "$1"
+  elif command -v psql >/dev/null 2>&1; then
+    PGPASSWORD="$PG_PASSWORD" psql -h "$PG_HOST" -p "$PG_PORT" -U "$PG_USER" -d "$PG_DB" -Atqc "$1"
+  else
+    return 1
+  fi
+}
+
+upload_batch() {
+  local token="$1"
+  local batch_type="${2:-VITALS_SHEET}"
+  local track="${3:-BACKFILL}"
+
+  curl -sS -X POST "$API_URL/api/v1/digitization-batches" \
+    -H "Authorization: Bearer $token" \
+    -F "file=@${FIXTURE_PDF};type=application/pdf" \
+    -F "batchType=$batch_type" \
+    -F "track=$track"
+}
+
+assign_batch() {
+  local token="$1"
+  local batch_id="$2"
+  local entry_clerk_id="$3"
+
+  curl -sS -X PATCH "$API_URL/api/v1/digitization-batches/$batch_id/assign" \
+    -H "Authorization: Bearer $token" \
+    -H 'Content-Type: application/json' \
+    -d "{\"entryClerkUserId\":\"$entry_clerk_id\"}"
+}
+
+verify_batch() {
+  local token="$1"
+  local batch_id="$2"
+  local body="$3"
+  json_post "$API_URL/api/v1/digitization-batches/$batch_id/verify" "$body" "$token"
+}
+
+reject_batch() {
+  local token="$1"
+  local batch_id="$2"
+  local reason="$3"
+  json_post "$API_URL/api/v1/digitization-batches/$batch_id/reject" \
+    "{\"reason\":\"$reason\"}" "$token"
+}
+
+approve_batch() {
+  local token="$1"
+  local batch_id="$2"
+  local idempotency_key="$3"
+  local body='{"enableRetroactiveAlerts":false}'
+  if [[ -n "${4:-}" ]]; then
+    body="$4"
+  fi
+
+  curl -sS -X POST "$API_URL/api/v1/digitization-batches/$batch_id/approve" \
+    -H "Authorization: Bearer $token" \
+    -H 'Content-Type: application/json' \
+    -H "Idempotency-Key: $idempotency_key" \
+    -d "$body"
+}
+
+create_assigned_batch() {
+  local intake_token="$1"
+  local upload_json batch_id entry_id
+
+  upload_json="$(upload_batch "$intake_token")"
+  if [[ "$(jq -er '.success' <<<"$upload_json")" != "true" ]]; then
+    return 1
+  fi
+
+  batch_id="$(extract_data_field "$upload_json" id)"
+  entry_id="$(extract_data_field "$(login entry1)" userId)"
+  assign_batch "$intake_token" "$batch_id" "$entry_id" >/dev/null
+  printf '%s' "$batch_id"
+}
+
+create_pending_vitals_batch() {
+  local intake_token="$1"
+  local entry_token batch_id submit_code
+
+  batch_id="$(create_assigned_batch "$intake_token")" || return 1
+  entry_token="$(extract_data_field "$(login entry1)" token)"
+
+  json_put \
+    "$API_URL/api/v1/digitization-batches/$batch_id/draft/patient" \
+    '{"fullName":"Phase8 Verify Patient","dateOfBirth":"1985-03-15","sex":"M","noKnownAllergies":true,"noActiveMedications":true}' \
+    "$entry_token" >/dev/null
+
+  json_put \
+    "$API_URL/api/v1/digitization-batches/$batch_id/draft/encounter" \
+    '{"admissionDate":"2025-01-01T08:00:00Z","department":"ICU","admissionReason":"Phase 8 verification"}' \
+    "$entry_token" >/dev/null
+
+  json_post \
+    "$API_URL/api/v1/digitization-batches/$batch_id/draft/observations" \
+    "{\"observationCode\":\"HEART_RATE\",\"value\":88,\"unit\":\"bpm\",\"recordedAt\":\"$RECORDED_AT\"}" \
+    "$entry_token" >/dev/null
+
+  submit_code="$(http_code -X POST \
+    "$API_URL/api/v1/digitization-batches/$batch_id/submit-for-verification" \
+    -H "Authorization: Bearer $entry_token")"
+  if [[ "$submit_code" != "200" ]]; then
+    return 1
+  fi
+
+  printf '%s' "$batch_id"
+}
+
+create_batch_ready_for_approval() {
+  local intake_token="$1"
+  local entry_token verifier_token batch_id entry_id verify_json verify_status submit_code
+
+  batch_id="$(create_assigned_batch "$intake_token")" || return 1
+  entry_id="$(extract_data_field "$(login entry1)" userId)"
+  entry_token="$(extract_data_field "$(login entry1)" token)"
+
+  json_put \
+    "$API_URL/api/v1/digitization-batches/$batch_id/draft/patient" \
+    '{"fullName":"Phase8 Promote Patient","dateOfBirth":"1990-05-15","sex":"M","bloodType":"A+","noKnownAllergies":true}' \
+    "$entry_token" >/dev/null
+
+  json_put \
+    "$API_URL/api/v1/digitization-batches/$batch_id/draft/encounter" \
+    '{"admissionDate":"2024-01-15T08:00:00Z","department":"General Medicine","roomBed":"301-A","admissionReason":"Routine checkup","status":"active"}' \
+    "$entry_token" >/dev/null
+
+  json_post \
+    "$API_URL/api/v1/digitization-batches/$batch_id/draft/observations" \
+    "{\"observationCode\":\"HR\",\"value\":88,\"unit\":\"bpm\",\"recordedAt\":\"$RECORDED_AT\"}" \
+    "$entry_token" >/dev/null
+
+  json_post \
+    "$API_URL/api/v1/digitization-batches/$batch_id/draft/observations" \
+    "{\"observationCode\":\"TEMP\",\"value\":37.2,\"unit\":\"C\",\"recordedAt\":\"$RECORDED_AT\"}" \
+    "$entry_token" >/dev/null
+
+  json_post \
+    "$API_URL/api/v1/digitization-batches/$batch_id/draft/observations" \
+    "{\"observationCode\":\"BP_SYS\",\"value\":120,\"unit\":\"mmHg\",\"recordedAt\":\"$RECORDED_AT_BP\"}" \
+    "$entry_token" >/dev/null
+
+  submit_code="$(http_code -X POST \
+    "$API_URL/api/v1/digitization-batches/$batch_id/submit-for-verification" \
+    -H "Authorization: Bearer $entry_token")"
+  if [[ "$submit_code" != "200" ]]; then
+    return 1
+  fi
+
+  verifier_token="$(extract_data_field "$(login verifier1)" token)"
+  verify_json="$(verify_batch "$verifier_token" "$batch_id" \
+    '{"fieldChecks":[{"fieldName":"patient.fullName","status":"ok","note":null},{"fieldName":"observation.heartRate","status":"ok","note":null}],"passed":true}')"
+  verify_status="$(extract_data_field "$verify_json" status)"
+
+  if [[ "$(jq -er '.success' <<<"$verify_json")" != "true" || "$verify_status" != "AWAITING_CLINICAL_APPROVAL" ]]; then
+    return 1
+  fi
+
+  printf '%s' "$batch_id"
+}
+
+test_metrics_endpoint() {
+  section "1. Prometheus metrics endpoint"
+
+  local metrics
+  metrics="$(fetch_metrics)"
+
+  if grep -q '^# HELP ' <<<"$metrics" && grep -q '^# TYPE ' <<<"$metrics"; then
+    pass "/metrics serves Prometheus exposition format"
+  else
+    fail "/metrics serves Prometheus exposition format"
+  fi
+
+  local metric
+  for metric in \
+    digitization_batches_by_status \
+    digitization_promotion_duration_seconds \
+    digitization_rejection_total \
+    digitization_queue_age_seconds \
+    http_request_duration_seconds; do
+    if grep -q "$metric" <<<"$metrics"; then
+      pass "metric registered: $metric"
+    else
+      fail "metric registered: $metric"
+    fi
+  done
+
+  local missing=0 status
+  for status in "${ALL_BATCH_STATUSES[@]}"; do
+    if ! grep -q "digitization_batches_by_status{status=\"${status}\"}" <<<"$metrics"; then
+      missing=$((missing + 1))
+    fi
+  done
+  if [[ "$missing" -eq 0 ]]; then
+    pass "digitization_batches_by_status exposes all 8 status labels"
+  else
+    fail "digitization_batches_by_status exposes all 8 status labels ($missing missing)"
+  fi
+
+  if grep -q 'digitization_rejection_total{reason_category="verification_failed"}' <<<"$metrics" \
+    && grep -q 'digitization_rejection_total{reason_category="clinical_rejected"}' <<<"$metrics"; then
+    pass "digitization_rejection_total exposes both reason_category labels"
+  else
+    fail "digitization_rejection_total exposes both reason_category labels"
+  fi
+}
+
+test_gauge_updates_after_upload() {
+  section "2. Gauge metrics update after creating a batch"
+
+  local intake_token before after upload_json
+
+  intake_token="$(extract_data_field "$(login intake1)" token)"
+  before="$(metric_gauge_status UPLOADED)"
+
+  upload_json="$(upload_batch "$intake_token")"
+  if [[ "$(jq -er '.success' <<<"$upload_json")" != "true" ]]; then
+    fail "upload batch for gauge test"
+    return
+  fi
+  pass "uploaded batch for gauge test"
+
+  log "  waiting ${METRICS_COLLECTOR_WAIT}s for MetricsCollectorService..."
+  sleep "$METRICS_COLLECTOR_WAIT"
+
+  after="$(metric_gauge_status UPLOADED)"
+  if awk -v before="$before" -v after="$after" 'BEGIN { exit (after > before) ? 0 : 1 }'; then
+    pass "digitization_batches_by_status UPLOADED increased ($before -> $after)"
+  else
+    fail "digitization_batches_by_status UPLOADED increased ($before -> $after)"
+  fi
+}
+
+test_rejection_counter() {
+  section "3. Rejection counter increments"
+
+  local intake_token verifier_token batch_id before after verify_json
+
+  intake_token="$(extract_data_field "$(login intake1)" token)"
+  batch_id="$(create_pending_vitals_batch "$intake_token")" || {
+    fail "setup pending batch for rejection counter"
+    return
+  }
+
+  before="$(metric_counter_rejection verification_failed)"
+  verifier_token="$(extract_data_field "$(login verifier1)" token)"
+
+  verify_json="$(verify_batch "$verifier_token" "$batch_id" \
+    '{"fieldChecks":[{"fieldName":"patient.fullName","status":"error","note":"Name mismatch"}],"passed":false}')"
+  if [[ "$(jq -er '.success' <<<"$verify_json")" != "true" ]]; then
+    fail "verify with passed=false returns success envelope"
+    return
+  fi
+  pass "verification failed via VerifyAsync (Passed=false)"
+
+  after="$(metric_counter_rejection verification_failed)"
+  if awk -v before="$before" -v after="$after" 'BEGIN { exit (after > before) ? 0 : 1 }'; then
+    pass "digitization_rejection_total verification_failed incremented ($before -> $after)"
+  else
+    fail "digitization_rejection_total verification_failed incremented ($before -> $after)"
+  fi
+}
+
+test_promotion_histogram() {
+  section "4. Promotion duration histogram"
+
+  local intake_token approver_token batch_id before after approve_json
+
+  before="$(metric_histogram_count)"
+  intake_token="$(extract_data_field "$(login intake1)" token)"
+  batch_id="$(create_batch_ready_for_approval "$intake_token")" || {
+    fail "setup batch ready for approval"
+    return
+  }
+
+  approver_token="$(extract_data_field "$(login approver1)" token)"
+  approve_json="$(approve_batch "$approver_token" "$batch_id" "$(new_idempotency_key)" \
+    '{"enableRetroactiveAlerts":false}')"
+
+  if [[ "$(jq -er '.success' <<<"$approve_json")" != "true" ]]; then
+    fail "approve batch for promotion histogram"
+    return
+  fi
+  pass "batch promoted via ApproveAndPromoteAsync"
+
+  after="$(metric_histogram_count)"
+  if awk -v before="$before" -v after="$after" 'BEGIN { exit (after > before) ? 0 : 1 }'; then
+    pass "digitization_promotion_duration_seconds_count increased ($before -> $after)"
+  else
+    fail "digitization_promotion_duration_seconds_count increased ($before -> $after)"
+  fi
+
+  if grep -q '^digitization_promotion_duration_seconds_sum ' <<<"$(fetch_metrics)"; then
+    pass "digitization_promotion_duration_seconds_sum present after promotion"
+  else
+    fail "digitization_promotion_duration_seconds_sum present after promotion"
+  fi
+}
+
+test_work_queue_overview() {
+  section "5. Work queue overview endpoint"
+
+  local admin_token entry_token overview_json code missing=0 status
+
+  admin_token="$(extract_data_field "$(login admin1)" token)"
+  overview_json="$(curl -sS "$API_URL/api/v1/work-queue/overview" \
+    -H "Authorization: Bearer $admin_token")"
+
+  if [[ "$(jq -er '.success' <<<"$overview_json")" == "true" ]]; then
+    pass "GET /work-queue/overview returns success for administrator"
+  else
+    fail "GET /work-queue/overview returns success for administrator"
+    return
+  fi
+
+  for status in "${ALL_BATCH_STATUSES[@]}"; do
+    if [[ "$(jq -er --arg s "$status" '.data.statusCounts[$s] | type' <<<"$overview_json")" != "number" ]]; then
+      missing=$((missing + 1))
+    fi
+  done
+  if [[ "$missing" -eq 0 ]]; then
+    pass "overview statusCounts includes all 8 statuses"
+  else
+    fail "overview statusCounts includes all 8 statuses ($missing missing)"
+  fi
+
+  if jq -er '.data.rejectRate | type' <<<"$overview_json" | grep -qx number; then
+    pass "overview rejectRate is numeric"
+  else
+    fail "overview rejectRate is numeric"
+  fi
+
+  if jq -er '.data.averageTimeInQueueMinutes | type' <<<"$overview_json" | grep -qx number \
+    && jq -er '.data.oldestPendingVerificationMinutes | type' <<<"$overview_json" | grep -qx number; then
+    pass "overview queue age fields are numeric"
+  else
+    fail "overview queue age fields are numeric"
+  fi
+
+  entry_token="$(extract_data_field "$(login entry1)" token)"
+  code="$(http_code "$API_URL/api/v1/work-queue/overview" \
+    -H "Authorization: Bearer $entry_token")"
+  if [[ "$code" == "403" ]]; then
+    pass "GET /work-queue/overview returns 403 for non-administrator"
+  else
+    fail "GET /work-queue/overview returns 403 for non-administrator (HTTP $code)"
+  fi
+}
+
+test_audit_trail_events() {
+  section "6. Cursor-paginated audit trail"
+
+  local admin_token intake_token batch_id page1 page2 code cursor encoded_cursor
+
+  admin_token="$(extract_data_field "$(login admin1)" token)"
+  intake_token="$(extract_data_field "$(login intake1)" token)"
+  batch_id="$(create_batch_ready_for_approval "$intake_token")" || {
+    fail "setup batch with multiple audit events"
+    return
+  }
+
+  page1="$(curl -sS "$API_URL/api/v1/digitization-batches/$batch_id/events?pageSize=2" \
+    -H "Authorization: Bearer $admin_token")"
+
+  if [[ "$(jq -er '.success' <<<"$page1")" != "true" ]]; then
+    fail "GET /events returns success envelope"
+    return
+  fi
+  pass "GET /events returns success envelope"
+
+  if [[ "$(jq -er '.data.items | length' <<<"$page1")" -ge 1 ]]; then
+    pass "events page contains at least one item"
+  else
+    fail "events page contains at least one item"
+  fi
+
+  if jq -er '.data.items[0] | has("actorUsername") and has("actorFullName") and has("eventType")' \
+    <<<"$page1" | grep -qx true; then
+    pass "event items include actorUsername and actorFullName"
+  else
+    fail "event items include actorUsername and actorFullName"
+  fi
+
+  if [[ "$(jq -er '.data.hasMore' <<<"$page1")" == "true" ]]; then
+    cursor="$(extract_data_field "$page1" nextCursor)"
+    encoded_cursor="$(python3 -c 'import urllib.parse,sys; print(urllib.parse.quote(sys.argv[1], safe=""))' "$cursor")"
+    page2="$(curl -sS "$API_URL/api/v1/digitization-batches/$batch_id/events?pageSize=2&after=$encoded_cursor" \
+      -H "Authorization: Bearer $admin_token")"
+    if [[ "$(jq -er '.success' <<<"$page2")" == "true" \
+      && "$(jq -er '.data.items | length' <<<"$page2")" -ge 1 ]]; then
+      pass "cursor pagination returns a second page"
+    else
+      fail "cursor pagination returns a second page"
+    fi
+  else
+    log "  SKIP: cursor second-page check (batch has <=2 events)"
+  fi
+
+  code="$(http_code "$API_URL/api/v1/digitization-batches/$batch_id/events?after=not-a-date" \
+    -H "Authorization: Bearer $admin_token")"
+  if [[ "$code" == "400" ]]; then
+    pass "invalid cursor returns 400"
+  else
+    fail "invalid cursor returns 400 (HTTP $code)"
+  fi
+
+  code="$(http_code "$API_URL/api/v1/digitization-batches/00000000-0000-0000-0000-000000000000/events" \
+    -H "Authorization: Bearer $admin_token")"
+  if [[ "$code" == "404" ]]; then
+    pass "missing batch returns 404"
+  else
+    fail "missing batch returns 404 (HTTP $code)"
+  fi
+}
+
+test_promotion_retry_schema() {
+  section "7. Promotion retry infrastructure"
+
+  if ! psql_available; then
+    log "  SKIP: promotion_attempts table check (postgres not reachable)"
+    return
+  fi
+
+  local table_exists
+  table_exists="$(psql_query "
+    SELECT EXISTS (
+      SELECT 1
+      FROM information_schema.tables
+      WHERE table_schema = 'public'
+        AND table_name = 'promotion_attempts'
+    );
+  ")"
+
+  if [[ "$table_exists" == "t" ]]; then
+    pass "promotion_attempts table exists"
+  else
+    fail "promotion_attempts table exists"
+  fi
+
+  local index_count
+  index_count="$(psql_query "
+    SELECT count(*)
+    FROM pg_indexes
+    WHERE tablename = 'promotion_attempts'
+      AND indexname IN (
+        'ix_promotion_attempts_batch_attempt',
+        'ix_promotion_attempts_pending_retry'
+      );
+  ")"
+
+  if [[ "$index_count" == "2" ]]; then
+    pass "promotion_attempts has batch/attempt and pending-retry indexes"
+  else
+    fail "promotion_attempts has batch/attempt and pending-retry indexes (got $index_count)"
+  fi
+
+  log "  NOTE: full PromotionRetryService retry flow requires simulating a deferred promotion failure (manual test in phase-8 plan section 7)."
+}
+
+test_monitoring_stack() {
+  section "8. Prometheus and Grafana stack"
+
+  if [[ "$SKIP_MONITORING_CHECKS" == "1" ]]; then
+    log "  SKIP: VIGILCARE_SKIP_MONITORING_CHECKS=1"
+    return
+  fi
+
+  if ! compose_service_running prometheus; then
+    log "  SKIP: prometheus container not running (docker compose up -d prometheus grafana)"
+    return
+  fi
+
+  local prom_health grafana_health target_health query_status
+
+  prom_health="$(http_code "$PROMETHEUS_URL/-/healthy" || true)"
+  if [[ "$prom_health" == "200" ]]; then
+    pass "Prometheus healthy at $PROMETHEUS_URL"
+  else
+    fail "Prometheus healthy at $PROMETHEUS_URL (HTTP $prom_health)"
+  fi
+
+  if compose_service_running grafana; then
+    grafana_health="$(http_code "$GRAFANA_URL/api/health" || true)"
+    if [[ "$grafana_health" == "200" ]]; then
+      pass "Grafana healthy at $GRAFANA_URL"
+    else
+      fail "Grafana healthy at $GRAFANA_URL (HTTP $grafana_health)"
+    fi
+  else
+    log "  SKIP: grafana container not running"
+  fi
+
+  target_health="$(curl -sS "$PROMETHEUS_URL/api/v1/targets" | jq -er '.data.activeTargets[0].health' 2>/dev/null || true)"
+  if [[ "$target_health" == "up" ]]; then
+    pass "Prometheus scrape target is up"
+  elif [[ "$target_health" == "down" ]]; then
+    fail "Prometheus scrape target is up (currently down — is the API running on port 5217?)"
+  else
+    fail "Prometheus scrape target health could not be determined"
+  fi
+
+  query_status="$(curl -sS "$PROMETHEUS_URL/api/v1/query?query=digitization_batches_by_status" | jq -er '.status' 2>/dev/null || true)"
+  if [[ "$query_status" == "success" ]]; then
+    pass "Prometheus query API returns digitization_batches_by_status"
+  else
+    fail "Prometheus query API returns digitization_batches_by_status"
+  fi
+
+  if [[ -f "$REPO_ROOT/prometheus.yml" ]]; then
+    if grep -q 'host.docker.internal:5217' "$REPO_ROOT/prometheus.yml" \
+      && grep -q 'vigilcare-records-api' "$REPO_ROOT/prometheus.yml"; then
+      pass "prometheus.yml targets host.docker.internal:5217"
+    else
+      fail "prometheus.yml targets host.docker.internal:5217"
+    fi
+  else
+    fail "prometheus.yml exists in repo root"
+  fi
+}
+
+main() {
+  require_cmd curl
+  require_cmd jq
+  require_cmd docker
+  require_cmd python3
+  ensure_fixture_pdf
+
+  log "VigilCare Records — Phase 8 verification"
+  log "API: $API_URL"
+  log "Prometheus: $PROMETHEUS_URL"
+  log "Grafana: $GRAFANA_URL"
+
+  assert_api_reachable
+
+  test_metrics_endpoint
+  test_gauge_updates_after_upload
+  test_rejection_counter
+  test_promotion_histogram
+  test_work_queue_overview
+  test_audit_trail_events
+  test_promotion_retry_schema
+  test_monitoring_stack
+
+  log ""
+  log "Results: $PASS_COUNT passed, $FAIL_COUNT failed"
+  if (( FAIL_COUNT > 0 )); then
+    log "Failed checks:"
+    for item in "${FAILED_TESTS[@]}"; do
+      log "  - $item"
+    done
+    exit 1
+  fi
+
+  log "All Phase 8 verification checks passed."
+}
+
+main "$@"
@@ -131,8 +131,12 @@ export async function get<T>(url: string, params?: Record<string, unknown>): Pro
  return response.data
 }

-export async function post<T>(url: string, data?: unknown): Promise<ApiResponse<T>> {
-  const response = await apiClient.post<ApiResponse<T>>(url, data)
+export async function post<T>(
+  url: string,
+  data?: unknown,
+  headers?: Record<string, string>
+): Promise<ApiResponse<T>> {
+  const response = await apiClient.post<ApiResponse<T>>(url, data, { headers })
  return response.data
 }

@@ -183,7 +183,9 @@ async function assignBatch(batchId: string, entryClerkUserId: string): Promise<v
  }

  async function approveBatch(batchId: string): Promise<void> {
-    await post<void>(`digitization-batches/${batchId}/approve`, null)
+    await post<void>(`digitization-batches/${batchId}/approve`, null, {
+      'Idempotency-Key': crypto.randomUUID(),
+    })
  }

  return {