feature: Explainable Alerts

2026-06-25 00:25:31 +08:00
parent 279add1e45
commit 666d683d67
61 changed files with 9553 additions and 125 deletions
@@ -0,0 +1,561 @@
+# Guide 2: Prometheus + Grafana Monitoring Stack
+
+## What Are Prometheus and Grafana?
+
+**Prometheus** is a time-series database designed for monitoring. Instead of your application pushing data to Prometheus, Prometheus **pulls** (or "scrapes") data from your application on a schedule — typically every 15 seconds. Your application exposes a `/metrics` HTTP endpoint with the current values of all metrics, and Prometheus stores a timestamped history of those values.
+
+A **time-series** is just a series of numbers recorded over time — like "at 14:00 there were 3 pending events, at 14:15 there were 5, at 14:30 there were 0." Prometheus stores millions of these series efficiently and lets you query them with its built-in query language, PromQL.
+
+**Grafana** is a visualization tool that connects to Prometheus (and other data sources) and renders interactive dashboards — line graphs, gauges, stat panels, and alerts. Prometheus stores the numbers; Grafana makes them visual.
+
+**Why use them together?** Logging tells you _what happened_ ("observation ingested for encounter X"). Metrics tell you _how the system is performing right now_ ("we're ingesting 50 observations per second and the p99 latency is 23ms"). When something goes wrong, metrics tell you instantly — often before anyone notices a problem.
+
+---
+
+## Why Prometheus and Grafana in This Project?
+
+VigilCareClinical is a patient safety system. A growing outbox means delayed alerts. High Kafka consumer lag means scores aren't being computed. An unacknowledged critical alert means a patient needs attention. Prometheus collects these metrics every 15 seconds, and Grafana visualizes them on dashboards that clinicians and engineers can watch in real time.
+
+---
+
+## Architecture Overview
+
+```
+.NET API (host)                    Docker
+┌──────────────────┐     scrape    ┌─────────────┐    query    ┌─────────────┐
+│  /metrics        │ ◄──────────── │  Prometheus  │ ◄───────── │   Grafana   │
+│  (prometheus-net)│   every 15s   │  :9101       │            │   :3101     │
+└──────────────────┘               └─────────────┘            └─────────────┘
+        ▲
+        │ expose
+┌───────┴──────────┐
+│  ClinicalMetrics │  (singleton)
+│  4 Collectors    │  (BackgroundServices)
+└──────────────────┘
+```
+
+1. **prometheus-net** (a .NET library) exposes a `/metrics` endpoint on the API that outputs all metric values in Prometheus's text format
+2. **Prometheus** (running in Docker) scrapes that endpoint every 15 seconds and stores the values with timestamps
+3. **Grafana** (also in Docker) queries Prometheus and renders the data as charts and dashboards
+
+---
+
+## Step 1: The NuGet Package
+
+```xml
+<!-- VigilCareClinicalAPI.csproj -->
+<PackageReference Include="prometheus-net.AspNetCore" Version="8.2.1" />
+```
+
+This package provides:
+- The `Metrics` factory for creating counters, histograms, and gauges
+- The `MapMetrics()` extension method to expose the `/metrics` HTTP endpoint
+- ASP.NET Core middleware integration
+
+---
+
+## Step 2: Registering the Metrics Endpoint
+
+In `Program.cs`:
+
+```csharp
+// Register ClinicalMetrics as a singleton so all services share the same instances
+builder.Services.AddSingleton<ClinicalMetrics>();
+
+// ... later, after building the app:
+
+// Expose /metrics for Prometheus to scrape
+app.MapMetrics("/metrics");
+```
+
+That single `MapMetrics("/metrics")` call serves the Prometheus text exposition format at `http://localhost:5270/metrics`.
+
+---
+
+## Step 3: Defining Custom Metrics — ClinicalMetrics
+
+All custom metrics are defined in a single class, registered as a singleton. This ensures every service that injects `ClinicalMetrics` increments the same counter instances.
+
+### Counters
+
+Prometheus has three core metric types. The first is a **counter**.
+
+A counter only goes up — it never decreases. Think of it like the odometer on a car: it tracks the total distance driven since the car was built. A counter tracks things like "total observations ingested since the app started." The raw number isn't that useful on its own (who cares that we've ingested 50,000 total?), but Prometheus's `rate()` function converts it to "observations per second over the last minute" — that's actionable.
+
+```csharp
+public sealed class ClinicalMetrics
+{
+    // Labeled by observation_code and source
+    public readonly Counter ObservationsIngestedTotal = Metrics.CreateCounter(
+        "observations_ingested_total",
+        "Total observations ingested, labeled by observation code and source.",
+        labelNames: new[] { "observation_code", "source" });
+
+    // Labeled by alert_type and severity
+    public readonly Counter ClinicalAlertsTotal = Metrics.CreateCounter(
+        "clinical_alerts_total",
+        "Total clinical alerts generated, labeled by type and severity.",
+        labelNames: new[] { "alert_type", "severity" });
+
+    public readonly Counter News2ScoresTotal = Metrics.CreateCounter(
+        "news2_scores_total",
+        "Total NEWS2 scores computed, labeled by risk level.",
+        labelNames: new[] { "risk_level" });
+
+    public readonly Counter EscalationsTotal = Metrics.CreateCounter(
+        "escalations_total",
+        "Total alert escalations processed through the DLQ escalation path.");
+
+    public readonly Counter TrendAlertsTotal = Metrics.CreateCounter(
+        "trend_alerts_total",
+        "Total RAPID_DETERIORATION alerts generated.",
+        labelNames: new[] { "observation_code" });
+
+    public readonly Counter AlertSuppressionsTotal = Metrics.CreateCounter(
+        "alert_suppressions_total",
+        "Total alert suppression windows set after acknowledgment.",
+        labelNames: new[] { "alert_type" });
+
+    public readonly Counter QsofaDetectionsTotal = Metrics.CreateCounter(
+        "qsofa_detections_total",
+        "Total QSOFA_SCREEN alerts generated.");
+
+    public readonly Counter SepsisBundleComplianceTotal = Metrics.CreateCounter(
+        "sepsis_bundle_compliance_total",
+        "Sepsis bundle compliance outcomes.",
+        labelNames: new[] { "status" });
+
+    public readonly Counter GcsScoresTotal = Metrics.CreateCounter(
+        "gcs_scores_total",
+        "GCS scores computed, labeled by classification.",
+        labelNames: new[] { "classification" });
+
+    public readonly Counter SofaScoresTotal = Metrics.CreateCounter(
+        "sofa_scores_total",
+        "SOFA scores computed.",
+        labelNames: new[] { "has_delta_alert" });
+
+    public readonly Counter FhirIngestTotal = Metrics.CreateCounter(
+        "fhir_ingest_total",
+        "FHIR resource ingest operations.",
+        labelNames: new[] { "resource_type", "outcome" });
+
+    public readonly Counter FhirReadTotal = Metrics.CreateCounter(
+        "fhir_read_total",
+        "FHIR resource read/search operations.",
+        labelNames: new[] { "resource_type", "interaction", "outcome" });
+
+    public readonly Counter AuthorizationFailuresTotal = Metrics.CreateCounter(
+        "authorization_failures_total",
+        "Authorization failures by permission and role.",
+        labelNames: new[] { "permission", "role" });
+
+    public readonly Counter FhirMappingErrorsTotal = Metrics.CreateCounter(
+        "fhir_mapping_errors_total",
+        "FHIR mapping failures.",
+        labelNames: new[] { "reason" });
+
+    public readonly Counter PhiAccessLogsTotal = Metrics.CreateCounter(
+        "phi_access_logs_total",
+        "PHI access log entries written.",
+        labelNames: new[] { "access_type" });
+
+    public readonly Counter ClinicalSyncBatchesTotal = Metrics.CreateCounter(
+        "clinical_sync_batches_total",
+        "Sync batches processed.",
+        labelNames: new[] { "status" });
+}
+```
+
+**What are labels?** Labels are key-value tags attached to a metric that let you slice and filter the data. Instead of creating separate counters like `critical_heart_rate_alerts_total` and `warning_spo2_alerts_total`, you create one counter `clinical_alerts_total` with labels `alert_type` and `severity`. This single counter then lets you query:
+- Total alerts: `clinical_alerts_total`
+- Only critical threshold breaches: `clinical_alerts_total{severity="Critical"}`
+- Only SOFA sepsis alerts: `clinical_alerts_total{alert_type="SOFA_SEPSIS"}`
+
+### Histograms
+
+A **histogram** tracks the _distribution_ of values, not just a total count. For example, "how long does it take to ingest an observation?" — a counter can tell you how many were ingested, but a histogram tells you that 50% completed in under 10ms, 95% in under 50ms, and the slowest 1% took over 200ms.
+
+Histograms work by defining **buckets** — thresholds like 5ms, 10ms, 25ms, 50ms, etc. Each observation is counted into all buckets it falls below. Prometheus then stores `_bucket` (how many observations fell into each bucket), `_sum` (total time across all observations), and `_count` (total number of observations). From these, you can calculate percentiles using the `histogram_quantile()` function in PromQL.
+
+```csharp
+public readonly Histogram ObservationIngestDuration = Metrics.CreateHistogram(
+    "observation_ingest_duration_seconds",
+    "Ingest transaction duration from request receipt to COMMIT.",
+    new HistogramConfiguration
+    {
+        Buckets = new[] { 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0 }
+    });
+
+public readonly Histogram News2ScoringDuration = Metrics.CreateHistogram(
+    "news2_scoring_duration_seconds",
+    "Time to compute a NEWS2 score from Redis state.",
+    new HistogramConfiguration
+    {
+        Buckets = new[] { 0.001, 0.005, 0.01, 0.025, 0.05, 0.1 }
+    });
+
+public readonly Histogram TrendAnalysisDuration = Metrics.CreateHistogram(
+    "trend_analysis_duration_seconds",
+    "Time to evaluate trend for one observation.",
+    new HistogramConfiguration
+    {
+        Buckets = new[] { 0.001, 0.005, 0.01, 0.025, 0.05, 0.1 }
+    });
+
+public readonly Histogram SofaScoringDuration = Metrics.CreateHistogram(
+    "sofa_scoring_duration_seconds",
+    "SOFA scoring computation time.",
+    new HistogramConfiguration
+    {
+        Buckets = new[] { 0.001, 0.005, 0.01, 0.025, 0.05, 0.1 }
+    });
+
+public readonly Histogram ClinicalSyncBatchDuration = Metrics.CreateHistogram(
+    "clinical_sync_batch_duration_seconds",
+    "Batch processing duration.",
+    new HistogramConfiguration
+    {
+        Buckets = new[] { 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0 }
+    });
+```
+
+**Bucket selection matters**: The ingest histogram uses fine-grained buckets (5ms to 1s) because sub-second latency is critical for patient safety. The sync batch histogram uses coarser buckets (100ms to 10s) because batch operations are inherently slower.
+
+### Gauges
+
+A **gauge** is a value that can go up or down — like a thermometer or a fuel gauge. It represents the _current state_ of something: "right now there are 3 unacknowledged alerts" or "the outbox has 47 pending events." Unlike counters (which only increase), gauges are set to an absolute value by background services that periodically check the current state.
+
+```csharp
+public readonly Gauge AlertsUnacknowledgedGauge = Metrics.CreateGauge(
+    "alerts_unacknowledged_gauge",
+    "Count of open CRITICAL alerts older than 5 minutes with no acknowledgment.");
+
+public readonly Gauge KafkaConsumerLag = Metrics.CreateGauge(
+    "kafka_consumer_lag",
+    "Approximate consumer group lag in messages.",
+    labelNames: new[] { "consumer_group" });
+
+public readonly Gauge OutboxPendingEvents = Metrics.CreateGauge(
+    "outbox_pending_events",
+    "Count of outbox events not yet relayed to Kafka.");
+
+public readonly Gauge WardGatewaysOffline = Metrics.CreateGauge(
+    "ward_gateways_offline_gauge",
+    "Ward gateways with status OFFLINE or DEGRADED.",
+    labelNames: new[] { "site_code" });
+
+public readonly Gauge WardGatewayBufferDepth = Metrics.CreateGauge(
+    "ward_gateway_buffer_depth",
+    "Reported unsynced event count per gateway.",
+    labelNames: new[] { "gateway_code", "department" });
+
+public readonly Gauge AlertAcknowledgementRate = Metrics.CreateGauge(
+    "vigilcare_alert_acknowledgement_rate",
+    "Alert acknowledgement rate by type.",
+    labelNames: new[] { "alert_type" });
+
+public readonly Gauge AlertFalsePositiveRate = Metrics.CreateGauge(
+    "vigilcare_alert_false_positive_rate",
+    "Clinician-reported false positive rate by type.",
+    labelNames: new[] { "alert_type" });
+
+public readonly Gauge AlertUsefulRate = Metrics.CreateGauge(
+    "vigilcare_alert_useful_rate",
+    "Clinician-reported useful rate by type.",
+    labelNames: new[] { "alert_type" });
+
+public readonly Gauge AlertAvgAckSeconds = Metrics.CreateGauge(
+    "vigilcare_alert_avg_ack_seconds",
+    "Average seconds from trigger to acknowledgement by type.",
+    labelNames: new[] { "alert_type" });
+```
+
+---
+
+## Step 4: Using Metrics in Application Code
+
+### Inline Counters and Timers
+
+In `ObservationService`, the ingest path both times the transaction and increments counters:
+
+```csharp
+using var timer = _metrics.ObservationIngestDuration.NewTimer();
+
+// ... perform the ingest transaction ...
+
+_metrics.ObservationsIngestedTotal
+    .WithLabels(req.ObservationCode, req.Source.ToDbString())
+    .Inc();
+
+if (alert is not null)
+{
+    _metrics.ClinicalAlertsTotal
+        .WithLabels(alert.AlertType.ToDbString(), alert.Severity.ToDbString())
+        .Inc();
+}
+```
+
+**How does `NewTimer()` work?** In C#, a `using` block runs some cleanup code when the block exits. `NewTimer()` starts a stopwatch and the `using` block ensures it records the elapsed time into the histogram when the code leaves the block — even if an exception is thrown. This means you don't need to manually calculate timing.
+
+**What does `WithLabels()` do?** It selects which specific "bucket" of the counter to increment. A counter with labels is really a _family_ of counters — one for each unique label combination. `.WithLabels("HEART_RATE", "DEVICE")` increments the counter for heart rate observations from devices specifically.
+
+---
+
+## Step 5: Background Collectors
+
+Gauges can't be updated inline in request handlers because they represent _current state_ ("how many right now?"), not events ("one more just happened"). You need a background process that periodically checks the current state and updates the gauge.
+
+In .NET, a `BackgroundService` is a class that runs continuously in the background for the lifetime of the application. Four of these poll data sources on a timer and update gauge values.
+
+### AlertsUnacknowledgedCollector (every 30s)
+
+The most clinically significant metric. Queries PostgreSQL for CRITICAL alerts that have been open more than 5 minutes with no acknowledgment:
+
+```csharp
+protected override async Task ExecuteAsync(CancellationToken ct)
+{
+    using var timer = new PeriodicTimer(TimeSpan.FromSeconds(30));
+    while (await timer.WaitForNextTickAsync(ct))
+        await CollectAsync(ct);
+}
+
+private async Task CollectAsync(CancellationToken ct)
+{
+    await using var scope = _scopes.CreateAsyncScope();
+    var db     = scope.ServiceProvider.GetRequiredService<AppDbContext>();
+    var cutoff = DateTimeOffset.UtcNow - TimeSpan.FromMinutes(5);
+
+    var count = await db.ClinicalAlerts
+        .CountAsync(a => a.Severity   == AlertSeverity.Critical
+                      && a.Status      == AlertStatus.Open
+                      && a.TriggeredAt < cutoff, ct);
+
+    _metrics.AlertsUnacknowledgedGauge.Set(count);
+
+    if (count > 0)
+        _logger.LogWarning(
+            "[PATIENT-SAFETY] alerts_unacknowledged_gauge={Count}", count);
+}
+```
+
+It also logs a warning when the count is non-zero — this shows up in Seq as a structured log entry tagged `[PATIENT-SAFETY]`.
+
+### OutboxPendingCollector (every 30s)
+
+Counts outbox events not yet relayed to Kafka. A growing number means the relay is falling behind or Kafka is unreachable:
+
+```csharp
+var count = await db.OutboxEvents
+    .CountAsync(e => e.ProcessedAt == null, ct);
+_metrics.OutboxPendingEvents.Set(count);
+```
+
+### KafkaConsumerLagCollector (every 30s)
+
+Measures how far behind each Kafka consumer group is. **Consumer lag** is the number of messages that have been published but not yet processed. Think of it like a queue at a bank — lag is how many people are still waiting. This collector uses Kafka's admin API to check the lag for each consumer group without actually consuming any messages:
+
+```csharp
+private static readonly string[] Groups =
+{
+    "es-indexer",
+    "sepsis-engine",
+    "notification-publisher",
+    "data-lake-writer",
+};
+
+private async Task CollectGroupLagAsync(string groupId, CancellationToken ct)
+{
+    using var admin = new AdminClientBuilder(adminConfig).Build();
+
+    // Get committed offsets for this consumer group
+    var result = await admin.ListConsumerGroupOffsetsAsync(...);
+
+    // Query high watermarks with a temporary consumer
+    using var tempConsumer = new ConsumerBuilder<Ignore, Ignore>(config).Build();
+
+    long totalLag = 0;
+    foreach (var tpo in partitions)
+    {
+        var watermarks = tempConsumer.QueryWatermarkOffsets(
+            tpo.TopicPartition, TimeSpan.FromSeconds(5));
+        var lag = watermarks.High.Value - tpo.Offset.Value;
+        totalLag += Math.Max(0L, lag);
+    }
+
+    _metrics.KafkaConsumerLag.WithLabels(groupId).Set(totalLag);
+}
+```
+
+The lag is per consumer group. If `sepsis-engine` has high lag but `es-indexer` doesn't, the Grafana panel makes it immediately visible which pipeline is falling behind.
+
+### WardGatewayMetricsCollector (every 60s)
+
+Queries PostgreSQL for gateway status and buffer depth:
+
+```csharp
+var offlineBySite = await db.WardGateways
+    .Include(g => g.Site)
+    .Where(g => g.Status != GatewayStatus.Online)
+    .GroupBy(g => g.Site.SiteCode)
+    .Select(g => new { SiteCode = g.Key, Count = g.Count() })
+    .ToListAsync(ct);
+
+foreach (var row in offlineBySite)
+    _metrics.WardGatewaysOffline.WithLabels(row.SiteCode).Set(row.Count);
+
+var allGateways = await db.WardGateways.AsNoTracking().ToListAsync(ct);
+foreach (var g in allGateways)
+    _metrics.WardGatewayBufferDepth
+        .WithLabels(g.GatewayCode, g.Department)
+        .Set(g.ReportedBufferDepth);
+```
+
+All four collectors follow the same pattern:
+1. **Extend `BackgroundService`** — .NET's base class for long-running background work
+2. **Use `PeriodicTimer` for the poll loop** — fires a callback at a fixed interval (30s or 60s)
+3. **Create a DI scope per tick** — in .NET's dependency injection (DI), database contexts are "scoped" (one per request/operation). Background services are singletons (one for the whole app), so they need to create a new scope each tick to get a fresh database context
+4. **Catch and log errors without crashing the collector** — a transient database timeout shouldn't kill the metrics collection permanently
+
+---
+
+## Step 6: Prometheus Configuration
+
+File: `infra/prometheus/prometheus.yml`
+
+```yaml
+global:
+  scrape_interval: 15s
+  evaluation_interval: 15s
+
+scrape_configs:
+  - job_name: vigilcare_api
+    static_configs:
+      - targets: ["host.docker.internal:5270"]
+    metrics_path: /metrics
+```
+
+- **`scrape_interval: 15s`**: Prometheus pulls metrics every 15 seconds. This determines the resolution (granularity) of your data — you'll have one data point every 15 seconds. A shorter interval gives more detail but uses more storage and CPU.
+- **`host.docker.internal:5270`**: The API runs on the host machine on port 5270. The `extra_hosts` entry in `docker-compose.yml` makes this hostname resolvable from inside the Prometheus container.
+- **`metrics_path: /metrics`**: Matches the `app.MapMetrics("/metrics")` endpoint in Program.cs.
+
+---
+
+## Step 7: Grafana Provisioning
+
+Grafana is configured entirely through file provisioning — no manual setup needed after `docker compose up`.
+
+### Datasource: `infra/grafana/provisioning/datasources/prometheus.yml`
+
+```yaml
+apiVersion: 1
+datasources:
+  - name: Prometheus
+    type: prometheus
+    url: http://prometheus:9090
+    isDefault: true
+    access: proxy
+```
+
+Uses the Docker service name `prometheus` (not `localhost`) because Grafana runs inside the Docker network.
+
+### Dashboard Provider: `infra/grafana/provisioning/datasources/dashboards/config.yml`
+
+```yaml
+apiVersion: 1
+providers:
+  - name: VigilCare
+    type: file
+    options:
+      path: /var/lib/grafana/dashboards
+```
+
+Tells Grafana to load all JSON files from `/var/lib/grafana/dashboards`, which is bind-mounted from `./infra/grafana/dashboards/`.
+
+---
+
+## Step 8: Grafana Dashboards
+
+### Clinical Operations Dashboard (`vigilcare.json`)
+
+| Panel | Type | Query | Purpose |
+|-------|------|-------|---------|
+| Unacknowledged Critical Alerts | Stat | `alerts_unacknowledged_gauge` | Primary safety indicator |
+| Observation Ingest Rate | Timeseries | `rate(observations_ingested_total[1m])` | Data flow health |
+| Clinical Alerts by Type (5m) | Bar gauge | `increase(clinical_alerts_total[5m])` | Alert volume breakdown |
+| Outbox Pending Events | Stat | `outbox_pending_events` | Outbox relay health |
+| Kafka Consumer Lag by Group | Timeseries | `kafka_consumer_lag` | Per-pipeline lag |
+| Ingest Latency p50 / p99 | Timeseries | `histogram_quantile(0.50, rate(...[5m]))` / `histogram_quantile(0.99, rate(...[5m]))` | Performance SLA |
+| Escalations Total | Stat | `escalations_total` | Missed alert tracking |
+| Ward Gateways Offline | Stat | `sum(ward_gateways_offline_gauge)` | Edge connectivity |
+| Gateway Buffer Depth | Bar gauge | `ward_gateway_buffer_depth` | Per-gateway backlog |
+| Clinical Sync Batches (5m rate) | Timeseries | `rate(clinical_sync_batches_total[5m])` | Sync throughput |
+| Total Sync Backlog | Stat | `sum(ward_gateway_buffer_depth)` | Fleet-wide backlog |
+
+### Alert Quality Dashboard (`alert-quality-dashboard.json`)
+
+| Panel | Type | Query | Purpose |
+|-------|------|-------|---------|
+| Acknowledgement Rate by Type | Timeseries | `vigilcare_alert_acknowledgement_rate` | Are alerts being seen? |
+| False Positive Rate by Type | Timeseries | `vigilcare_alert_false_positive_rate` | Alert fatigue tracking |
+| Useful Rate by Type | Timeseries | `vigilcare_alert_useful_rate` | Clinical value |
+| Avg Ack Seconds by Type | Timeseries | `vigilcare_alert_avg_ack_seconds` | Response time SLA |
+
+---
+
+## Metrics Reference
+
+### All Counters
+
+| Metric | Labels | What It Tracks |
+|--------|--------|---------------|
+| `observations_ingested_total` | `observation_code`, `source` | Vital sign ingest volume |
+| `clinical_alerts_total` | `alert_type`, `severity` | Alert generation rate |
+| `news2_scores_total` | `risk_level` | NEWS2 scoring frequency |
+| `escalations_total` | — | Unacknowledged alert escalations |
+| `trend_alerts_total` | `observation_code` | Rapid deterioration detections |
+| `alert_suppressions_total` | `alert_type` | Suppression window activations |
+| `qsofa_detections_total` | — | qSOFA screen triggers |
+| `sepsis_bundle_compliance_total` | `status` | Bundle compliance outcomes |
+| `gcs_scores_total` | `classification` | GCS scoring by severity |
+| `sofa_scores_total` | `has_delta_alert` | SOFA scoring with/without alerts |
+| `fhir_ingest_total` | `resource_type`, `outcome` | FHIR inbound operations |
+| `fhir_read_total` | `resource_type`, `interaction`, `outcome` | FHIR read/search operations |
+| `authorization_failures_total` | `permission`, `role` | Failed auth attempts |
+| `fhir_mapping_errors_total` | `reason` | FHIR mapping failures |
+| `phi_access_logs_total` | `access_type` | PHI access audit entries |
+| `clinical_sync_batches_total` | `status` | Gateway sync batch outcomes |
+
+### All Histograms
+
+| Metric | Buckets | What It Tracks |
+|--------|---------|---------------|
+| `observation_ingest_duration_seconds` | 5ms–1s | Full ingest transaction time |
+| `news2_scoring_duration_seconds` | 1ms–100ms | NEWS2 computation time |
+| `trend_analysis_duration_seconds` | 1ms–100ms | Trend evaluation time |
+| `sofa_scoring_duration_seconds` | 1ms–100ms | SOFA computation time |
+| `clinical_sync_batch_duration_seconds` | 100ms–10s | Batch processing time |
+
+### All Gauges
+
+| Metric | Labels | Collector | Interval |
+|--------|--------|-----------|----------|
+| `alerts_unacknowledged_gauge` | — | `AlertsUnacknowledgedCollector` | 30s |
+| `kafka_consumer_lag` | `consumer_group` | `KafkaConsumerLagCollector` | 30s |
+| `outbox_pending_events` | — | `OutboxPendingCollector` | 30s |
+| `ward_gateways_offline_gauge` | `site_code` | `WardGatewayMetricsCollector` | 60s |
+| `ward_gateway_buffer_depth` | `gateway_code`, `department` | `WardGatewayMetricsCollector` | 60s |
+| `vigilcare_alert_acknowledgement_rate` | `alert_type` | `AlertQualityAggregatorService` | 60min |
+| `vigilcare_alert_false_positive_rate` | `alert_type` | `AlertQualityAggregatorService` | 60min |
+| `vigilcare_alert_useful_rate` | `alert_type` | `AlertQualityAggregatorService` | 60min |
+| `vigilcare_alert_avg_ack_seconds` | `alert_type` | `AlertQualityAggregatorService` | 60min |
+
+---
+
+## Accessing the Stack
+
+| Tool | URL | Credentials |
+|------|-----|-------------|
+| Prometheus | http://localhost:9101 | None |
+| Grafana | http://localhost:3101 | admin / admin |
+| Raw metrics | http://localhost:5270/metrics | JWT or anonymous |