562 lines
25 KiB
Markdown
562 lines
25 KiB
Markdown
# Guide 2: Prometheus + Grafana Monitoring Stack
|
||
|
||
## What Are Prometheus and Grafana?
|
||
|
||
**Prometheus** is a time-series database designed for monitoring. Instead of your application pushing data to Prometheus, Prometheus **pulls** (or "scrapes") data from your application on a schedule — typically every 15 seconds. Your application exposes a `/metrics` HTTP endpoint with the current values of all metrics, and Prometheus stores a timestamped history of those values.
|
||
|
||
A **time-series** is just a series of numbers recorded over time — like "at 14:00 there were 3 pending events, at 14:15 there were 5, at 14:30 there were 0." Prometheus stores millions of these series efficiently and lets you query them with its built-in query language, PromQL.
|
||
|
||
**Grafana** is a visualization tool that connects to Prometheus (and other data sources) and renders interactive dashboards — line graphs, gauges, stat panels, and alerts. Prometheus stores the numbers; Grafana makes them visual.
|
||
|
||
**Why use them together?** Logging tells you _what happened_ ("observation ingested for encounter X"). Metrics tell you _how the system is performing right now_ ("we're ingesting 50 observations per second and the p99 latency is 23ms"). When something goes wrong, metrics tell you instantly — often before anyone notices a problem.
|
||
|
||
---
|
||
|
||
## Why Prometheus and Grafana in This Project?
|
||
|
||
VigilCareClinical is a patient safety system. A growing outbox means delayed alerts. High Kafka consumer lag means scores aren't being computed. An unacknowledged critical alert means a patient needs attention. Prometheus collects these metrics every 15 seconds, and Grafana visualizes them on dashboards that clinicians and engineers can watch in real time.
|
||
|
||
---
|
||
|
||
## Architecture Overview
|
||
|
||
```
|
||
.NET API (host) Docker
|
||
┌──────────────────┐ scrape ┌─────────────┐ query ┌─────────────┐
|
||
│ /metrics │ ◄──────────── │ Prometheus │ ◄───────── │ Grafana │
|
||
│ (prometheus-net)│ every 15s │ :9101 │ │ :3101 │
|
||
└──────────────────┘ └─────────────┘ └─────────────┘
|
||
▲
|
||
│ expose
|
||
┌───────┴──────────┐
|
||
│ ClinicalMetrics │ (singleton)
|
||
│ 4 Collectors │ (BackgroundServices)
|
||
└──────────────────┘
|
||
```
|
||
|
||
1. **prometheus-net** (a .NET library) exposes a `/metrics` endpoint on the API that outputs all metric values in Prometheus's text format
|
||
2. **Prometheus** (running in Docker) scrapes that endpoint every 15 seconds and stores the values with timestamps
|
||
3. **Grafana** (also in Docker) queries Prometheus and renders the data as charts and dashboards
|
||
|
||
---
|
||
|
||
## Step 1: The NuGet Package
|
||
|
||
```xml
|
||
<!-- VigilCareClinicalAPI.csproj -->
|
||
<PackageReference Include="prometheus-net.AspNetCore" Version="8.2.1" />
|
||
```
|
||
|
||
This package provides:
|
||
- The `Metrics` factory for creating counters, histograms, and gauges
|
||
- The `MapMetrics()` extension method to expose the `/metrics` HTTP endpoint
|
||
- ASP.NET Core middleware integration
|
||
|
||
---
|
||
|
||
## Step 2: Registering the Metrics Endpoint
|
||
|
||
In `Program.cs`:
|
||
|
||
```csharp
|
||
// Register ClinicalMetrics as a singleton so all services share the same instances
|
||
builder.Services.AddSingleton<ClinicalMetrics>();
|
||
|
||
// ... later, after building the app:
|
||
|
||
// Expose /metrics for Prometheus to scrape
|
||
app.MapMetrics("/metrics");
|
||
```
|
||
|
||
That single `MapMetrics("/metrics")` call serves the Prometheus text exposition format at `http://localhost:5270/metrics`.
|
||
|
||
---
|
||
|
||
## Step 3: Defining Custom Metrics — ClinicalMetrics
|
||
|
||
All custom metrics are defined in a single class, registered as a singleton. This ensures every service that injects `ClinicalMetrics` increments the same counter instances.
|
||
|
||
### Counters
|
||
|
||
Prometheus has three core metric types. The first is a **counter**.
|
||
|
||
A counter only goes up — it never decreases. Think of it like the odometer on a car: it tracks the total distance driven since the car was built. A counter tracks things like "total observations ingested since the app started." The raw number isn't that useful on its own (who cares that we've ingested 50,000 total?), but Prometheus's `rate()` function converts it to "observations per second over the last minute" — that's actionable.
|
||
|
||
```csharp
|
||
public sealed class ClinicalMetrics
|
||
{
|
||
// Labeled by observation_code and source
|
||
public readonly Counter ObservationsIngestedTotal = Metrics.CreateCounter(
|
||
"observations_ingested_total",
|
||
"Total observations ingested, labeled by observation code and source.",
|
||
labelNames: new[] { "observation_code", "source" });
|
||
|
||
// Labeled by alert_type and severity
|
||
public readonly Counter ClinicalAlertsTotal = Metrics.CreateCounter(
|
||
"clinical_alerts_total",
|
||
"Total clinical alerts generated, labeled by type and severity.",
|
||
labelNames: new[] { "alert_type", "severity" });
|
||
|
||
public readonly Counter News2ScoresTotal = Metrics.CreateCounter(
|
||
"news2_scores_total",
|
||
"Total NEWS2 scores computed, labeled by risk level.",
|
||
labelNames: new[] { "risk_level" });
|
||
|
||
public readonly Counter EscalationsTotal = Metrics.CreateCounter(
|
||
"escalations_total",
|
||
"Total alert escalations processed through the DLQ escalation path.");
|
||
|
||
public readonly Counter TrendAlertsTotal = Metrics.CreateCounter(
|
||
"trend_alerts_total",
|
||
"Total RAPID_DETERIORATION alerts generated.",
|
||
labelNames: new[] { "observation_code" });
|
||
|
||
public readonly Counter AlertSuppressionsTotal = Metrics.CreateCounter(
|
||
"alert_suppressions_total",
|
||
"Total alert suppression windows set after acknowledgment.",
|
||
labelNames: new[] { "alert_type" });
|
||
|
||
public readonly Counter QsofaDetectionsTotal = Metrics.CreateCounter(
|
||
"qsofa_detections_total",
|
||
"Total QSOFA_SCREEN alerts generated.");
|
||
|
||
public readonly Counter SepsisBundleComplianceTotal = Metrics.CreateCounter(
|
||
"sepsis_bundle_compliance_total",
|
||
"Sepsis bundle compliance outcomes.",
|
||
labelNames: new[] { "status" });
|
||
|
||
public readonly Counter GcsScoresTotal = Metrics.CreateCounter(
|
||
"gcs_scores_total",
|
||
"GCS scores computed, labeled by classification.",
|
||
labelNames: new[] { "classification" });
|
||
|
||
public readonly Counter SofaScoresTotal = Metrics.CreateCounter(
|
||
"sofa_scores_total",
|
||
"SOFA scores computed.",
|
||
labelNames: new[] { "has_delta_alert" });
|
||
|
||
public readonly Counter FhirIngestTotal = Metrics.CreateCounter(
|
||
"fhir_ingest_total",
|
||
"FHIR resource ingest operations.",
|
||
labelNames: new[] { "resource_type", "outcome" });
|
||
|
||
public readonly Counter FhirReadTotal = Metrics.CreateCounter(
|
||
"fhir_read_total",
|
||
"FHIR resource read/search operations.",
|
||
labelNames: new[] { "resource_type", "interaction", "outcome" });
|
||
|
||
public readonly Counter AuthorizationFailuresTotal = Metrics.CreateCounter(
|
||
"authorization_failures_total",
|
||
"Authorization failures by permission and role.",
|
||
labelNames: new[] { "permission", "role" });
|
||
|
||
public readonly Counter FhirMappingErrorsTotal = Metrics.CreateCounter(
|
||
"fhir_mapping_errors_total",
|
||
"FHIR mapping failures.",
|
||
labelNames: new[] { "reason" });
|
||
|
||
public readonly Counter PhiAccessLogsTotal = Metrics.CreateCounter(
|
||
"phi_access_logs_total",
|
||
"PHI access log entries written.",
|
||
labelNames: new[] { "access_type" });
|
||
|
||
public readonly Counter ClinicalSyncBatchesTotal = Metrics.CreateCounter(
|
||
"clinical_sync_batches_total",
|
||
"Sync batches processed.",
|
||
labelNames: new[] { "status" });
|
||
}
|
||
```
|
||
|
||
**What are labels?** Labels are key-value tags attached to a metric that let you slice and filter the data. Instead of creating separate counters like `critical_heart_rate_alerts_total` and `warning_spo2_alerts_total`, you create one counter `clinical_alerts_total` with labels `alert_type` and `severity`. This single counter then lets you query:
|
||
- Total alerts: `clinical_alerts_total`
|
||
- Only critical threshold breaches: `clinical_alerts_total{severity="Critical"}`
|
||
- Only SOFA sepsis alerts: `clinical_alerts_total{alert_type="SOFA_SEPSIS"}`
|
||
|
||
### Histograms
|
||
|
||
A **histogram** tracks the _distribution_ of values, not just a total count. For example, "how long does it take to ingest an observation?" — a counter can tell you how many were ingested, but a histogram tells you that 50% completed in under 10ms, 95% in under 50ms, and the slowest 1% took over 200ms.
|
||
|
||
Histograms work by defining **buckets** — thresholds like 5ms, 10ms, 25ms, 50ms, etc. Each observation is counted into all buckets it falls below. Prometheus then stores `_bucket` (how many observations fell into each bucket), `_sum` (total time across all observations), and `_count` (total number of observations). From these, you can calculate percentiles using the `histogram_quantile()` function in PromQL.
|
||
|
||
```csharp
|
||
public readonly Histogram ObservationIngestDuration = Metrics.CreateHistogram(
|
||
"observation_ingest_duration_seconds",
|
||
"Ingest transaction duration from request receipt to COMMIT.",
|
||
new HistogramConfiguration
|
||
{
|
||
Buckets = new[] { 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0 }
|
||
});
|
||
|
||
public readonly Histogram News2ScoringDuration = Metrics.CreateHistogram(
|
||
"news2_scoring_duration_seconds",
|
||
"Time to compute a NEWS2 score from Redis state.",
|
||
new HistogramConfiguration
|
||
{
|
||
Buckets = new[] { 0.001, 0.005, 0.01, 0.025, 0.05, 0.1 }
|
||
});
|
||
|
||
public readonly Histogram TrendAnalysisDuration = Metrics.CreateHistogram(
|
||
"trend_analysis_duration_seconds",
|
||
"Time to evaluate trend for one observation.",
|
||
new HistogramConfiguration
|
||
{
|
||
Buckets = new[] { 0.001, 0.005, 0.01, 0.025, 0.05, 0.1 }
|
||
});
|
||
|
||
public readonly Histogram SofaScoringDuration = Metrics.CreateHistogram(
|
||
"sofa_scoring_duration_seconds",
|
||
"SOFA scoring computation time.",
|
||
new HistogramConfiguration
|
||
{
|
||
Buckets = new[] { 0.001, 0.005, 0.01, 0.025, 0.05, 0.1 }
|
||
});
|
||
|
||
public readonly Histogram ClinicalSyncBatchDuration = Metrics.CreateHistogram(
|
||
"clinical_sync_batch_duration_seconds",
|
||
"Batch processing duration.",
|
||
new HistogramConfiguration
|
||
{
|
||
Buckets = new[] { 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0 }
|
||
});
|
||
```
|
||
|
||
**Bucket selection matters**: The ingest histogram uses fine-grained buckets (5ms to 1s) because sub-second latency is critical for patient safety. The sync batch histogram uses coarser buckets (100ms to 10s) because batch operations are inherently slower.
|
||
|
||
### Gauges
|
||
|
||
A **gauge** is a value that can go up or down — like a thermometer or a fuel gauge. It represents the _current state_ of something: "right now there are 3 unacknowledged alerts" or "the outbox has 47 pending events." Unlike counters (which only increase), gauges are set to an absolute value by background services that periodically check the current state.
|
||
|
||
```csharp
|
||
public readonly Gauge AlertsUnacknowledgedGauge = Metrics.CreateGauge(
|
||
"alerts_unacknowledged_gauge",
|
||
"Count of open CRITICAL alerts older than 5 minutes with no acknowledgment.");
|
||
|
||
public readonly Gauge KafkaConsumerLag = Metrics.CreateGauge(
|
||
"kafka_consumer_lag",
|
||
"Approximate consumer group lag in messages.",
|
||
labelNames: new[] { "consumer_group" });
|
||
|
||
public readonly Gauge OutboxPendingEvents = Metrics.CreateGauge(
|
||
"outbox_pending_events",
|
||
"Count of outbox events not yet relayed to Kafka.");
|
||
|
||
public readonly Gauge WardGatewaysOffline = Metrics.CreateGauge(
|
||
"ward_gateways_offline_gauge",
|
||
"Ward gateways with status OFFLINE or DEGRADED.",
|
||
labelNames: new[] { "site_code" });
|
||
|
||
public readonly Gauge WardGatewayBufferDepth = Metrics.CreateGauge(
|
||
"ward_gateway_buffer_depth",
|
||
"Reported unsynced event count per gateway.",
|
||
labelNames: new[] { "gateway_code", "department" });
|
||
|
||
public readonly Gauge AlertAcknowledgementRate = Metrics.CreateGauge(
|
||
"vigilcare_alert_acknowledgement_rate",
|
||
"Alert acknowledgement rate by type.",
|
||
labelNames: new[] { "alert_type" });
|
||
|
||
public readonly Gauge AlertFalsePositiveRate = Metrics.CreateGauge(
|
||
"vigilcare_alert_false_positive_rate",
|
||
"Clinician-reported false positive rate by type.",
|
||
labelNames: new[] { "alert_type" });
|
||
|
||
public readonly Gauge AlertUsefulRate = Metrics.CreateGauge(
|
||
"vigilcare_alert_useful_rate",
|
||
"Clinician-reported useful rate by type.",
|
||
labelNames: new[] { "alert_type" });
|
||
|
||
public readonly Gauge AlertAvgAckSeconds = Metrics.CreateGauge(
|
||
"vigilcare_alert_avg_ack_seconds",
|
||
"Average seconds from trigger to acknowledgement by type.",
|
||
labelNames: new[] { "alert_type" });
|
||
```
|
||
|
||
---
|
||
|
||
## Step 4: Using Metrics in Application Code
|
||
|
||
### Inline Counters and Timers
|
||
|
||
In `ObservationService`, the ingest path both times the transaction and increments counters:
|
||
|
||
```csharp
|
||
using var timer = _metrics.ObservationIngestDuration.NewTimer();
|
||
|
||
// ... perform the ingest transaction ...
|
||
|
||
_metrics.ObservationsIngestedTotal
|
||
.WithLabels(req.ObservationCode, req.Source.ToDbString())
|
||
.Inc();
|
||
|
||
if (alert is not null)
|
||
{
|
||
_metrics.ClinicalAlertsTotal
|
||
.WithLabels(alert.AlertType.ToDbString(), alert.Severity.ToDbString())
|
||
.Inc();
|
||
}
|
||
```
|
||
|
||
**How does `NewTimer()` work?** In C#, a `using` block runs some cleanup code when the block exits. `NewTimer()` starts a stopwatch and the `using` block ensures it records the elapsed time into the histogram when the code leaves the block — even if an exception is thrown. This means you don't need to manually calculate timing.
|
||
|
||
**What does `WithLabels()` do?** It selects which specific "bucket" of the counter to increment. A counter with labels is really a _family_ of counters — one for each unique label combination. `.WithLabels("HEART_RATE", "DEVICE")` increments the counter for heart rate observations from devices specifically.
|
||
|
||
---
|
||
|
||
## Step 5: Background Collectors
|
||
|
||
Gauges can't be updated inline in request handlers because they represent _current state_ ("how many right now?"), not events ("one more just happened"). You need a background process that periodically checks the current state and updates the gauge.
|
||
|
||
In .NET, a `BackgroundService` is a class that runs continuously in the background for the lifetime of the application. Four of these poll data sources on a timer and update gauge values.
|
||
|
||
### AlertsUnacknowledgedCollector (every 30s)
|
||
|
||
The most clinically significant metric. Queries PostgreSQL for CRITICAL alerts that have been open more than 5 minutes with no acknowledgment:
|
||
|
||
```csharp
|
||
protected override async Task ExecuteAsync(CancellationToken ct)
|
||
{
|
||
using var timer = new PeriodicTimer(TimeSpan.FromSeconds(30));
|
||
while (await timer.WaitForNextTickAsync(ct))
|
||
await CollectAsync(ct);
|
||
}
|
||
|
||
private async Task CollectAsync(CancellationToken ct)
|
||
{
|
||
await using var scope = _scopes.CreateAsyncScope();
|
||
var db = scope.ServiceProvider.GetRequiredService<AppDbContext>();
|
||
var cutoff = DateTimeOffset.UtcNow - TimeSpan.FromMinutes(5);
|
||
|
||
var count = await db.ClinicalAlerts
|
||
.CountAsync(a => a.Severity == AlertSeverity.Critical
|
||
&& a.Status == AlertStatus.Open
|
||
&& a.TriggeredAt < cutoff, ct);
|
||
|
||
_metrics.AlertsUnacknowledgedGauge.Set(count);
|
||
|
||
if (count > 0)
|
||
_logger.LogWarning(
|
||
"[PATIENT-SAFETY] alerts_unacknowledged_gauge={Count}", count);
|
||
}
|
||
```
|
||
|
||
It also logs a warning when the count is non-zero — this shows up in Seq as a structured log entry tagged `[PATIENT-SAFETY]`.
|
||
|
||
### OutboxPendingCollector (every 30s)
|
||
|
||
Counts outbox events not yet relayed to Kafka. A growing number means the relay is falling behind or Kafka is unreachable:
|
||
|
||
```csharp
|
||
var count = await db.OutboxEvents
|
||
.CountAsync(e => e.ProcessedAt == null, ct);
|
||
_metrics.OutboxPendingEvents.Set(count);
|
||
```
|
||
|
||
### KafkaConsumerLagCollector (every 30s)
|
||
|
||
Measures how far behind each Kafka consumer group is. **Consumer lag** is the number of messages that have been published but not yet processed. Think of it like a queue at a bank — lag is how many people are still waiting. This collector uses Kafka's admin API to check the lag for each consumer group without actually consuming any messages:
|
||
|
||
```csharp
|
||
private static readonly string[] Groups =
|
||
{
|
||
"es-indexer",
|
||
"sepsis-engine",
|
||
"notification-publisher",
|
||
"data-lake-writer",
|
||
};
|
||
|
||
private async Task CollectGroupLagAsync(string groupId, CancellationToken ct)
|
||
{
|
||
using var admin = new AdminClientBuilder(adminConfig).Build();
|
||
|
||
// Get committed offsets for this consumer group
|
||
var result = await admin.ListConsumerGroupOffsetsAsync(...);
|
||
|
||
// Query high watermarks with a temporary consumer
|
||
using var tempConsumer = new ConsumerBuilder<Ignore, Ignore>(config).Build();
|
||
|
||
long totalLag = 0;
|
||
foreach (var tpo in partitions)
|
||
{
|
||
var watermarks = tempConsumer.QueryWatermarkOffsets(
|
||
tpo.TopicPartition, TimeSpan.FromSeconds(5));
|
||
var lag = watermarks.High.Value - tpo.Offset.Value;
|
||
totalLag += Math.Max(0L, lag);
|
||
}
|
||
|
||
_metrics.KafkaConsumerLag.WithLabels(groupId).Set(totalLag);
|
||
}
|
||
```
|
||
|
||
The lag is per consumer group. If `sepsis-engine` has high lag but `es-indexer` doesn't, the Grafana panel makes it immediately visible which pipeline is falling behind.
|
||
|
||
### WardGatewayMetricsCollector (every 60s)
|
||
|
||
Queries PostgreSQL for gateway status and buffer depth:
|
||
|
||
```csharp
|
||
var offlineBySite = await db.WardGateways
|
||
.Include(g => g.Site)
|
||
.Where(g => g.Status != GatewayStatus.Online)
|
||
.GroupBy(g => g.Site.SiteCode)
|
||
.Select(g => new { SiteCode = g.Key, Count = g.Count() })
|
||
.ToListAsync(ct);
|
||
|
||
foreach (var row in offlineBySite)
|
||
_metrics.WardGatewaysOffline.WithLabels(row.SiteCode).Set(row.Count);
|
||
|
||
var allGateways = await db.WardGateways.AsNoTracking().ToListAsync(ct);
|
||
foreach (var g in allGateways)
|
||
_metrics.WardGatewayBufferDepth
|
||
.WithLabels(g.GatewayCode, g.Department)
|
||
.Set(g.ReportedBufferDepth);
|
||
```
|
||
|
||
All four collectors follow the same pattern:
|
||
1. **Extend `BackgroundService`** — .NET's base class for long-running background work
|
||
2. **Use `PeriodicTimer` for the poll loop** — fires a callback at a fixed interval (30s or 60s)
|
||
3. **Create a DI scope per tick** — in .NET's dependency injection (DI), database contexts are "scoped" (one per request/operation). Background services are singletons (one for the whole app), so they need to create a new scope each tick to get a fresh database context
|
||
4. **Catch and log errors without crashing the collector** — a transient database timeout shouldn't kill the metrics collection permanently
|
||
|
||
---
|
||
|
||
## Step 6: Prometheus Configuration
|
||
|
||
File: `infra/prometheus/prometheus.yml`
|
||
|
||
```yaml
|
||
global:
|
||
scrape_interval: 15s
|
||
evaluation_interval: 15s
|
||
|
||
scrape_configs:
|
||
- job_name: vigilcare_api
|
||
static_configs:
|
||
- targets: ["host.docker.internal:5270"]
|
||
metrics_path: /metrics
|
||
```
|
||
|
||
- **`scrape_interval: 15s`**: Prometheus pulls metrics every 15 seconds. This determines the resolution (granularity) of your data — you'll have one data point every 15 seconds. A shorter interval gives more detail but uses more storage and CPU.
|
||
- **`host.docker.internal:5270`**: The API runs on the host machine on port 5270. The `extra_hosts` entry in `docker-compose.yml` makes this hostname resolvable from inside the Prometheus container.
|
||
- **`metrics_path: /metrics`**: Matches the `app.MapMetrics("/metrics")` endpoint in Program.cs.
|
||
|
||
---
|
||
|
||
## Step 7: Grafana Provisioning
|
||
|
||
Grafana is configured entirely through file provisioning — no manual setup needed after `docker compose up`.
|
||
|
||
### Datasource: `infra/grafana/provisioning/datasources/prometheus.yml`
|
||
|
||
```yaml
|
||
apiVersion: 1
|
||
datasources:
|
||
- name: Prometheus
|
||
type: prometheus
|
||
url: http://prometheus:9090
|
||
isDefault: true
|
||
access: proxy
|
||
```
|
||
|
||
Uses the Docker service name `prometheus` (not `localhost`) because Grafana runs inside the Docker network.
|
||
|
||
### Dashboard Provider: `infra/grafana/provisioning/datasources/dashboards/config.yml`
|
||
|
||
```yaml
|
||
apiVersion: 1
|
||
providers:
|
||
- name: VigilCare
|
||
type: file
|
||
options:
|
||
path: /var/lib/grafana/dashboards
|
||
```
|
||
|
||
Tells Grafana to load all JSON files from `/var/lib/grafana/dashboards`, which is bind-mounted from `./infra/grafana/dashboards/`.
|
||
|
||
---
|
||
|
||
## Step 8: Grafana Dashboards
|
||
|
||
### Clinical Operations Dashboard (`vigilcare.json`)
|
||
|
||
| Panel | Type | Query | Purpose |
|
||
|-------|------|-------|---------|
|
||
| Unacknowledged Critical Alerts | Stat | `alerts_unacknowledged_gauge` | Primary safety indicator |
|
||
| Observation Ingest Rate | Timeseries | `rate(observations_ingested_total[1m])` | Data flow health |
|
||
| Clinical Alerts by Type (5m) | Bar gauge | `increase(clinical_alerts_total[5m])` | Alert volume breakdown |
|
||
| Outbox Pending Events | Stat | `outbox_pending_events` | Outbox relay health |
|
||
| Kafka Consumer Lag by Group | Timeseries | `kafka_consumer_lag` | Per-pipeline lag |
|
||
| Ingest Latency p50 / p99 | Timeseries | `histogram_quantile(0.50, rate(...[5m]))` / `histogram_quantile(0.99, rate(...[5m]))` | Performance SLA |
|
||
| Escalations Total | Stat | `escalations_total` | Missed alert tracking |
|
||
| Ward Gateways Offline | Stat | `sum(ward_gateways_offline_gauge)` | Edge connectivity |
|
||
| Gateway Buffer Depth | Bar gauge | `ward_gateway_buffer_depth` | Per-gateway backlog |
|
||
| Clinical Sync Batches (5m rate) | Timeseries | `rate(clinical_sync_batches_total[5m])` | Sync throughput |
|
||
| Total Sync Backlog | Stat | `sum(ward_gateway_buffer_depth)` | Fleet-wide backlog |
|
||
|
||
### Alert Quality Dashboard (`alert-quality-dashboard.json`)
|
||
|
||
| Panel | Type | Query | Purpose |
|
||
|-------|------|-------|---------|
|
||
| Acknowledgement Rate by Type | Timeseries | `vigilcare_alert_acknowledgement_rate` | Are alerts being seen? |
|
||
| False Positive Rate by Type | Timeseries | `vigilcare_alert_false_positive_rate` | Alert fatigue tracking |
|
||
| Useful Rate by Type | Timeseries | `vigilcare_alert_useful_rate` | Clinical value |
|
||
| Avg Ack Seconds by Type | Timeseries | `vigilcare_alert_avg_ack_seconds` | Response time SLA |
|
||
|
||
---
|
||
|
||
## Metrics Reference
|
||
|
||
### All Counters
|
||
|
||
| Metric | Labels | What It Tracks |
|
||
|--------|--------|---------------|
|
||
| `observations_ingested_total` | `observation_code`, `source` | Vital sign ingest volume |
|
||
| `clinical_alerts_total` | `alert_type`, `severity` | Alert generation rate |
|
||
| `news2_scores_total` | `risk_level` | NEWS2 scoring frequency |
|
||
| `escalations_total` | — | Unacknowledged alert escalations |
|
||
| `trend_alerts_total` | `observation_code` | Rapid deterioration detections |
|
||
| `alert_suppressions_total` | `alert_type` | Suppression window activations |
|
||
| `qsofa_detections_total` | — | qSOFA screen triggers |
|
||
| `sepsis_bundle_compliance_total` | `status` | Bundle compliance outcomes |
|
||
| `gcs_scores_total` | `classification` | GCS scoring by severity |
|
||
| `sofa_scores_total` | `has_delta_alert` | SOFA scoring with/without alerts |
|
||
| `fhir_ingest_total` | `resource_type`, `outcome` | FHIR inbound operations |
|
||
| `fhir_read_total` | `resource_type`, `interaction`, `outcome` | FHIR read/search operations |
|
||
| `authorization_failures_total` | `permission`, `role` | Failed auth attempts |
|
||
| `fhir_mapping_errors_total` | `reason` | FHIR mapping failures |
|
||
| `phi_access_logs_total` | `access_type` | PHI access audit entries |
|
||
| `clinical_sync_batches_total` | `status` | Gateway sync batch outcomes |
|
||
|
||
### All Histograms
|
||
|
||
| Metric | Buckets | What It Tracks |
|
||
|--------|---------|---------------|
|
||
| `observation_ingest_duration_seconds` | 5ms–1s | Full ingest transaction time |
|
||
| `news2_scoring_duration_seconds` | 1ms–100ms | NEWS2 computation time |
|
||
| `trend_analysis_duration_seconds` | 1ms–100ms | Trend evaluation time |
|
||
| `sofa_scoring_duration_seconds` | 1ms–100ms | SOFA computation time |
|
||
| `clinical_sync_batch_duration_seconds` | 100ms–10s | Batch processing time |
|
||
|
||
### All Gauges
|
||
|
||
| Metric | Labels | Collector | Interval |
|
||
|--------|--------|-----------|----------|
|
||
| `alerts_unacknowledged_gauge` | — | `AlertsUnacknowledgedCollector` | 30s |
|
||
| `kafka_consumer_lag` | `consumer_group` | `KafkaConsumerLagCollector` | 30s |
|
||
| `outbox_pending_events` | — | `OutboxPendingCollector` | 30s |
|
||
| `ward_gateways_offline_gauge` | `site_code` | `WardGatewayMetricsCollector` | 60s |
|
||
| `ward_gateway_buffer_depth` | `gateway_code`, `department` | `WardGatewayMetricsCollector` | 60s |
|
||
| `vigilcare_alert_acknowledgement_rate` | `alert_type` | `AlertQualityAggregatorService` | 60min |
|
||
| `vigilcare_alert_false_positive_rate` | `alert_type` | `AlertQualityAggregatorService` | 60min |
|
||
| `vigilcare_alert_useful_rate` | `alert_type` | `AlertQualityAggregatorService` | 60min |
|
||
| `vigilcare_alert_avg_ack_seconds` | `alert_type` | `AlertQualityAggregatorService` | 60min |
|
||
|
||
---
|
||
|
||
## Accessing the Stack
|
||
|
||
| Tool | URL | Credentials |
|
||
|------|-----|-------------|
|
||
| Prometheus | http://localhost:9101 | None |
|
||
| Grafana | http://localhost:3101 | admin / admin |
|
||
| Raw metrics | http://localhost:5270/metrics | JWT or anonymous |
|