vigilcare-clinical/docs/guides/02-prometheus-grafana-monitoring.md

Guide 2: Prometheus + Grafana Monitoring Stack

What Are Prometheus and Grafana?

Prometheus is a time-series database designed for monitoring. Instead of your application pushing data to Prometheus, Prometheus pulls (or "scrapes") data from your application on a schedule — typically every 15 seconds. Your application exposes a /metrics HTTP endpoint with the current values of all metrics, and Prometheus stores a timestamped history of those values.

A time-series is just a series of numbers recorded over time — like "at 14:00 there were 3 pending events, at 14:15 there were 5, at 14:30 there were 0." Prometheus stores millions of these series efficiently and lets you query them with its built-in query language, PromQL.

Grafana is a visualization tool that connects to Prometheus (and other data sources) and renders interactive dashboards — line graphs, gauges, stat panels, and alerts. Prometheus stores the numbers; Grafana makes them visual.

Why use them together? Logging tells you what happened ("observation ingested for encounter X"). Metrics tell you how the system is performing right now ("we're ingesting 50 observations per second and the p99 latency is 23ms"). When something goes wrong, metrics tell you instantly — often before anyone notices a problem.

---

Why Prometheus and Grafana in This Project?

VigilCareClinical is a patient safety system. A growing outbox means delayed alerts. High Kafka consumer lag means scores aren't being computed. An unacknowledged critical alert means a patient needs attention. Prometheus collects these metrics every 15 seconds, and Grafana visualizes them on dashboards that clinicians and engineers can watch in real time.

---

Architecture Overview

.NET API (host)                    Docker
┌──────────────────┐     scrape    ┌─────────────┐    query    ┌─────────────┐
│  /metrics        │ ◄──────────── │  Prometheus  │ ◄───────── │   Grafana   │
│  (prometheus-net)│   every 15s   │  :9101       │            │   :3101     │
└──────────────────┘               └─────────────┘            └─────────────┘
        ▲
        │ expose
┌───────┴──────────┐
│  ClinicalMetrics │  (singleton)
│  4 Collectors    │  (BackgroundServices)
└──────────────────┘

1. prometheus-net (a .NET library) exposes a /metrics endpoint on the API that outputs all metric values in Prometheus's text format
2. Prometheus (running in Docker) scrapes that endpoint every 15 seconds and stores the values with timestamps
3. Grafana (also in Docker) queries Prometheus and renders the data as charts and dashboards

---

Step 1: The NuGet Package

<!-- VigilCareClinicalAPI.csproj -->
<PackageReference Include="prometheus-net.AspNetCore" Version="8.2.1" />

This package provides:

• The Metrics factory for creating counters, histograms, and gauges
• The MapMetrics() extension method to expose the /metrics HTTP endpoint
• ASP.NET Core middleware integration

---

Step 2: Registering the Metrics Endpoint

In Program.cs:

// Register ClinicalMetrics as a singleton so all services share the same instances
builder.Services.AddSingleton<ClinicalMetrics>();

// ... later, after building the app:

// Expose /metrics for Prometheus to scrape
app.MapMetrics("/metrics");

That single MapMetrics("/metrics") call serves the Prometheus text exposition format at http://localhost:5270/metrics.

---

Step 3: Defining Custom Metrics — ClinicalMetrics

All custom metrics are defined in a single class, registered as a singleton. This ensures every service that injects ClinicalMetrics increments the same counter instances.

Counters

Prometheus has three core metric types. The first is a counter.

A counter only goes up — it never decreases. Think of it like the odometer on a car: it tracks the total distance driven since the car was built. A counter tracks things like "total observations ingested since the app started." The raw number isn't that useful on its own (who cares that we've ingested 50,000 total?), but Prometheus's rate() function converts it to "observations per second over the last minute" — that's actionable.

public sealed class ClinicalMetrics
{
    // Labeled by observation_code and source
    public readonly Counter ObservationsIngestedTotal = Metrics.CreateCounter(
        "observations_ingested_total",
        "Total observations ingested, labeled by observation code and source.",
        labelNames: new[] { "observation_code", "source" });

    // Labeled by alert_type and severity
    public readonly Counter ClinicalAlertsTotal = Metrics.CreateCounter(
        "clinical_alerts_total",
        "Total clinical alerts generated, labeled by type and severity.",
        labelNames: new[] { "alert_type", "severity" });

    public readonly Counter News2ScoresTotal = Metrics.CreateCounter(
        "news2_scores_total",
        "Total NEWS2 scores computed, labeled by risk level.",
        labelNames: new[] { "risk_level" });

    public readonly Counter EscalationsTotal = Metrics.CreateCounter(
        "escalations_total",
        "Total alert escalations processed through the DLQ escalation path.");

    public readonly Counter TrendAlertsTotal = Metrics.CreateCounter(
        "trend_alerts_total",
        "Total RAPID_DETERIORATION alerts generated.",
        labelNames: new[] { "observation_code" });

    public readonly Counter AlertSuppressionsTotal = Metrics.CreateCounter(
        "alert_suppressions_total",
        "Total alert suppression windows set after acknowledgment.",
        labelNames: new[] { "alert_type" });

    public readonly Counter QsofaDetectionsTotal = Metrics.CreateCounter(
        "qsofa_detections_total",
        "Total QSOFA_SCREEN alerts generated.");

    public readonly Counter SepsisBundleComplianceTotal = Metrics.CreateCounter(
        "sepsis_bundle_compliance_total",
        "Sepsis bundle compliance outcomes.",
        labelNames: new[] { "status" });

    public readonly Counter GcsScoresTotal = Metrics.CreateCounter(
        "gcs_scores_total",
        "GCS scores computed, labeled by classification.",
        labelNames: new[] { "classification" });

    public readonly Counter SofaScoresTotal = Metrics.CreateCounter(
        "sofa_scores_total",
        "SOFA scores computed.",
        labelNames: new[] { "has_delta_alert" });

    public readonly Counter FhirIngestTotal = Metrics.CreateCounter(
        "fhir_ingest_total",
        "FHIR resource ingest operations.",
        labelNames: new[] { "resource_type", "outcome" });

    public readonly Counter FhirReadTotal = Metrics.CreateCounter(
        "fhir_read_total",
        "FHIR resource read/search operations.",
        labelNames: new[] { "resource_type", "interaction", "outcome" });

    public readonly Counter AuthorizationFailuresTotal = Metrics.CreateCounter(
        "authorization_failures_total",
        "Authorization failures by permission and role.",
        labelNames: new[] { "permission", "role" });

    public readonly Counter FhirMappingErrorsTotal = Metrics.CreateCounter(
        "fhir_mapping_errors_total",
        "FHIR mapping failures.",
        labelNames: new[] { "reason" });

    public readonly Counter PhiAccessLogsTotal = Metrics.CreateCounter(
        "phi_access_logs_total",
        "PHI access log entries written.",
        labelNames: new[] { "access_type" });

    public readonly Counter ClinicalSyncBatchesTotal = Metrics.CreateCounter(
        "clinical_sync_batches_total",
        "Sync batches processed.",
        labelNames: new[] { "status" });
}

What are labels? Labels are key-value tags attached to a metric that let you slice and filter the data. Instead of creating separate counters like critical_heart_rate_alerts_total and warning_spo2_alerts_total, you create one counter clinical_alerts_total with labels alert_type and severity. This single counter then lets you query:

• Total alerts: clinical_alerts_total
• Only critical threshold breaches: clinical_alerts_total{severity="Critical"}
• Only SOFA sepsis alerts: clinical_alerts_total{alert_type="SOFA_SEPSIS"}

Histograms

A histogram tracks the distribution of values, not just a total count. For example, "how long does it take to ingest an observation?" — a counter can tell you how many were ingested, but a histogram tells you that 50% completed in under 10ms, 95% in under 50ms, and the slowest 1% took over 200ms.

Histograms work by defining buckets — thresholds like 5ms, 10ms, 25ms, 50ms, etc. Each observation is counted into all buckets it falls below. Prometheus then stores _bucket (how many observations fell into each bucket), _sum (total time across all observations), and _count (total number of observations). From these, you can calculate percentiles using the histogram_quantile() function in PromQL.

public readonly Histogram ObservationIngestDuration = Metrics.CreateHistogram(
    "observation_ingest_duration_seconds",
    "Ingest transaction duration from request receipt to COMMIT.",
    new HistogramConfiguration
    {
        Buckets = new[] { 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0 }
    });

public readonly Histogram News2ScoringDuration = Metrics.CreateHistogram(
    "news2_scoring_duration_seconds",
    "Time to compute a NEWS2 score from Redis state.",
    new HistogramConfiguration
    {
        Buckets = new[] { 0.001, 0.005, 0.01, 0.025, 0.05, 0.1 }
    });

public readonly Histogram TrendAnalysisDuration = Metrics.CreateHistogram(
    "trend_analysis_duration_seconds",
    "Time to evaluate trend for one observation.",
    new HistogramConfiguration
    {
        Buckets = new[] { 0.001, 0.005, 0.01, 0.025, 0.05, 0.1 }
    });

public readonly Histogram SofaScoringDuration = Metrics.CreateHistogram(
    "sofa_scoring_duration_seconds",
    "SOFA scoring computation time.",
    new HistogramConfiguration
    {
        Buckets = new[] { 0.001, 0.005, 0.01, 0.025, 0.05, 0.1 }
    });

public readonly Histogram ClinicalSyncBatchDuration = Metrics.CreateHistogram(
    "clinical_sync_batch_duration_seconds",
    "Batch processing duration.",
    new HistogramConfiguration
    {
        Buckets = new[] { 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0 }
    });

Bucket selection matters: The ingest histogram uses fine-grained buckets (5ms to 1s) because sub-second latency is critical for patient safety. The sync batch histogram uses coarser buckets (100ms to 10s) because batch operations are inherently slower.

Gauges

A gauge is a value that can go up or down — like a thermometer or a fuel gauge. It represents the current state of something: "right now there are 3 unacknowledged alerts" or "the outbox has 47 pending events." Unlike counters (which only increase), gauges are set to an absolute value by background services that periodically check the current state.

public readonly Gauge AlertsUnacknowledgedGauge = Metrics.CreateGauge(
    "alerts_unacknowledged_gauge",
    "Count of open CRITICAL alerts older than 5 minutes with no acknowledgment.");

public readonly Gauge KafkaConsumerLag = Metrics.CreateGauge(
    "kafka_consumer_lag",
    "Approximate consumer group lag in messages.",
    labelNames: new[] { "consumer_group" });

public readonly Gauge OutboxPendingEvents = Metrics.CreateGauge(
    "outbox_pending_events",
    "Count of outbox events not yet relayed to Kafka.");

public readonly Gauge WardGatewaysOffline = Metrics.CreateGauge(
    "ward_gateways_offline_gauge",
    "Ward gateways with status OFFLINE or DEGRADED.",
    labelNames: new[] { "site_code" });

public readonly Gauge WardGatewayBufferDepth = Metrics.CreateGauge(
    "ward_gateway_buffer_depth",
    "Reported unsynced event count per gateway.",
    labelNames: new[] { "gateway_code", "department" });

public readonly Gauge AlertAcknowledgementRate = Metrics.CreateGauge(
    "vigilcare_alert_acknowledgement_rate",
    "Alert acknowledgement rate by type.",
    labelNames: new[] { "alert_type" });

public readonly Gauge AlertFalsePositiveRate = Metrics.CreateGauge(
    "vigilcare_alert_false_positive_rate",
    "Clinician-reported false positive rate by type.",
    labelNames: new[] { "alert_type" });

public readonly Gauge AlertUsefulRate = Metrics.CreateGauge(
    "vigilcare_alert_useful_rate",
    "Clinician-reported useful rate by type.",
    labelNames: new[] { "alert_type" });

public readonly Gauge AlertAvgAckSeconds = Metrics.CreateGauge(
    "vigilcare_alert_avg_ack_seconds",
    "Average seconds from trigger to acknowledgement by type.",
    labelNames: new[] { "alert_type" });

---

Step 4: Using Metrics in Application Code

Inline Counters and Timers

In ObservationService, the ingest path both times the transaction and increments counters:

using var timer = _metrics.ObservationIngestDuration.NewTimer();

// ... perform the ingest transaction ...

_metrics.ObservationsIngestedTotal
    .WithLabels(req.ObservationCode, req.Source.ToDbString())
    .Inc();

if (alert is not null)
{
    _metrics.ClinicalAlertsTotal
        .WithLabels(alert.AlertType.ToDbString(), alert.Severity.ToDbString())
        .Inc();
}

How does NewTimer() work? In C#, a using block runs some cleanup code when the block exits. NewTimer() starts a stopwatch and the using block ensures it records the elapsed time into the histogram when the code leaves the block — even if an exception is thrown. This means you don't need to manually calculate timing.

What does WithLabels() do? It selects which specific "bucket" of the counter to increment. A counter with labels is really a family of counters — one for each unique label combination. .WithLabels("HEART_RATE", "DEVICE") increments the counter for heart rate observations from devices specifically.

---

Step 5: Background Collectors

Gauges can't be updated inline in request handlers because they represent current state ("how many right now?"), not events ("one more just happened"). You need a background process that periodically checks the current state and updates the gauge.

In .NET, a BackgroundService is a class that runs continuously in the background for the lifetime of the application. Four of these poll data sources on a timer and update gauge values.

AlertsUnacknowledgedCollector (every 30s)

The most clinically significant metric. Queries PostgreSQL for CRITICAL alerts that have been open more than 5 minutes with no acknowledgment:

protected override async Task ExecuteAsync(CancellationToken ct)
{
    using var timer = new PeriodicTimer(TimeSpan.FromSeconds(30));
    while (await timer.WaitForNextTickAsync(ct))
        await CollectAsync(ct);
}

private async Task CollectAsync(CancellationToken ct)
{
    await using var scope = _scopes.CreateAsyncScope();
    var db     = scope.ServiceProvider.GetRequiredService<AppDbContext>();
    var cutoff = DateTimeOffset.UtcNow - TimeSpan.FromMinutes(5);

    var count = await db.ClinicalAlerts
        .CountAsync(a => a.Severity   == AlertSeverity.Critical
                      && a.Status      == AlertStatus.Open
                      && a.TriggeredAt < cutoff, ct);

    _metrics.AlertsUnacknowledgedGauge.Set(count);

    if (count > 0)
        _logger.LogWarning(
            "[PATIENT-SAFETY] alerts_unacknowledged_gauge={Count}", count);
}

It also logs a warning when the count is non-zero — this shows up in Seq as a structured log entry tagged [PATIENT-SAFETY].

OutboxPendingCollector (every 30s)

Counts outbox events not yet relayed to Kafka. A growing number means the relay is falling behind or Kafka is unreachable:

var count = await db.OutboxEvents
    .CountAsync(e => e.ProcessedAt == null, ct);
_metrics.OutboxPendingEvents.Set(count);

KafkaConsumerLagCollector (every 30s)

Measures how far behind each Kafka consumer group is. Consumer lag is the number of messages that have been published but not yet processed. Think of it like a queue at a bank — lag is how many people are still waiting. This collector uses Kafka's admin API to check the lag for each consumer group without actually consuming any messages:

private static readonly string[] Groups =
{
    "es-indexer",
    "sepsis-engine",
    "notification-publisher",
    "data-lake-writer",
};

private async Task CollectGroupLagAsync(string groupId, CancellationToken ct)
{
    using var admin = new AdminClientBuilder(adminConfig).Build();

    // Get committed offsets for this consumer group
    var result = await admin.ListConsumerGroupOffsetsAsync(...);

    // Query high watermarks with a temporary consumer
    using var tempConsumer = new ConsumerBuilder<Ignore, Ignore>(config).Build();

    long totalLag = 0;
    foreach (var tpo in partitions)
    {
        var watermarks = tempConsumer.QueryWatermarkOffsets(
            tpo.TopicPartition, TimeSpan.FromSeconds(5));
        var lag = watermarks.High.Value - tpo.Offset.Value;
        totalLag += Math.Max(0L, lag);
    }

    _metrics.KafkaConsumerLag.WithLabels(groupId).Set(totalLag);
}

The lag is per consumer group. If sepsis-engine has high lag but es-indexer doesn't, the Grafana panel makes it immediately visible which pipeline is falling behind.

WardGatewayMetricsCollector (every 60s)

Queries PostgreSQL for gateway status and buffer depth:

var offlineBySite = await db.WardGateways
    .Include(g => g.Site)
    .Where(g => g.Status != GatewayStatus.Online)
    .GroupBy(g => g.Site.SiteCode)
    .Select(g => new { SiteCode = g.Key, Count = g.Count() })
    .ToListAsync(ct);

foreach (var row in offlineBySite)
    _metrics.WardGatewaysOffline.WithLabels(row.SiteCode).Set(row.Count);

var allGateways = await db.WardGateways.AsNoTracking().ToListAsync(ct);
foreach (var g in allGateways)
    _metrics.WardGatewayBufferDepth
        .WithLabels(g.GatewayCode, g.Department)
        .Set(g.ReportedBufferDepth);

All four collectors follow the same pattern:

1. Extend BackgroundService — .NET's base class for long-running background work
2. Use PeriodicTimer for the poll loop — fires a callback at a fixed interval (30s or 60s)
3. Create a DI scope per tick — in .NET's dependency injection (DI), database contexts are "scoped" (one per request/operation). Background services are singletons (one for the whole app), so they need to create a new scope each tick to get a fresh database context
4. Catch and log errors without crashing the collector — a transient database timeout shouldn't kill the metrics collection permanently

---

Step 6: Prometheus Configuration

File: infra/prometheus/prometheus.yml

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: vigilcare_api
    static_configs:
      - targets: ["host.docker.internal:5270"]
    metrics_path: /metrics

• scrape_interval: 15s: Prometheus pulls metrics every 15 seconds. This determines the resolution (granularity) of your data — you'll have one data point every 15 seconds. A shorter interval gives more detail but uses more storage and CPU.
• host.docker.internal:5270: The API runs on the host machine on port 5270. The extra_hosts entry in docker-compose.yml makes this hostname resolvable from inside the Prometheus container.
• metrics_path: /metrics: Matches the app.MapMetrics("/metrics") endpoint in Program.cs.

---

Step 7: Grafana Provisioning

Grafana is configured entirely through file provisioning — no manual setup needed after docker compose up.

Datasource: infra/grafana/provisioning/datasources/prometheus.yml

apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    url: http://prometheus:9090
    isDefault: true
    access: proxy

Uses the Docker service name prometheus (not localhost) because Grafana runs inside the Docker network.

Dashboard Provider: infra/grafana/provisioning/datasources/dashboards/config.yml

apiVersion: 1
providers:
  - name: VigilCare
    type: file
    options:
      path: /var/lib/grafana/dashboards

Tells Grafana to load all JSON files from /var/lib/grafana/dashboards, which is bind-mounted from ./infra/grafana/dashboards/.

---

Step 8: Grafana Dashboards

Clinical Operations Dashboard (vigilcare.json)

Panel	Type	Query	Purpose
Unacknowledged Critical Alerts	Stat	alerts_unacknowledged_gauge	Primary safety indicator
Observation Ingest Rate	Timeseries	rate(observations_ingested_total[1m])	Data flow health
Clinical Alerts by Type (5m)	Bar gauge	increase(clinical_alerts_total[5m])	Alert volume breakdown
Outbox Pending Events	Stat	outbox_pending_events	Outbox relay health
Kafka Consumer Lag by Group	Timeseries	kafka_consumer_lag	Per-pipeline lag
Ingest Latency p50 / p99	Timeseries	histogram_quantile(0.50, rate(...[5m])) / histogram_quantile(0.99, rate(...[5m]))	Performance SLA
Escalations Total	Stat	escalations_total	Missed alert tracking
Ward Gateways Offline	Stat	sum(ward_gateways_offline_gauge)	Edge connectivity
Gateway Buffer Depth	Bar gauge	ward_gateway_buffer_depth	Per-gateway backlog
Clinical Sync Batches (5m rate)	Timeseries	rate(clinical_sync_batches_total[5m])	Sync throughput
Total Sync Backlog	Stat	sum(ward_gateway_buffer_depth)	Fleet-wide backlog

Alert Quality Dashboard (alert-quality-dashboard.json)

Panel	Type	Query	Purpose
Acknowledgement Rate by Type	Timeseries	vigilcare_alert_acknowledgement_rate	Are alerts being seen?
False Positive Rate by Type	Timeseries	vigilcare_alert_false_positive_rate	Alert fatigue tracking
Useful Rate by Type	Timeseries	vigilcare_alert_useful_rate	Clinical value
Avg Ack Seconds by Type	Timeseries	vigilcare_alert_avg_ack_seconds	Response time SLA

---

Metrics Reference

All Counters

Metric	Labels	What It Tracks
observations_ingested_total	observation_code, source	Vital sign ingest volume
clinical_alerts_total	alert_type, severity	Alert generation rate
news2_scores_total	risk_level	NEWS2 scoring frequency
escalations_total	—	Unacknowledged alert escalations
trend_alerts_total	observation_code	Rapid deterioration detections
alert_suppressions_total	alert_type	Suppression window activations
qsofa_detections_total	—	qSOFA screen triggers
sepsis_bundle_compliance_total	status	Bundle compliance outcomes
gcs_scores_total	classification	GCS scoring by severity
sofa_scores_total	has_delta_alert	SOFA scoring with/without alerts
fhir_ingest_total	resource_type, outcome	FHIR inbound operations
fhir_read_total	resource_type, interaction, outcome	FHIR read/search operations
authorization_failures_total	permission, role	Failed auth attempts
fhir_mapping_errors_total	reason	FHIR mapping failures
phi_access_logs_total	access_type	PHI access audit entries
clinical_sync_batches_total	status	Gateway sync batch outcomes

All Histograms

Metric	Buckets	What It Tracks
observation_ingest_duration_seconds	5ms–1s	Full ingest transaction time
news2_scoring_duration_seconds	1ms–100ms	NEWS2 computation time
trend_analysis_duration_seconds	1ms–100ms	Trend evaluation time
sofa_scoring_duration_seconds	1ms–100ms	SOFA computation time
clinical_sync_batch_duration_seconds	100ms–10s	Batch processing time

All Gauges

Metric	Labels	Collector	Interval
alerts_unacknowledged_gauge	—	AlertsUnacknowledgedCollector	30s
kafka_consumer_lag	consumer_group	KafkaConsumerLagCollector	30s
outbox_pending_events	—	OutboxPendingCollector	30s
ward_gateways_offline_gauge	site_code	WardGatewayMetricsCollector	60s
ward_gateway_buffer_depth	gateway_code, department	WardGatewayMetricsCollector	60s
vigilcare_alert_acknowledgement_rate	alert_type	AlertQualityAggregatorService	60min
vigilcare_alert_false_positive_rate	alert_type	AlertQualityAggregatorService	60min
vigilcare_alert_useful_rate	alert_type	AlertQualityAggregatorService	60min
vigilcare_alert_avg_ack_seconds	alert_type	AlertQualityAggregatorService	60min

---

Accessing the Stack

Tool	URL	Credentials
Prometheus	http://localhost:9101	None
Grafana	http://localhost:3101	admin / admin
Raw metrics	http://localhost:5270/metrics	JWT or anonymous