feature: Observability: Prometheus Metrics and Grafana
This commit is contained in:
voltsrage
@@ -54,11 +54,11 @@ An `OutboxEvent` is written in the same transaction as any observation or alert,
|
||||
- **Kafka Pipeline** — three topics (`observation.recorded`, `alert.generated`, `encounter.status.changed`) with six partitions each; KRaft mode, no Zookeeper; `KAFKA_AUTO_CREATE_TOPICS_ENABLE=false` — topics are provisioned explicitly by `KafkaTopicProvisioner`
|
||||
- **Elasticsearch CQRS Projection** — `EsIndexerService` consumer group upserts `patient_encounters` documents, appends to the `observations` index, and updates `openAlertCount` on alert events; patient/encounter search; per-encounter observation trend (hourly avg/min/max); alert volume summary by department and severity; population query (numeric range aggregation across all patients)
|
||||
- **Sepsis Early Warning Engine** — `SepsisEngineService` Kafka consumer evaluates SIRS criteria (temperature, heart rate, respiratory rate, WBC) per encounter using Redis keys with a 30-minute TTL sliding window; on ≥2 active criteria, inserts a `SEPSIS_WARNING / CRITICAL` alert idempotently (`INSERT WHERE NOT EXISTS`)
|
||||
- **RabbitMQ Notification Workers** — `NotificationPublisherService` reads `alert.generated` from Kafka and publishes paging jobs to `alerts.paging.queue`; `PagingWorkerService` sends the page and waits for acknowledgment; NACK on timeout routes to `alerts.paging.dlq` with `x-message-ttl = 300000ms`; `EscalationWorkerService` pages the on-call backup and sets alert status to `escalated`; `DischargeSummaryWorkerService` reads `encounter.status.changed`, generates a discharge summary, and stores it in MinIO under `/discharge-summaries/{encounterId}/summary.pdf`
|
||||
- **RabbitMQ Notification Workers** — `NotificationPublisherService` reads `alert.generated` from Kafka and publishes paging jobs to `alerts.paging.queue`; `PagingWorkerService` sends the page and waits for acknowledgment; if no ack arrives before timeout it NACKs to `alerts.paging.dlq` with `x-message-ttl = 300000ms`; if the host is stopping, in-flight paging messages are NACKed with `requeue=true` so they are retried after restart and do not false-escalate; `EscalationWorkerService` pages the on-call backup and sets alert status to `escalated`; `DischargeSummaryWorkerService` reads `encounter.status.changed`, generates a discharge summary, and stores it in MinIO under `/discharge-summaries/{encounterId}/summary.pdf`
|
||||
- **Data Lake Writer** — Kafka consumer writing partitioned Parquet files to MinIO (`/observations/`, `/alerts/`, `/encounters/` by date); flush policy: 1,000 events or 5 minutes, whichever comes first; columnar format for 10-year regulatory retention
|
||||
- **Reconciliation Jobs** — three scheduled checks: (1) unacknowledged CRITICAL alerts older than 30 minutes, (2) pending orders without results after 4 hours, (3) active inpatients with no observation in 2 hours; each finding creates a `reconciliation_alerts` row and publishes to RabbitMQ
|
||||
- **Standard Envelope** — all responses use a consistent `{ success, statusCode, data, error }` wrapper; validation errors use the same shape; `ApiBehaviorOptions` overridden so model validation also produces the standard envelope
|
||||
- **Observability** — Serilog structured logging enriched with `correlationId`, `encounterId`, `patientId` on alert paths; Seq sink; per-request correlation IDs in request logs and response headers
|
||||
- **Observability** — Serilog structured logging enriched with `correlationId`, `encounterId`, `patientId` on alert paths; Seq sink (`http://localhost:5345`); Prometheus (`http://localhost:9101`) scrapes `GET /metrics`; Grafana dashboards (`http://localhost:3101`, admin/admin) for clinical metrics including `alerts_unacknowledged_gauge`; per-request correlation IDs in request logs and response headers
|
||||
- **Swagger UI** — OpenAPI spec via Swashbuckle (Development only)
|
||||
|
||||
---
|
||||
@@ -84,9 +84,10 @@ IHostedServices (background):
|
||||
EsIndexerService → Kafka → Elasticsearch (consumer group: es-indexer)
|
||||
SepsisEngineService → Kafka → Redis SIRS state → PostgreSQL alert (consumer group: sepsis-engine)
|
||||
NotificationPublisherService → Kafka → RabbitMQ paging.queue (consumer group: notification-publisher)
|
||||
PagingWorkerService → RabbitMQ paging.queue → log page → NACK on timeout
|
||||
PagingWorkerService → RabbitMQ paging.queue → log page → NACK on timeout (or requeue on shutdown)
|
||||
EscalationWorkerService → RabbitMQ escalation.queue → update alert status
|
||||
DischargeSummaryWorkerService → RabbitMQ discharge.queue → MinIO Parquet
|
||||
DischargeSummaryWorkerService → RabbitMQ discharge.queue → MinIO PDF
|
||||
ReconciliationScheduler → three scheduled safety checks → reconciliation_alerts + RabbitMQ
|
||||
```
|
||||
|
||||
**Why Kafka and RabbitMQ coexist:** Kafka is an append-only log — the same observation event reaches the Elasticsearch indexer, the sepsis engine, and the data lake independently without coordination. Each consumer holds its own offset and can replay from the beginning. RabbitMQ handles the action side: one message, one worker, one page. A duplicate page at 3am is a patient safety concern, not a minor inconvenience — RabbitMQ's acknowledgment-then-delete model is correct here. The DLQ TTL-based escalation has no equivalent in Kafka.
|
||||
@@ -105,6 +106,7 @@ IHostedServices (background):
|
||||
| Search / analytics | Elasticsearch 8.13 (CQRS read projection) |
|
||||
| Data lake | MinIO (Parquet, S3-compatible) |
|
||||
| Logging | Serilog + Seq sink |
|
||||
| Metrics / dashboards | Prometheus 2.52 + Grafana 10.4 |
|
||||
| Docs | Swagger / OpenAPI (Swashbuckle) |
|
||||
| Testing | xUnit + Testcontainers |
|
||||
|
||||
@@ -115,7 +117,7 @@ IHostedServices (background):
|
||||
```
|
||||
VigilCareClinicalAPI/
|
||||
├── Program.cs # Service registration, middleware, seed on startup
|
||||
├── appsettings.json # Connection strings, Kafka, Elasticsearch, RabbitMQ, MinIO, Serilog
|
||||
├── appsettings.json # Connection strings, Kafka, Elasticsearch, RabbitMQ, MinIO, Serilog, ReconciliationJobs
|
||||
├── Controllers/
|
||||
│ ├── PatientsController.cs # Patient CRUD, search by name/MRN
|
||||
│ ├── EncountersController.cs # Encounter open, status PATCH, timeline
|
||||
@@ -140,7 +142,7 @@ VigilCareClinicalAPI/
|
||||
│ ├── AlertStatus.cs # Open, Acknowledged, Resolved, Escalated
|
||||
│ ├── AlertType.cs # ThresholdBreach, SepsisWarning, …
|
||||
│ ├── ObservationSource.cs # Device, Manual, Lab
|
||||
│ └── OrderType.cs / ReconciliationCheckType.cs
|
||||
│ └── OrderType.cs / ReconciliationCheckType.cs / Department.cs / OrderStatus.cs
|
||||
├── Services/
|
||||
│ ├── PatientService.cs
|
||||
│ ├── EncounterService.cs # Status state machine + ConflictException on invalid transitions
|
||||
@@ -153,15 +155,22 @@ VigilCareClinicalAPI/
|
||||
├── BackgroundServices/
|
||||
│ ├── ThresholdCacheLoader.cs # Pre-loads all thresholds into Redis on startup
|
||||
│ ├── KafkaTopicProvisioner.cs # Creates topics with NumPartitions from config
|
||||
│ ├── ElasticIndexProvisioner.cs # Creates patient_encounters, observations, clinical_alerts indices
|
||||
│ ├── OutboxRelayService.cs # Polls outbox every 500ms; publishes to Kafka; marks processed
|
||||
│ ├── EsIndexerService.cs # consumer group: es-indexer; upserts Elasticsearch documents
|
||||
│ ├── ElasticsSearch/
|
||||
│ │ ├── ElasticIndexProvisioner.cs # Creates patient_encounters, observations, clinical_alerts indices
|
||||
│ │ └── EsIndexerService.cs # consumer group: es-indexer; upserts Elasticsearch documents
|
||||
│ ├── SepsisEngineService.cs # consumer group: sepsis-engine; SIRS eval via Redis TTL keys
|
||||
│ └── Notifications/
|
||||
│ ├── NotificationPublisherService.cs # consumer group: notification-publisher; alert.generated → RabbitMQ paging.queue
|
||||
│ ├── PagingWorkerService.cs # RabbitMQ consumer; logs page; NACK on ack timeout → DLQ
|
||||
│ ├── EscalationWorkerService.cs # RabbitMQ escalation.queue; logs escalation; sets alert.status = escalated
|
||||
│ └── DischargeSummaryWorkerService.cs # RabbitMQ discharge.queue; generates summary; uploads to MinIO
|
||||
│ ├── Notifications/
|
||||
│ │ ├── NotificationPublisherService.cs # consumer group: notification-publisher; alert.generated → RabbitMQ paging.queue
|
||||
│ │ ├── PagingWorkerService.cs # RabbitMQ consumer; logs page; NACK on ack timeout → DLQ, requeue on graceful shutdown
|
||||
│ │ ├── EscalationWorkerService.cs # RabbitMQ escalation.queue; logs escalation; sets alert.status = escalated
|
||||
│ │ └── DischargeSummaryWorkerService.cs # RabbitMQ discharge.queue; generates summary PDF; uploads to MinIO
|
||||
│ └── Reconciliation/
|
||||
│ ├── ReconciliationScheduler.cs # Runs three safety checks on a configurable interval
|
||||
│ ├── UnacknowledgedAlertsCheck.cs # CRITICAL alerts unacknowledged > 30 min
|
||||
│ ├── PendingOrdersCheck.cs # Pending orders without results > 4 hours
|
||||
│ ├── DisconnectedMonitorsCheck.cs # Active inpatients with no observation > 2 hours
|
||||
│ └── ReconciliationPublisher.cs # Publishes findings to notifications.reconciliation.queue
|
||||
├── Sepsis/
|
||||
│ ├── SirsDetector.cs # Redis SIRS state management (SET/DEL/MGET)
|
||||
│ └── SirsEvaluator.cs # Per-code criterion evaluation
|
||||
@@ -190,13 +199,21 @@ VigilCareClinicalAPI/
|
||||
│ └── ExceptionHandlerMiddleware.cs
|
||||
└── Migrations/
|
||||
|
||||
infra/
|
||||
├── prometheus/
|
||||
│ └── prometheus.yml # Scrape config for vigilcare_api /metrics
|
||||
└── grafana/
|
||||
├── provisioning/ # Datasource + dashboard provider config
|
||||
└── dashboards/ # vigilcare.json clinical dashboard
|
||||
|
||||
tests/
|
||||
└── VigilCareClinicalAPI.Tests/
|
||||
├── ObservationIngestTests.cs # Ingest happy path, critical alert creation, discharged encounter rejection, idempotency
|
||||
├── AlertLifecycleTests.cs # Acknowledge, resolve, escalation guard
|
||||
├── SirsDetectorTests.cs # Redis SIRS state SET/DEL/MGET logic
|
||||
├── SirsEvaluatorTests.cs # Per-code criterion evaluation
|
||||
└── NotificationPipelineTests.cs # RabbitMQ topology, DLQ routing
|
||||
├── NotificationPipelineTests.cs # RabbitMQ topology, DLQ routing
|
||||
└── ReconciliationTests.cs # Three reconciliation checks, deduplication, RabbitMQ publish
|
||||
```
|
||||
|
||||
---
|
||||
@@ -239,7 +256,7 @@ PostgreSQL is always the write side and the source of truth. Elasticsearch is a
|
||||
|
||||
### DLQ as a Clinical Escalation Protocol
|
||||
|
||||
The five-minute escalation is not a retry — it is a clinical workflow. When an alert is created, the paging worker sends a page to the attending physician. If no acknowledgment arrives within five minutes, the message NACKs to `alerts.paging.dlq` with `x-message-ttl = 300000ms`. After TTL expires, the DLQ re-routes to `alerts.escalation.queue` and the on-call backup is paged. This pattern has no equivalent in Kafka — Kafka has no concept of per-message TTL or conditional re-routing based on consumer acknowledgment.
|
||||
The five-minute escalation is not a retry — it is a clinical workflow. When an alert is created, the paging worker sends a page to the attending physician. If no acknowledgment arrives within five minutes, the message NACKs to `alerts.paging.dlq` with `x-message-ttl = 300000ms`. After TTL expires, the DLQ re-routes to `alerts.escalation.queue` and the on-call backup is paged. During graceful shutdown, cancellation of the in-flight wait loop is treated as non-failure and the message is NACKed with `requeue=true`, preventing false escalation during deploy/restart windows. This pattern has no equivalent in Kafka — Kafka has no concept of per-message TTL or conditional re-routing based on consumer acknowledgment.
|
||||
|
||||
### Why Not a Time-Series Database for Observations?
|
||||
|
||||
@@ -260,6 +277,8 @@ A medium hospital with 200 concurrent inpatients at five observations per patien
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
All services join the `vigilcare_net` bridge network so containers can reach each other by service name (e.g. Grafana → `http://prometheus:9090`). Connection strings in `appsettings.json` use **host** ports when running `dotnet run` on your machine.
|
||||
|
||||
| Service | Host Port | Notes |
|
||||
|---|---|---|
|
||||
| PostgreSQL 16 | 5436 | Database: `vigilcare`, user: `postgres`, password: `password` |
|
||||
@@ -269,9 +288,30 @@ docker compose up -d
|
||||
| Elasticsearch 8.13 | 9200 | Security disabled for development |
|
||||
| RabbitMQ 3.13 | 5674 (AMQP), 15674 (UI) | login: `guest` / `guest` |
|
||||
| MinIO | 9005 (S3 API), 9006 (console) | login: `minioadmin` / `minioadmin` |
|
||||
| Prometheus 2.52 | 9101 | UI at `http://localhost:9101` — scrapes `GET /metrics` on the API |
|
||||
| Grafana 10.4 | 3101 | UI at `http://localhost:3101` — login: `admin` / `admin` |
|
||||
|
||||
**Seq first-run:** `SEQ_FIRSTRUN_ADMINPASSWORD=admin` is set in `docker-compose.yml`. This password is only applied on the very first container start (when the `/data` volume is empty). After initialization, the password is stored in the volume and this env var is ignored.
|
||||
|
||||
### Docker notes for Linux
|
||||
|
||||
Prometheus scrapes the API using `host.docker.internal:5270`. On Linux, two things are required:
|
||||
|
||||
1. In `docker-compose.yml` under `prometheus`:
|
||||
```yaml
|
||||
extra_hosts:
|
||||
- "host.docker.internal:host-gateway"
|
||||
```
|
||||
2. Run the API bound to all interfaces (not only loopback), so containers can reach it:
|
||||
- Use `http://0.0.0.0:5270` (or `ASPNETCORE_URLS=http://0.0.0.0:5270`)
|
||||
|
||||
Without this, Prometheus may show target errors like:
|
||||
- `lookup host.docker.internal ... no such host` (DNS mapping missing), or
|
||||
- `dial tcp 172.17.0.1:5270: connect: connection refused` (API bound only to `127.0.0.1`)
|
||||
|
||||
For full Docker troubleshooting and recovery steps, see:
|
||||
- `docs/docker-compose-usage-and-troubleshooting.md`
|
||||
|
||||
### Install and Run
|
||||
|
||||
```bash
|
||||
@@ -286,6 +326,7 @@ On startup the application:
|
||||
3. Pre-loads all thresholds into Redis
|
||||
4. Provisions Kafka topics and Elasticsearch indices
|
||||
5. Declares the RabbitMQ exchange and queue topology
|
||||
6. Starts the reconciliation scheduler (three safety checks on a configurable interval)
|
||||
|
||||
Swagger UI is available at `http://localhost:<port>/swagger` in Development.
|
||||
|
||||
@@ -681,6 +722,7 @@ Exchange: `clinical.notifications.exchange` (direct)
|
||||
| `alerts.paging.dlq` | Dead-letter queue; `x-message-ttl = 300000ms` | → `alerts.escalation.queue` on TTL expiry |
|
||||
| `alerts.escalation.queue` | On-call backup paging | — |
|
||||
| `notifications.discharge.queue` | Discharge summary PDF generation + MinIO upload | — |
|
||||
| `notifications.reconciliation.queue` | Reconciliation safety findings from scheduled checks | — |
|
||||
| `notifications.appointment.queue` | Appointment reminder SMS | — |
|
||||
|
||||
---
|
||||
@@ -773,3 +815,6 @@ Observation history uses cursor pagination on `(recorded_at DESC, id DESC)`. Off
|
||||
| 4 | Elasticsearch CQRS projection (`EsIndexerService`); patient search; observation trend; alert summary; population aggregation; replay procedure | Done |
|
||||
| 5 | Sepsis detection engine (`SepsisEngineService`); Redis SIRS state with 30-min TTL; idempotent alert creation; integration tests | Done |
|
||||
| 6 | RabbitMQ exchange and queue topology; `NotificationPublisherService`; `PagingWorkerService`; DLQ escalation (`EscalationWorkerService`); discharge summary (`DischargeSummaryWorkerService` → MinIO); integration tests | Done |
|
||||
| 7 | Reconciliation scheduler — unacknowledged critical alerts, stale pending orders, disconnected monitors; `reconciliation_alerts` table; RabbitMQ publish; integration tests | Done |
|
||||
| 8 | Prometheus metrics (`GET /metrics`); Grafana dashboards; eight application metric families | In progress |
|
||||
| 9 | Data lake writer — Kafka consumer group `data-lake-writer`; Parquet flush to MinIO | Planned |
|
||||
|
||||
@@ -0,0 +1,181 @@
|
||||
using System.Globalization;
|
||||
using System.Net.Http.Json;
|
||||
using Microsoft.EntityFrameworkCore;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using StackExchange.Redis;
|
||||
|
||||
public class ObservabilityPhase8Tests : IClassFixture<ApiFixture>
|
||||
{
|
||||
private static readonly string[] ExpectedMetrics =
|
||||
{
|
||||
"observations_ingested_total",
|
||||
"observation_ingest_duration_seconds",
|
||||
"clinical_alerts_total",
|
||||
"alerts_unacknowledged_gauge",
|
||||
"kafka_consumer_lag",
|
||||
"outbox_pending_events",
|
||||
"sirs_detections_total",
|
||||
"escalations_total",
|
||||
};
|
||||
|
||||
private readonly ApiFixture _fixture;
|
||||
private readonly HttpClient _http;
|
||||
|
||||
public ObservabilityPhase8Tests(ApiFixture fixture)
|
||||
{
|
||||
_fixture = fixture;
|
||||
_http = fixture.CreateClient();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task MetricsEndpoint_ReturnsAllEightMetricFamilies()
|
||||
{
|
||||
var resp = await _http.GetAsync("/metrics");
|
||||
resp.EnsureSuccessStatusCode();
|
||||
|
||||
Assert.Equal("text/plain", resp.Content.Headers.ContentType?.MediaType);
|
||||
|
||||
var body = await resp.Content.ReadAsStringAsync();
|
||||
|
||||
foreach (var metric in ExpectedMetrics)
|
||||
{
|
||||
Assert.Contains(metric, body);
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task CorrelationMiddleware_AddsXCorrelationIdHeader()
|
||||
{
|
||||
var resp = await _http.GetAsync("/api/v1/alert-thresholds");
|
||||
resp.EnsureSuccessStatusCode();
|
||||
|
||||
Assert.True(resp.Headers.Contains("X-Correlation-Id"),
|
||||
"X-Correlation-Id response header is missing.");
|
||||
var value = resp.Headers.GetValues("X-Correlation-Id").First();
|
||||
Assert.NotEmpty(value);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task CorrelationMiddleware_EchoesIncomingCorrelationId()
|
||||
{
|
||||
var correlationId = "test-correlation-abc123";
|
||||
var req = new HttpRequestMessage(HttpMethod.Get, "/api/v1/alert-thresholds");
|
||||
req.Headers.Add("X-Correlation-Id", correlationId);
|
||||
|
||||
var resp = await _http.SendAsync(req);
|
||||
resp.EnsureSuccessStatusCode();
|
||||
|
||||
var echoed = resp.Headers.GetValues("X-Correlation-Id").First();
|
||||
Assert.Equal(correlationId, echoed);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ObservationIngest_IncrementsObservationsIngestedTotal()
|
||||
{
|
||||
var beforeBody = await (await _http.GetAsync("/metrics")).Content.ReadAsStringAsync();
|
||||
var before = ParseCounterValue(beforeBody, "observations_ingested_total");
|
||||
|
||||
await EnsureHeartRateThresholdAsync();
|
||||
var encounterId = await CreateActiveEncounterAsync();
|
||||
|
||||
var ingestResp = await _http.PostAsJsonAsync(
|
||||
$"/api/v1/encounters/{encounterId}/observations",
|
||||
new BatchIngestRequest(new List<IngestObservationRequest>
|
||||
{
|
||||
new("HEART_RATE", 72, "bpm", ObservationSource.Device, DateTimeOffset.UtcNow, null)
|
||||
}));
|
||||
ingestResp.EnsureSuccessStatusCode();
|
||||
|
||||
var afterBody = await (await _http.GetAsync("/metrics")).Content.ReadAsStringAsync();
|
||||
var after = ParseCounterValue(afterBody, "observations_ingested_total");
|
||||
|
||||
Assert.True(after > before,
|
||||
$"observations_ingested_total did not increase. Before={before} After={after}");
|
||||
}
|
||||
|
||||
private async Task EnsureHeartRateThresholdAsync()
|
||||
{
|
||||
using var scope = _fixture.Services.CreateScope();
|
||||
var db = scope.ServiceProvider.GetRequiredService<AppDbContext>();
|
||||
|
||||
var existing = await db.AlertThresholds
|
||||
.FirstOrDefaultAsync(t => t.ObservationCode == "HEART_RATE");
|
||||
|
||||
if (existing is null)
|
||||
{
|
||||
db.AlertThresholds.Add(new AlertThreshold
|
||||
{
|
||||
Id = Guid.NewGuid(),
|
||||
ObservationCode = "HEART_RATE",
|
||||
DisplayName = "Heart Rate",
|
||||
Unit = "bpm",
|
||||
CriticalLow = 30,
|
||||
WarningLow = 50,
|
||||
WarningHigh = 100,
|
||||
CriticalHigh = 150,
|
||||
CreatedAt = DateTimeOffset.UtcNow
|
||||
});
|
||||
await db.SaveChangesAsync();
|
||||
}
|
||||
|
||||
var redis = scope.ServiceProvider.GetRequiredService<IConnectionMultiplexer>();
|
||||
var cache = redis.GetDatabase(1);
|
||||
await cache.StringSetAsync("threshold:HEART_RATE",
|
||||
"""{"ObservationCode":"HEART_RATE","CriticalLow":30,"WarningLow":50,"WarningHigh":100,"CriticalHigh":150}""");
|
||||
}
|
||||
|
||||
// Parse the sum of all label combinations for a counter family.
|
||||
private static double ParseCounterValue(string metricsBody, string metricName)
|
||||
{
|
||||
double total = 0;
|
||||
foreach (var rawLine in metricsBody.Split('\n', StringSplitOptions.RemoveEmptyEntries))
|
||||
{
|
||||
var line = rawLine.Trim();
|
||||
if (line.StartsWith('#')) continue;
|
||||
if (!line.StartsWith(metricName, StringComparison.Ordinal)) continue;
|
||||
|
||||
var valueToken = line.Split(' ', StringSplitOptions.RemoveEmptyEntries).LastOrDefault();
|
||||
if (valueToken is null) continue;
|
||||
if (double.TryParse(valueToken, NumberStyles.Float, CultureInfo.InvariantCulture, out var value))
|
||||
{
|
||||
total += value;
|
||||
}
|
||||
}
|
||||
return total;
|
||||
}
|
||||
|
||||
private async Task<Guid> CreateActiveEncounterAsync()
|
||||
{
|
||||
using var scope = _fixture.Services.CreateScope();
|
||||
var db = scope.ServiceProvider.GetRequiredService<AppDbContext>();
|
||||
|
||||
var patient = new Patient
|
||||
{
|
||||
Id = Guid.NewGuid(),
|
||||
Mrn = $"MRN-METRICS-{Guid.NewGuid():N}"[..18],
|
||||
FirstName = "Metrics",
|
||||
LastName = "Test",
|
||||
DateOfBirth = new DateOnly(1975, 3, 20),
|
||||
Gender = "Female",
|
||||
CreatedAt = DateTimeOffset.UtcNow
|
||||
};
|
||||
|
||||
var encounter = new Encounter
|
||||
{
|
||||
Id = Guid.NewGuid(),
|
||||
PatientId = patient.Id,
|
||||
EncounterType = EncounterType.Inpatient,
|
||||
Status = EncounterStatus.Active,
|
||||
Department = Department.Icu,
|
||||
AttendingPhysician = "Dr. Osei",
|
||||
AdmittedAt = DateTimeOffset.UtcNow,
|
||||
CreatedAt = DateTimeOffset.UtcNow
|
||||
};
|
||||
|
||||
db.Patients.Add(patient);
|
||||
db.Encounters.Add(encounter);
|
||||
await db.SaveChangesAsync();
|
||||
|
||||
return encounter.Id;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,54 @@
|
||||
using Microsoft.EntityFrameworkCore;
|
||||
|
||||
public sealed class AlertsUnacknowledgedCollector : BackgroundService
|
||||
{
|
||||
// 5 minutes matches the DLQ TTL — an alert that survived escalation is still open.
|
||||
private static readonly TimeSpan UnacknowledgedThreshold = TimeSpan.FromMinutes(5);
|
||||
|
||||
private readonly IServiceScopeFactory _scopes;
|
||||
private readonly ClinicalMetrics _metrics;
|
||||
private readonly ILogger<AlertsUnacknowledgedCollector> _logger;
|
||||
|
||||
public AlertsUnacknowledgedCollector(
|
||||
IServiceScopeFactory scopes,
|
||||
ClinicalMetrics metrics,
|
||||
ILogger<AlertsUnacknowledgedCollector> logger)
|
||||
{
|
||||
_scopes = scopes;
|
||||
_metrics = metrics;
|
||||
_logger = logger;
|
||||
}
|
||||
|
||||
protected override async Task ExecuteAsync(CancellationToken ct)
|
||||
{
|
||||
using var timer = new PeriodicTimer(TimeSpan.FromSeconds(30));
|
||||
while (await timer.WaitForNextTickAsync(ct))
|
||||
await CollectAsync(ct);
|
||||
}
|
||||
|
||||
private async Task CollectAsync(CancellationToken ct)
|
||||
{
|
||||
try
|
||||
{
|
||||
await using var scope = _scopes.CreateAsyncScope();
|
||||
var db = scope.ServiceProvider.GetRequiredService<AppDbContext>();
|
||||
var cutoff = DateTimeOffset.UtcNow - UnacknowledgedThreshold;
|
||||
|
||||
var count = await db.ClinicalAlerts
|
||||
.CountAsync(a => a.Severity == AlertSeverity.Critical
|
||||
&& a.Status == AlertStatus.Open
|
||||
&& a.TriggeredAt < cutoff, ct);
|
||||
|
||||
_metrics.AlertsUnacknowledgedGauge.Set(count);
|
||||
|
||||
if (count > 0)
|
||||
_logger.LogWarning(
|
||||
"[PATIENT-SAFETY] alerts_unacknowledged_gauge={Count} " +
|
||||
"(CRITICAL alerts open > 5 min)", count);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogError(ex, "AlertsUnacknowledgedCollector failed");
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,91 @@
|
||||
using Confluent.Kafka;
|
||||
using Confluent.Kafka.Admin;
|
||||
using Microsoft.Extensions.Options;
|
||||
|
||||
public sealed class KafkaConsumerLagCollector : BackgroundService
|
||||
{
|
||||
private static readonly string[] Groups =
|
||||
{
|
||||
"es-indexer",
|
||||
"sepsis-engine",
|
||||
"notification-publisher",
|
||||
"data-lake-writer",
|
||||
};
|
||||
|
||||
private readonly KafkaOptions _kafkaOptions;
|
||||
private readonly ClinicalMetrics _metrics;
|
||||
private readonly ILogger<KafkaConsumerLagCollector> _logger;
|
||||
|
||||
public KafkaConsumerLagCollector(
|
||||
IOptions<KafkaOptions> kafkaOptions,
|
||||
ClinicalMetrics metrics,
|
||||
ILogger<KafkaConsumerLagCollector> logger)
|
||||
{
|
||||
_kafkaOptions = kafkaOptions.Value;
|
||||
_metrics = metrics;
|
||||
_logger = logger;
|
||||
}
|
||||
|
||||
protected override async Task ExecuteAsync(CancellationToken ct)
|
||||
{
|
||||
// Initial delay so Kafka is reachable before the first collection.
|
||||
await Task.Delay(TimeSpan.FromSeconds(20), ct);
|
||||
|
||||
using var timer = new PeriodicTimer(TimeSpan.FromSeconds(30));
|
||||
while (await timer.WaitForNextTickAsync(ct))
|
||||
{
|
||||
foreach (var group in Groups)
|
||||
{
|
||||
try { await CollectGroupLagAsync(group, ct); }
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogDebug(ex, "KafkaConsumerLagCollector: failed for group {Group}", group);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private async Task CollectGroupLagAsync(string groupId, CancellationToken ct)
|
||||
{
|
||||
var adminConfig = new AdminClientConfig
|
||||
{ BootstrapServers = _kafkaOptions.BootstrapServers };
|
||||
|
||||
using var admin = new AdminClientBuilder(adminConfig).Build();
|
||||
|
||||
// List the committed offsets for this consumer group across all its partitions.
|
||||
var result = await admin.ListConsumerGroupOffsetsAsync(
|
||||
new[] { new ConsumerGroupTopicPartitions(groupId, null) },
|
||||
new ListConsumerGroupOffsetsOptions { RequireStableOffsets = false });
|
||||
|
||||
var partitions = result.FirstOrDefault()?.Partitions ?? [];
|
||||
if (!partitions.Any())
|
||||
{
|
||||
_metrics.KafkaConsumerLag.WithLabels(groupId).Set(0);
|
||||
return;
|
||||
}
|
||||
|
||||
// Query high watermarks using a temporary consumer (does not join the group).
|
||||
using var tempConsumer = new ConsumerBuilder<Ignore, Ignore>(new ConsumerConfig
|
||||
{
|
||||
BootstrapServers = _kafkaOptions.BootstrapServers,
|
||||
GroupId = $"__lag-probe",
|
||||
}).Build();
|
||||
|
||||
long totalLag = 0;
|
||||
foreach (var tpo in partitions)
|
||||
{
|
||||
if (tpo.Error.IsError || tpo.Offset == Offset.Unset) continue;
|
||||
|
||||
var watermarks = tempConsumer.QueryWatermarkOffsets(
|
||||
tpo.TopicPartition, TimeSpan.FromSeconds(5));
|
||||
|
||||
// Committed offset is the next offset the consumer will read.
|
||||
// High watermark is the latest available offset + 1.
|
||||
// Lag = messages the consumer has not yet read.
|
||||
var lag = watermarks.High.Value - tpo.Offset.Value;
|
||||
totalLag += Math.Max(0L, lag);
|
||||
}
|
||||
|
||||
_metrics.KafkaConsumerLag.WithLabels(groupId).Set(totalLag);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,42 @@
|
||||
using Microsoft.EntityFrameworkCore;
|
||||
|
||||
public sealed class OutboxPendingCollector : BackgroundService
|
||||
{
|
||||
private readonly IServiceScopeFactory _scopes;
|
||||
private readonly ClinicalMetrics _metrics;
|
||||
private readonly ILogger<OutboxPendingCollector> _logger;
|
||||
|
||||
public OutboxPendingCollector(
|
||||
IServiceScopeFactory scopes,
|
||||
ClinicalMetrics metrics,
|
||||
ILogger<OutboxPendingCollector> logger)
|
||||
{
|
||||
_scopes = scopes;
|
||||
_metrics = metrics;
|
||||
_logger = logger;
|
||||
}
|
||||
|
||||
protected override async Task ExecuteAsync(CancellationToken ct)
|
||||
{
|
||||
using var timer = new PeriodicTimer(TimeSpan.FromSeconds(30));
|
||||
while (await timer.WaitForNextTickAsync(ct))
|
||||
await CollectAsync(ct);
|
||||
}
|
||||
|
||||
private async Task CollectAsync(CancellationToken ct)
|
||||
{
|
||||
try
|
||||
{
|
||||
await using var scope = _scopes.CreateAsyncScope();
|
||||
var db = scope.ServiceProvider.GetRequiredService<AppDbContext>();
|
||||
var count = await db.OutboxEvents
|
||||
.CountAsync(e => e.ProcessedAt == null, ct);
|
||||
|
||||
_metrics.OutboxPendingEvents.Set(count);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogError(ex, "OutboxPendingCollector failed");
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -3,21 +3,25 @@ using System.Text.Json;
|
||||
using Microsoft.Extensions.Options;
|
||||
using RabbitMQ.Client;
|
||||
using RabbitMQ.Client.Events;
|
||||
using Serilog.Context;
|
||||
|
||||
public sealed class EscalationWorkerService : BackgroundService
|
||||
{
|
||||
private readonly IOptions<RabbitMqOptions> _opts;
|
||||
private readonly IServiceScopeFactory _scopes;
|
||||
private readonly ClinicalMetrics _metrics;
|
||||
private readonly ILogger<EscalationWorkerService> _logger;
|
||||
|
||||
public EscalationWorkerService(
|
||||
IOptions<RabbitMqOptions> opts,
|
||||
IServiceScopeFactory scopes,
|
||||
ClinicalMetrics metrics,
|
||||
ILogger<EscalationWorkerService> logger)
|
||||
{
|
||||
_opts = opts;
|
||||
_scopes = scopes;
|
||||
_logger = logger;
|
||||
_opts = opts;
|
||||
_scopes = scopes;
|
||||
_metrics = metrics;
|
||||
_logger = logger;
|
||||
}
|
||||
|
||||
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
|
||||
@@ -57,15 +61,20 @@ public sealed class EscalationWorkerService : BackgroundService
|
||||
var payload = Encoding.UTF8.GetString(ea.Body.Span);
|
||||
var doc = JsonDocument.Parse(payload);
|
||||
var alertId = Guid.Parse(doc.RootElement.GetProperty("alertId").GetString()!);
|
||||
var encounterId = doc.RootElement.GetProperty("encounterId").GetString();
|
||||
var encounterId = Guid.Parse(doc.RootElement.GetProperty("encounterId").GetString()!);
|
||||
|
||||
_logger.LogCritical(
|
||||
"[ESCALATION] Paging on-call backup — AlertId={AlertId} EncounterId={EncounterId}",
|
||||
alertId, encounterId);
|
||||
using (LogContext.PushProperty("EncounterId", encounterId))
|
||||
using (LogContext.PushProperty("AlertId", alertId))
|
||||
{
|
||||
_logger.LogCritical("[ESCALATION] Paging on-call backup for alert {AlertId}", alertId);
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
await UpdateAlertStatusEscalatedAsync(alertId, ct);
|
||||
var escalated = await UpdateAlertStatusEscalatedAsync(alertId, ct);
|
||||
if (escalated)
|
||||
_metrics.EscalationsTotal.Inc();
|
||||
|
||||
channel.BasicAck(ea.DeliveryTag, multiple: false);
|
||||
|
||||
_logger.LogWarning(
|
||||
@@ -78,18 +87,19 @@ public sealed class EscalationWorkerService : BackgroundService
|
||||
}
|
||||
}
|
||||
|
||||
private async Task UpdateAlertStatusEscalatedAsync(Guid alertId, CancellationToken ct)
|
||||
private async Task<bool> UpdateAlertStatusEscalatedAsync(Guid alertId, CancellationToken ct)
|
||||
{
|
||||
await using var scope = _scopes.CreateAsyncScope();
|
||||
var db = scope.ServiceProvider.GetRequiredService<AppDbContext>();
|
||||
|
||||
var alert = await db.ClinicalAlerts.FindAsync(new object[] { alertId }, ct);
|
||||
if (alert is null) return;
|
||||
if (alert is null) return false;
|
||||
|
||||
// Only escalate if still open — if acknowledged between NACK and TTL expiry, leave it.
|
||||
if (alert.Status != AlertStatus.Open) return;
|
||||
if (alert.Status != AlertStatus.Open) return false;
|
||||
|
||||
alert.Status = AlertStatus.Escalated;
|
||||
await db.SaveChangesAsync(ct);
|
||||
return true;
|
||||
}
|
||||
}
|
||||
@@ -51,6 +51,12 @@ public sealed class PagingWorkerService : BackgroundService
|
||||
{
|
||||
await HandlePageAsync(channel, ea, stoppingToken);
|
||||
}
|
||||
catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested)
|
||||
{
|
||||
// Host is stopping while we were waiting for ack — requeue so restart does not false-escalate.
|
||||
_logger.LogInformation("PagingWorker stopping — requeueing in-flight page message");
|
||||
channel.BasicNack(ea.DeliveryTag, multiple: false, requeue: true);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogError(ex, "PagingWorker failed — NACKing to DLQ");
|
||||
|
||||
@@ -0,0 +1,68 @@
|
||||
using Prometheus;
|
||||
|
||||
public sealed class ClinicalMetrics
|
||||
{
|
||||
// --- Counters ---
|
||||
|
||||
// Labeled by observation_code and source so clinicians can see which device types
|
||||
// and which codes dominate the ingest volume.
|
||||
public readonly Counter ObservationsIngestedTotal = Metrics.CreateCounter(
|
||||
"observations_ingested_total",
|
||||
"Total observations ingested, labeled by observation code and source.",
|
||||
labelNames: new[] { "observation_code", "source" });
|
||||
|
||||
// Labeled by alert_type (THRESHOLD_BREACH, SEPSIS_WARNING) and severity
|
||||
// (Critical, Warning) so the dashboard can show Critical vs Warning rates separately.
|
||||
public readonly Counter ClinicalAlertsTotal = Metrics.CreateCounter(
|
||||
"clinical_alerts_total",
|
||||
"Total clinical alerts generated, labeled by type and severity.",
|
||||
labelNames: new[] { "alert_type", "severity" });
|
||||
|
||||
// Incremented only when INSERT WHERE NOT EXISTS succeeds — duplicate-suppressed
|
||||
// SIRS detections do not count. This is the true detection rate, not the evaluation rate.
|
||||
public readonly Counter SirsDetectionsTotal = Metrics.CreateCounter(
|
||||
"sirs_detections_total",
|
||||
"Total SEPSIS_WARNING alerts generated by the sepsis detection engine.");
|
||||
|
||||
// Incremented by EscalationWorkerService when it processes a message from
|
||||
// alerts.escalation.queue. A rising escalations_total is the strongest operational
|
||||
// signal that critical alerts are not being acknowledged by the attending physician.
|
||||
public readonly Counter EscalationsTotal = Metrics.CreateCounter(
|
||||
"escalations_total",
|
||||
"Total alert escalations processed through the DLQ escalation path.");
|
||||
|
||||
// --- Histograms ---
|
||||
|
||||
// Measures the full ingest transaction: Redis cache lookup + alert evaluation +
|
||||
// outbox write + COMMIT. The 99th percentile matters for patient safety —
|
||||
// a slow ingest path delays the critical alert creation.
|
||||
public readonly Histogram ObservationIngestDuration = Metrics.CreateHistogram(
|
||||
"observation_ingest_duration_seconds",
|
||||
"Ingest transaction duration from request receipt to COMMIT.",
|
||||
new HistogramConfiguration
|
||||
{
|
||||
Buckets = new[] { 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0 }
|
||||
});
|
||||
|
||||
// --- Gauges (set by background collectors, not incremented inline) ---
|
||||
|
||||
// The most clinically significant panel. A non-zero value means a patient's
|
||||
// critical alert has gone unacknowledged for more than 5 minutes.
|
||||
// In a real deployment this panel drives an on-call pager alert at the nurse station.
|
||||
public readonly Gauge AlertsUnacknowledgedGauge = Metrics.CreateGauge(
|
||||
"alerts_unacknowledged_gauge",
|
||||
"Count of open CRITICAL alerts older than 5 minutes with no acknowledgment.");
|
||||
|
||||
// Per consumer group so the dashboard can show whether es-indexer, sepsis-engine,
|
||||
// or data-lake-writer is falling behind the observation stream.
|
||||
public readonly Gauge KafkaConsumerLag = Metrics.CreateGauge(
|
||||
"kafka_consumer_lag",
|
||||
"Approximate consumer group lag in messages, labeled by consumer group.",
|
||||
labelNames: new[] { "consumer_group" });
|
||||
|
||||
// An outbox that is growing means the relay is not keeping up or Kafka is unavailable.
|
||||
// In a patient safety system, a growing outbox delays alert delivery to all consumers.
|
||||
public readonly Gauge OutboxPendingEvents = Metrics.CreateGauge(
|
||||
"outbox_pending_events",
|
||||
"Count of outbox events not yet relayed to Kafka.");
|
||||
}
|
||||
@@ -1,5 +1,6 @@
|
||||
using Elastic.Clients.Elasticsearch;
|
||||
using Microsoft.EntityFrameworkCore;
|
||||
using Prometheus;
|
||||
using Serilog;
|
||||
using StackExchange.Redis;
|
||||
using System.Text.Json.Serialization;
|
||||
@@ -19,7 +20,9 @@ try
|
||||
builder.Host.UseSerilog((ctx, services, config) =>
|
||||
config.ReadFrom.Configuration(ctx.Configuration)
|
||||
.ReadFrom.Services(services)
|
||||
.Enrich.FromLogContext());
|
||||
.Enrich.FromLogContext()
|
||||
.Enrich.WithMachineName()
|
||||
.Enrich.WithThreadId());
|
||||
}
|
||||
|
||||
builder.Services.AddDbContext<AppDbContext>(opts =>
|
||||
@@ -77,6 +80,10 @@ try
|
||||
builder.Services.AddHostedService<EscalationWorkerService>();
|
||||
builder.Services.AddHostedService<DischargeSummaryWorkerService>();
|
||||
builder.Services.AddHostedService<ReconciliationScheduler>();
|
||||
builder.Services.AddSingleton<ClinicalMetrics>();
|
||||
builder.Services.AddHostedService<AlertsUnacknowledgedCollector>();
|
||||
builder.Services.AddHostedService<OutboxPendingCollector>();
|
||||
builder.Services.AddHostedService<KafkaConsumerLagCollector>();
|
||||
|
||||
builder.Services.AddControllers()
|
||||
.AddJsonOptions(opts =>
|
||||
@@ -118,6 +125,7 @@ try
|
||||
app.UseSwaggerUI();
|
||||
}
|
||||
|
||||
app.MapMetrics("/metrics");
|
||||
app.MapControllers();
|
||||
app.Run();
|
||||
|
||||
|
||||
@@ -14,7 +14,7 @@
|
||||
"dotnetRunMessages": true,
|
||||
"launchBrowser": true,
|
||||
"launchUrl": "swagger",
|
||||
"applicationUrl": "http://localhost:5270",
|
||||
"applicationUrl": "http://0.0.0.0:5270",
|
||||
"environmentVariables": {
|
||||
"ASPNETCORE_ENVIRONMENT": "Development"
|
||||
}
|
||||
@@ -24,7 +24,7 @@
|
||||
"dotnetRunMessages": true,
|
||||
"launchBrowser": true,
|
||||
"launchUrl": "swagger",
|
||||
"applicationUrl": "https://localhost:7146;http://localhost:5270",
|
||||
"applicationUrl": "https://localhost:7146;http://0.0.0.0:5270",
|
||||
"environmentVariables": {
|
||||
"ASPNETCORE_ENVIRONMENT": "Development"
|
||||
}
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
using System.Text.Json;
|
||||
using Microsoft.EntityFrameworkCore;
|
||||
using Serilog.Context;
|
||||
using StackExchange.Redis;
|
||||
|
||||
public class SirsDetector
|
||||
@@ -12,15 +13,18 @@ public class SirsDetector
|
||||
private readonly IConnectionMultiplexer _redis;
|
||||
private readonly IServiceProvider _services;
|
||||
private readonly ILogger<SirsDetector> _logger;
|
||||
private readonly ClinicalMetrics _metrics;
|
||||
|
||||
public SirsDetector(
|
||||
IConnectionMultiplexer redis,
|
||||
IServiceProvider services,
|
||||
ILogger<SirsDetector> logger)
|
||||
ILogger<SirsDetector> logger,
|
||||
ClinicalMetrics metrics)
|
||||
{
|
||||
_redis = redis;
|
||||
_services = services;
|
||||
_logger = logger;
|
||||
_metrics = metrics;
|
||||
}
|
||||
|
||||
public async Task<SirsResult> ProcessObservationAsync(
|
||||
@@ -151,10 +155,18 @@ public class SirsDetector
|
||||
await db.SaveChangesAsync(ct);
|
||||
await tx.CommitAsync(ct);
|
||||
|
||||
_logger.LogWarning(
|
||||
"SEPSIS_WARNING alert {AlertId} created for encounter {EncounterId} " +
|
||||
"— {Active}/4 SIRS criteria active",
|
||||
alertId, encounterId, activeCount);
|
||||
_metrics.SirsDetectionsTotal.Inc();
|
||||
_metrics.ClinicalAlertsTotal
|
||||
.WithLabels(AlertType.SepsisWarning.ToDbString(), AlertSeverity.Critical.ToDbString())
|
||||
.Inc();
|
||||
|
||||
using (LogContext.PushProperty("EncounterId", encounterId))
|
||||
using (LogContext.PushProperty("PatientId", patientId))
|
||||
{
|
||||
_logger.LogWarning(
|
||||
"SEPSIS_WARNING created. ActiveCriteriaCount={Count} AlertId={AlertId}",
|
||||
activeCount, alertId);
|
||||
}
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
using System.Text.Json;
|
||||
using Microsoft.EntityFrameworkCore;
|
||||
using Prometheus;
|
||||
using Serilog.Context;
|
||||
using StackExchange.Redis;
|
||||
|
||||
public class ObservationService : IObservationService
|
||||
@@ -7,15 +9,18 @@ public class ObservationService : IObservationService
|
||||
private readonly AppDbContext _db;
|
||||
private readonly IConnectionMultiplexer _redis;
|
||||
private readonly ILogger<ObservationService> _logger;
|
||||
private readonly ClinicalMetrics _metrics;
|
||||
|
||||
public ObservationService(
|
||||
AppDbContext db,
|
||||
IConnectionMultiplexer redis,
|
||||
ILogger<ObservationService> logger)
|
||||
ILogger<ObservationService> logger,
|
||||
ClinicalMetrics metrics)
|
||||
{
|
||||
_db = db;
|
||||
_redis = redis;
|
||||
_logger = logger;
|
||||
_metrics = metrics;
|
||||
}
|
||||
|
||||
public async Task<IngestResult> IngestAsync(Guid encounterId, IngestObservationRequest req)
|
||||
@@ -56,114 +61,133 @@ public class ObservationService : IObservationService
|
||||
if (!PlausibilityValidator.IsPlausible(req.ObservationCode, req.Value, out var reason))
|
||||
throw new ValidationException(reason!, "OBSERVATION_OUT_OF_PLAUSIBLE_RANGE");
|
||||
|
||||
// Steps 4–8 are one atomic transaction
|
||||
await using var tx = await _db.Database.BeginTransactionAsync();
|
||||
try
|
||||
using (LogContext.PushProperty("EncounterId", encounterId))
|
||||
using (LogContext.PushProperty("PatientId", encounter.PatientId))
|
||||
{
|
||||
// Step 4 — insert observation
|
||||
var observation = new Observation
|
||||
using var timer = _metrics.ObservationIngestDuration.NewTimer();
|
||||
|
||||
await using var tx = await _db.Database.BeginTransactionAsync();
|
||||
try
|
||||
{
|
||||
Id = Guid.NewGuid(),
|
||||
EncounterId = encounterId,
|
||||
ObservationCode = req.ObservationCode,
|
||||
Value = req.Value,
|
||||
Unit = req.Unit,
|
||||
Source = req.Source,
|
||||
IdempotencyKey = req.IdempotencyKey,
|
||||
RecordedAt = req.RecordedAt,
|
||||
CreatedAt = DateTimeOffset.UtcNow
|
||||
};
|
||||
_db.Observations.Add(observation);
|
||||
|
||||
// Step 5 — load threshold from Redis; fall back to PostgreSQL on miss
|
||||
var threshold = await LoadThresholdAsync(req.ObservationCode);
|
||||
|
||||
if (threshold is null)
|
||||
throw new ValidationException(
|
||||
$"No alert threshold is configured for observation code '{req.ObservationCode}'. " +
|
||||
"Register a threshold before recording observations for this code.",
|
||||
"UNKNOWN_OBSERVATION_CODE");
|
||||
|
||||
ClinicalAlert? alert = null;
|
||||
|
||||
// Step 6 — critical threshold detection (synchronous)
|
||||
// WARNING detection is intentionally deferred to the Kafka consumer.
|
||||
// A critical potassium of 2.1 mEq/L is immediately life-threatening — the alert
|
||||
// must exist before this API call returns. A warning heart rate of 95 bpm warrants
|
||||
// attention but not an emergency page; the additional Kafka latency is clinically safe.
|
||||
if (IsCriticalBreach(req.Value, threshold))
|
||||
{
|
||||
alert = new ClinicalAlert
|
||||
// Step 4 — insert observation
|
||||
var observation = new Observation
|
||||
{
|
||||
Id = Guid.NewGuid(),
|
||||
EncounterId = encounterId,
|
||||
PatientId = encounter.PatientId,
|
||||
ObservationId = observation.Id,
|
||||
AlertType = AlertTypeExtensions.CriticalFor(req.ObservationCode),
|
||||
Severity = AlertSeverity.Critical,
|
||||
Details = BuildCriticalDetails(req, threshold),
|
||||
Status = AlertStatus.Open,
|
||||
TriggeredAt = DateTimeOffset.UtcNow
|
||||
ObservationCode = req.ObservationCode,
|
||||
Value = req.Value,
|
||||
Unit = req.Unit,
|
||||
Source = req.Source,
|
||||
IdempotencyKey = req.IdempotencyKey,
|
||||
RecordedAt = req.RecordedAt,
|
||||
CreatedAt = DateTimeOffset.UtcNow
|
||||
};
|
||||
_db.ClinicalAlerts.Add(alert);
|
||||
_db.Observations.Add(observation);
|
||||
|
||||
// Step 6b — outbox event for the alert (relay picks this up in Phase 3)
|
||||
_db.OutboxEvents.Add(BuildOutboxEvent("alert.generated", new
|
||||
// Step 5 — load threshold from Redis; fall back to PostgreSQL on miss
|
||||
var threshold = await LoadThresholdAsync(req.ObservationCode);
|
||||
|
||||
if (threshold is null)
|
||||
throw new ValidationException(
|
||||
$"No alert threshold is configured for observation code '{req.ObservationCode}'. " +
|
||||
"Register a threshold before recording observations for this code.",
|
||||
"UNKNOWN_OBSERVATION_CODE");
|
||||
|
||||
ClinicalAlert? alert = null;
|
||||
|
||||
// Step 6 — critical threshold detection (synchronous)
|
||||
// WARNING detection is intentionally deferred to the Kafka consumer.
|
||||
// A critical potassium of 2.1 mEq/L is immediately life-threatening — the alert
|
||||
// must exist before this API call returns. A warning heart rate of 95 bpm warrants
|
||||
// attention but not an emergency page; the additional Kafka latency is clinically safe.
|
||||
if (IsCriticalBreach(req.Value, threshold))
|
||||
{
|
||||
alertId = alert.Id,
|
||||
alert = new ClinicalAlert
|
||||
{
|
||||
Id = Guid.NewGuid(),
|
||||
EncounterId = encounterId,
|
||||
PatientId = encounter.PatientId,
|
||||
ObservationId = observation.Id,
|
||||
AlertType = AlertTypeExtensions.CriticalFor(req.ObservationCode),
|
||||
Severity = AlertSeverity.Critical,
|
||||
Details = BuildCriticalDetails(req, threshold),
|
||||
Status = AlertStatus.Open,
|
||||
TriggeredAt = DateTimeOffset.UtcNow
|
||||
};
|
||||
_db.ClinicalAlerts.Add(alert);
|
||||
|
||||
// Step 6b — outbox event for the alert (relay picks this up in Phase 3)
|
||||
_db.OutboxEvents.Add(BuildOutboxEvent("alert.generated", new
|
||||
{
|
||||
alertId = alert.Id,
|
||||
encounterId,
|
||||
patientId = encounter.PatientId,
|
||||
department = encounter.Department.ToDbString(),
|
||||
alertType = alert.AlertType.ToDbString(),
|
||||
severity = alert.Severity.ToDbString(),
|
||||
details = alert.Details,
|
||||
attendingPhysician = encounter.AttendingPhysician,
|
||||
triggeredAt = alert.TriggeredAt,
|
||||
partitionKey = encounterId.ToString()
|
||||
}, encounterId.ToString()));
|
||||
}
|
||||
|
||||
// Step 7 — outbox event for the observation (always; Kafka consumer handles warnings)
|
||||
_db.OutboxEvents.Add(BuildOutboxEvent("observation.recorded", new
|
||||
{
|
||||
observationId = observation.Id,
|
||||
encounterId,
|
||||
patientId = encounter.PatientId,
|
||||
department = encounter.Department.ToDbString(),
|
||||
alertType = alert.AlertType.ToDbString(),
|
||||
severity = alert.Severity.ToDbString(),
|
||||
details = alert.Details,
|
||||
attendingPhysician = encounter.AttendingPhysician,
|
||||
triggeredAt = alert.TriggeredAt,
|
||||
mrn = encounter.Patient?.Mrn,
|
||||
observationCode = req.ObservationCode,
|
||||
value = req.Value,
|
||||
unit = req.Unit,
|
||||
source = req.Source.ToDbString(),
|
||||
recordedAt = req.RecordedAt,
|
||||
partitionKey = encounterId.ToString()
|
||||
}, encounterId.ToString()));
|
||||
|
||||
// Step 8 — COMMIT
|
||||
await _db.SaveChangesAsync();
|
||||
await tx.CommitAsync();
|
||||
|
||||
_metrics.ObservationsIngestedTotal
|
||||
.WithLabels(req.ObservationCode, req.Source.ToDbString())
|
||||
.Inc();
|
||||
|
||||
if (alert is not null)
|
||||
{
|
||||
_metrics.ClinicalAlertsTotal
|
||||
.WithLabels(alert.AlertType.ToDbString(), alert.Severity.ToDbString())
|
||||
.Inc();
|
||||
_logger.LogWarning(
|
||||
"Critical threshold breach. Code={Code} Value={Value} AlertId={AlertId}",
|
||||
req.ObservationCode, req.Value, alert.Id);
|
||||
}
|
||||
|
||||
_logger.LogInformation(
|
||||
"Observation {ObservationId} ingested for encounter {EncounterId}. AlertCreated={AlertCreated}",
|
||||
observation.Id, encounterId, alert is not null);
|
||||
|
||||
return IngestResult.Created(observation, alert);
|
||||
}
|
||||
|
||||
// Step 7 — outbox event for the observation (always; Kafka consumer handles warnings)
|
||||
_db.OutboxEvents.Add(BuildOutboxEvent("observation.recorded", new
|
||||
catch (DbUpdateException ex) when (IsUniqueViolation(ex))
|
||||
{
|
||||
observationId = observation.Id,
|
||||
encounterId,
|
||||
patientId = encounter.PatientId,
|
||||
mrn = encounter.Patient?.Mrn,
|
||||
observationCode = req.ObservationCode,
|
||||
value = req.Value,
|
||||
unit = req.Unit,
|
||||
source = req.Source.ToDbString(),
|
||||
recordedAt = req.RecordedAt,
|
||||
partitionKey = encounterId.ToString()
|
||||
}, encounterId.ToString()));
|
||||
|
||||
// Step 8 — COMMIT
|
||||
await _db.SaveChangesAsync();
|
||||
await tx.CommitAsync();
|
||||
|
||||
_logger.LogInformation(
|
||||
"Observation {ObservationId} ingested for encounter {EncounterId}. AlertCreated={AlertCreated}",
|
||||
observation.Id, encounterId, alert is not null);
|
||||
|
||||
return IngestResult.Created(observation, alert);
|
||||
}
|
||||
catch (DbUpdateException ex) when (IsUniqueViolation(ex))
|
||||
{
|
||||
// Race condition: two concurrent retries both passed the pre-check above.
|
||||
// The unique partial index caught it. Roll back and return the existing row.
|
||||
await tx.RollbackAsync();
|
||||
var existing = await _db.Observations
|
||||
.AsNoTracking()
|
||||
.FirstOrDefaultAsync(o => o.IdempotencyKey == req.IdempotencyKey);
|
||||
if (existing is not null)
|
||||
return IngestResult.Duplicate(existing);
|
||||
throw new ConflictException("Concurrent duplicate submission.", "CONCURRENT_DUPLICATE");
|
||||
}
|
||||
catch
|
||||
{
|
||||
await tx.RollbackAsync();
|
||||
throw;
|
||||
// Race condition: two concurrent retries both passed the pre-check above.
|
||||
// The unique partial index caught it. Roll back and return the existing row.
|
||||
await tx.RollbackAsync();
|
||||
var existing = await _db.Observations
|
||||
.AsNoTracking()
|
||||
.FirstOrDefaultAsync(o => o.IdempotencyKey == req.IdempotencyKey);
|
||||
if (existing is not null)
|
||||
return IngestResult.Duplicate(existing);
|
||||
throw new ConflictException("Concurrent duplicate submission.", "CONCURRENT_DUPLICATE");
|
||||
}
|
||||
catch
|
||||
{
|
||||
await tx.RollbackAsync();
|
||||
throw;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -18,8 +18,11 @@
|
||||
</PackageReference>
|
||||
<PackageReference Include="Minio" Version="6.0.3" />
|
||||
<PackageReference Include="Npgsql.EntityFrameworkCore.PostgreSQL" Version="8.0.4" />
|
||||
<PackageReference Include="prometheus-net.AspNetCore" Version="8.2.1" />
|
||||
<PackageReference Include="RabbitMQ.Client" Version="6.8.1" />
|
||||
<PackageReference Include="Serilog.AspNetCore" Version="10.0.0" />
|
||||
<PackageReference Include="Serilog.Enrichers.Environment" Version="2.3.0" />
|
||||
<PackageReference Include="Serilog.Enrichers.Thread" Version="3.1.0" />
|
||||
<PackageReference Include="Serilog.Sinks.Console" Version="6.1.1" />
|
||||
<PackageReference Include="Serilog.Sinks.Seq" Version="9.1.0" />
|
||||
<PackageReference Include="StackExchange.Redis" Version="3.0.0" />
|
||||
|
||||
@@ -26,7 +26,7 @@
|
||||
}
|
||||
}
|
||||
],
|
||||
"Enrich": [ "FromLogContext" ]
|
||||
"Enrich": [ "FromLogContext", "WithMachineName", "WithThreadId" ]
|
||||
},
|
||||
"Logging": {
|
||||
"LogLevel": {
|
||||
|
||||
+55
-1
@@ -9,11 +9,15 @@ services:
|
||||
- "5436:5432"
|
||||
volumes:
|
||||
- pg_data:/var/lib/postgresql/data
|
||||
networks:
|
||||
- vigilcare_net
|
||||
|
||||
redis:
|
||||
image: redis:7-alpine
|
||||
ports:
|
||||
- "6382:6379"
|
||||
networks:
|
||||
- vigilcare_net
|
||||
|
||||
seq:
|
||||
image: datalust/seq:latest
|
||||
@@ -24,6 +28,8 @@ services:
|
||||
- "5345:80"
|
||||
volumes:
|
||||
- seq_data:/data
|
||||
networks:
|
||||
- vigilcare_net
|
||||
|
||||
kafka:
|
||||
image: apache/kafka:3.7.0
|
||||
@@ -49,6 +55,8 @@ services:
|
||||
CLUSTER_ID: "MkU3OEVBNTcwNTJENDM2Qk"
|
||||
volumes:
|
||||
- kafka_data:/var/lib/kafka/data
|
||||
networks:
|
||||
- vigilcare_net
|
||||
|
||||
elasticsearch:
|
||||
image: docker.elastic.co/elasticsearch/elasticsearch:8.13.0
|
||||
@@ -68,6 +76,8 @@ services:
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 10
|
||||
networks:
|
||||
- vigilcare_net
|
||||
|
||||
rabbitmq:
|
||||
image: rabbitmq:3.13-management-alpine
|
||||
@@ -83,6 +93,8 @@ services:
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
networks:
|
||||
- vigilcare_net
|
||||
|
||||
minio:
|
||||
image: minio/minio:RELEASE.2024-07-04T14-25-45Z
|
||||
@@ -96,10 +108,52 @@ services:
|
||||
MINIO_ROOT_PASSWORD: minioadmin
|
||||
volumes:
|
||||
- minio_data:/data
|
||||
networks:
|
||||
- vigilcare_net
|
||||
|
||||
prometheus:
|
||||
image: prom/prometheus:v2.52.0
|
||||
container_name: vigilcare_prometheus
|
||||
command:
|
||||
- "--config.file=/etc/prometheus/prometheus.yml"
|
||||
- "--storage.tsdb.retention.time=7d"
|
||||
ports:
|
||||
- "9101:9090"
|
||||
volumes:
|
||||
- ./infra/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
|
||||
- prometheus_data:/prometheus
|
||||
extra_hosts:
|
||||
- "host.docker.internal:host-gateway"
|
||||
networks:
|
||||
- vigilcare_net
|
||||
|
||||
grafana:
|
||||
image: grafana/grafana:10.4.3
|
||||
container_name: vigilcare_grafana
|
||||
ports:
|
||||
- "3101:3000"
|
||||
environment:
|
||||
GF_SECURITY_ADMIN_USER: admin
|
||||
GF_SECURITY_ADMIN_PASSWORD: admin
|
||||
GF_USERS_ALLOW_SIGN_UP: "false"
|
||||
volumes:
|
||||
- ./infra/grafana/provisioning:/etc/grafana/provisioning:ro
|
||||
- ./infra/grafana/dashboards:/var/lib/grafana/dashboards:ro
|
||||
- grafana_data:/var/lib/grafana
|
||||
depends_on:
|
||||
- prometheus
|
||||
networks:
|
||||
- vigilcare_net
|
||||
|
||||
volumes:
|
||||
pg_data:
|
||||
seq_data:
|
||||
kafka_data:
|
||||
es_data:
|
||||
minio_data:
|
||||
minio_data:
|
||||
prometheus_data:
|
||||
grafana_data:
|
||||
|
||||
networks:
|
||||
vigilcare_net:
|
||||
driver: bridge
|
||||
|
||||
@@ -0,0 +1,315 @@
|
||||
# Docker & Docker Compose Guide (VigilCare)
|
||||
|
||||
This guide is a practical reference for running this project with Docker, plus troubleshooting for common issues seen in this repo.
|
||||
|
||||
It is written for junior developers, so each section explains not just what to do, but why.
|
||||
|
||||
---
|
||||
|
||||
## 1) Quick start
|
||||
|
||||
From repo root:
|
||||
|
||||
```bash
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
Check status:
|
||||
|
||||
```bash
|
||||
docker compose ps
|
||||
```
|
||||
|
||||
Stop everything (keep data):
|
||||
|
||||
```bash
|
||||
docker compose stop
|
||||
```
|
||||
|
||||
Start again:
|
||||
|
||||
```bash
|
||||
docker compose start
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2) Core concepts (simple mental model)
|
||||
|
||||
### Container
|
||||
- A running process with its own filesystem and network namespace.
|
||||
- Example: `vigilcare_prometheus` is one container.
|
||||
|
||||
### Service (in `docker-compose.yml`)
|
||||
- A recipe for how to run a container.
|
||||
- Example: the `prometheus:` section defines image, ports, volumes, networks.
|
||||
|
||||
### Image
|
||||
- Template used to create a container.
|
||||
- Example: `prom/prometheus:v2.52.0`.
|
||||
|
||||
### Volume
|
||||
- Persistent storage managed by Docker.
|
||||
- Survives container restarts/recreates.
|
||||
- Example: `prometheus_data`, `grafana_data`, `pg_data`.
|
||||
|
||||
### Network
|
||||
- Virtual network connecting containers.
|
||||
- Containers can reach each other by service name (DNS).
|
||||
- Example: Grafana reaches Prometheus at `http://prometheus:9090` inside Docker.
|
||||
|
||||
---
|
||||
|
||||
## 3) Host ports vs container ports
|
||||
|
||||
In Compose, this format is used:
|
||||
|
||||
```yaml
|
||||
ports:
|
||||
- "HOST:CONTAINER"
|
||||
```
|
||||
|
||||
Example from this project:
|
||||
- Prometheus: `"9101:9090"`
|
||||
- Open in browser with `http://localhost:9101`
|
||||
- Inside Docker, service still listens on `9090`
|
||||
- Grafana: `"3101:3000"`
|
||||
- Open in browser with `http://localhost:3101`
|
||||
|
||||
If a UI is not loading, first verify host port mappings in `docker-compose.yml`.
|
||||
|
||||
---
|
||||
|
||||
## 4) Project networking (`vigilcare_net`)
|
||||
|
||||
This project uses a user-defined bridge network:
|
||||
|
||||
```yaml
|
||||
networks:
|
||||
vigilcare_net:
|
||||
driver: bridge
|
||||
```
|
||||
|
||||
All services should join it:
|
||||
|
||||
```yaml
|
||||
networks:
|
||||
- vigilcare_net
|
||||
```
|
||||
|
||||
Why this matters:
|
||||
- Service-to-service DNS works (`prometheus`, `grafana`, `postgres`, etc.).
|
||||
- Keeps local environment predictable.
|
||||
|
||||
### Important Linux note: `host.docker.internal`
|
||||
|
||||
Prometheus scrapes the API via host address in this project:
|
||||
- `http://host.docker.internal:5270/metrics`
|
||||
|
||||
On Linux, `host.docker.internal` may not resolve by default.
|
||||
Fix by adding this to the `prometheus` service:
|
||||
|
||||
```yaml
|
||||
extra_hosts:
|
||||
- "host.docker.internal:host-gateway"
|
||||
```
|
||||
|
||||
Then recreate Prometheus:
|
||||
|
||||
```bash
|
||||
docker compose up -d --force-recreate prometheus
|
||||
```
|
||||
|
||||
Symptom when missing:
|
||||
- Prometheus target `vigilcare_api` is `down`
|
||||
- Error: `lookup host.docker.internal ... no such host`
|
||||
|
||||
### Important: API must listen on all interfaces (not only `localhost`)
|
||||
|
||||
After `extra_hosts` is fixed, Prometheus may still show:
|
||||
|
||||
```
|
||||
dial tcp 172.17.0.1:5270: connect: connection refused
|
||||
```
|
||||
|
||||
**Why:** `dotnet run` with `http://localhost:5270` binds only to `127.0.0.1`.
|
||||
`localhost` inside a container means the container itself, not your host machine.
|
||||
Prometheus inside Docker reaches the host via the gateway IP (`172.17.0.1` via `host.docker.internal`), not host loopback.
|
||||
|
||||
**Check binding:**
|
||||
|
||||
```bash
|
||||
ss -tlnp | rg ':5270'
|
||||
```
|
||||
|
||||
If you see `127.0.0.1:5270`, Prometheus cannot scrape from Docker.
|
||||
|
||||
**Fix (local dev):** bind on all interfaces in `VigilCareClinicalAPI/Properties/launchSettings.json`:
|
||||
|
||||
```json
|
||||
"applicationUrl": "http://0.0.0.0:5270"
|
||||
```
|
||||
|
||||
Or start the API with:
|
||||
|
||||
```bash
|
||||
ASPNETCORE_URLS=http://0.0.0.0:5270 dotnet run --project VigilCareClinicalAPI
|
||||
```
|
||||
|
||||
Then restart the API and confirm:
|
||||
|
||||
```bash
|
||||
ss -tlnp | rg ':5270' # should show 0.0.0.0:5270
|
||||
curl -sS http://localhost:5270/metrics | head
|
||||
```
|
||||
|
||||
**Security note:** `0.0.0.0` is fine for local development. In production, bind explicitly and use proper network controls.
|
||||
|
||||
---
|
||||
|
||||
## 5) Volumes and persistence
|
||||
|
||||
This repo uses named volumes for persistent data:
|
||||
- `pg_data`
|
||||
- `seq_data`
|
||||
- `kafka_data`
|
||||
- `es_data`
|
||||
- `minio_data`
|
||||
- `prometheus_data`
|
||||
- `grafana_data`
|
||||
|
||||
### Why your data still exists after restart
|
||||
- `docker compose up -d --force-recreate` recreates containers, but volumes remain.
|
||||
- This is expected and usually desired.
|
||||
|
||||
### Full reset (destructive)
|
||||
If you need a totally clean environment:
|
||||
|
||||
```bash
|
||||
docker compose down -v
|
||||
```
|
||||
|
||||
Warning:
|
||||
- `-v` removes named volumes (database/log/index data lost).
|
||||
|
||||
---
|
||||
|
||||
## 6) Common commands and when to use them
|
||||
|
||||
### Apply config change to one service
|
||||
Use when you changed only one section (e.g., Prometheus `extra_hosts`):
|
||||
|
||||
```bash
|
||||
docker compose up -d --force-recreate prometheus
|
||||
```
|
||||
|
||||
### Restart service without recreate
|
||||
Use when config did not change and you just want a restart:
|
||||
|
||||
```bash
|
||||
docker compose restart prometheus
|
||||
```
|
||||
|
||||
### Rebuild image service
|
||||
Use when Dockerfile/app code in image changed:
|
||||
|
||||
```bash
|
||||
docker compose up -d --build <service>
|
||||
```
|
||||
|
||||
### View service logs
|
||||
|
||||
```bash
|
||||
docker compose logs -f prometheus
|
||||
docker compose logs -f grafana
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7) Troubleshooting playbook
|
||||
|
||||
### A) “Service is up but endpoint won’t open”
|
||||
1. Check container state:
|
||||
```bash
|
||||
docker compose ps
|
||||
```
|
||||
2. Verify port mapping in `docker-compose.yml`.
|
||||
3. Check logs:
|
||||
```bash
|
||||
docker compose logs --tail=100 <service>
|
||||
```
|
||||
|
||||
### B) “Prometheus healthy, but target is DOWN”
|
||||
1. Open Prometheus targets page:
|
||||
- `http://localhost:9101/targets`
|
||||
2. Read the exact `lastError`.
|
||||
3. If error mentions `host.docker.internal` on Linux:
|
||||
- add `extra_hosts` fix (section 4)
|
||||
- recreate Prometheus.
|
||||
4. If error is `connection refused` to `172.17.0.1:5270`:
|
||||
- API is likely bound to `127.0.0.1` only
|
||||
- use `http://0.0.0.0:5270` and restart API (section 4).
|
||||
|
||||
### C) “Docker compose command cannot connect to daemon”
|
||||
Example:
|
||||
- `failed to connect to the docker API at unix:///var/run/docker.sock`
|
||||
|
||||
Fix:
|
||||
- Start Docker Desktop / Docker daemon.
|
||||
- Re-run `docker compose ps`.
|
||||
|
||||
### D) “Permission denied writing files under bind-mounted folder”
|
||||
This can happen when directories/files were created as `root`.
|
||||
|
||||
Symptoms:
|
||||
- Cannot create/edit files in folders like Grafana dashboard path.
|
||||
|
||||
Fix options:
|
||||
1. Correct ownership on host:
|
||||
```bash
|
||||
sudo chown -R $USER:$USER <folder>
|
||||
```
|
||||
2. Recreate problematic directory as your user.
|
||||
|
||||
### E) “Script fails preflight even though services seem running”
|
||||
Check these directly:
|
||||
|
||||
```bash
|
||||
curl -sS -o /dev/null -w "%{http_code}\n" http://localhost:5270/api/v1/alert-thresholds
|
||||
curl -sS -o /dev/null -w "%{http_code}\n" http://localhost:9101/-/healthy
|
||||
curl -sS -o /dev/null -w "%{http_code}\n" http://localhost:5345
|
||||
```
|
||||
|
||||
Expected: `200`, `200`, `200`.
|
||||
|
||||
---
|
||||
|
||||
## 8) Current project-specific paths and notes
|
||||
|
||||
- Compose file: `docker-compose.yml`
|
||||
- Prometheus config: `infra/prometheus/prometheus.yml`
|
||||
- Grafana provisioning:
|
||||
- `infra/grafana/provisioning/datasources/prometheus.yml`
|
||||
- `infra/grafana/provisioning/datasources/dashboards/config.yml`
|
||||
- Grafana dashboards expected path:
|
||||
- `infra/grafana/dashboards/`
|
||||
|
||||
Note: if you accidentally create a typo folder like `dashbpards`, Grafana provisioning will not load dashboards from it.
|
||||
|
||||
---
|
||||
|
||||
## 9) Safe workflow for config changes (recommended)
|
||||
|
||||
1. Edit `docker-compose.yml`.
|
||||
2. Recreate only changed services:
|
||||
```bash
|
||||
docker compose up -d --force-recreate <service>
|
||||
```
|
||||
3. Verify logs and health endpoints.
|
||||
4. Run project verification scripts (example):
|
||||
```bash
|
||||
./scripts/run-phase8-verification.sh
|
||||
```
|
||||
|
||||
This avoids unnecessary full resets and speeds up local development.
|
||||
|
||||
@@ -350,7 +350,7 @@ sirs:{encounterId}:WBC_K_UL → "1" (TTL: 30 minutes)
|
||||
|
||||
### 8. RabbitMQ Notification Workers and Escalation
|
||||
|
||||
**Description:** The notification worker reads `alert.generated` events from Kafka and dispatches paging jobs to RabbitMQ. The RabbitMQ consumer sends the page and waits for acknowledgment. If no acknowledgment arrives within five minutes, the dead-letter queue escalates to the on-call backup.
|
||||
**Description:** The notification worker reads `alert.generated` events from Kafka and dispatches paging jobs to RabbitMQ. The RabbitMQ consumer sends the page and waits for acknowledgment. If no acknowledgment arrives within five minutes, the dead-letter queue escalates to the on-call backup. If the API host is stopping while a page is in flight, the cancellation path requeues the message rather than escalating it.
|
||||
|
||||
**Exchange topology:**
|
||||
```
|
||||
@@ -373,6 +373,10 @@ clinical.notifications.exchange (direct)
|
||||
c. After TTL: message routes back to alerts.escalation.queue
|
||||
d. Escalation worker pages the on-call backup
|
||||
e. clinical_alert.status → 'escalated' in PostgreSQL
|
||||
5. If host shutdown occurs during paging wait:
|
||||
a. Cancellation is treated as graceful stop, not failure
|
||||
b. NACK with requeue=true → message returns to alerts.paging.queue
|
||||
c. No DLQ route, so no false escalation during restart/deploy
|
||||
```
|
||||
|
||||
**Discharge summary job:** When an encounter status changes to `discharged`, the outbox relay publishes to Kafka `encounter.status.changed`. The notification Kafka consumer reads this and publishes to `notifications.discharge.queue`. The worker generates a PDF summary (log the content; no real PDF library required), stores it in MinIO under `/discharge-summaries/{encounterId}/summary.pdf`, and marks the job complete.
|
||||
|
||||
@@ -0,0 +1,197 @@
|
||||
{
|
||||
"uid": "vigilcare-v1",
|
||||
"title": "VigilCare Clinical",
|
||||
"tags": ["vigilcare", "clinical", "prometheus"],
|
||||
"timezone": "browser",
|
||||
"schemaVersion": 38,
|
||||
"version": 1,
|
||||
"refresh": "30s",
|
||||
"time": {
|
||||
"from": "now-1h",
|
||||
"to": "now"
|
||||
},
|
||||
"panels": [
|
||||
{
|
||||
"id": 1,
|
||||
"type": "stat",
|
||||
"title": "Unacknowledged Critical Alerts",
|
||||
"gridPos": { "x": 0, "y": 0, "w": 6, "h": 4 },
|
||||
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||
"targets": [
|
||||
{
|
||||
"expr": "alerts_unacknowledged_gauge",
|
||||
"legendFormat": "open > 5 min",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"options": {
|
||||
"colorMode": "background",
|
||||
"graphMode": "none",
|
||||
"justifyMode": "auto",
|
||||
"reduceOptions": {
|
||||
"values": false,
|
||||
"calcs": ["lastNotNull"],
|
||||
"fields": ""
|
||||
}
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{ "color": "green", "value": null },
|
||||
{ "color": "red", "value": 1 }
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 2,
|
||||
"type": "timeseries",
|
||||
"title": "Observation Ingest Rate",
|
||||
"gridPos": { "x": 6, "y": 0, "w": 12, "h": 8 },
|
||||
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||
"targets": [
|
||||
{
|
||||
"expr": "rate(observations_ingested_total[1m])",
|
||||
"legendFormat": "{{observation_code}} / {{source}}",
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 3,
|
||||
"type": "bargauge",
|
||||
"title": "Clinical Alerts by Type and Severity (5m)",
|
||||
"gridPos": { "x": 18, "y": 0, "w": 6, "h": 8 },
|
||||
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||
"targets": [
|
||||
{
|
||||
"expr": "increase(clinical_alerts_total[5m])",
|
||||
"legendFormat": "{{alert_type}} / {{severity}}",
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 4,
|
||||
"type": "stat",
|
||||
"title": "Outbox Pending Events",
|
||||
"gridPos": { "x": 0, "y": 8, "w": 6, "h": 4 },
|
||||
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||
"targets": [
|
||||
{
|
||||
"expr": "outbox_pending_events",
|
||||
"legendFormat": "pending",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"options": {
|
||||
"colorMode": "background",
|
||||
"graphMode": "none",
|
||||
"justifyMode": "auto",
|
||||
"reduceOptions": {
|
||||
"values": false,
|
||||
"calcs": ["lastNotNull"],
|
||||
"fields": ""
|
||||
}
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{ "color": "green", "value": null },
|
||||
{ "color": "yellow", "value": 100 },
|
||||
{ "color": "red", "value": 500 }
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 5,
|
||||
"type": "timeseries",
|
||||
"title": "Kafka Consumer Lag by Group",
|
||||
"gridPos": { "x": 6, "y": 8, "w": 12, "h": 8 },
|
||||
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||
"targets": [
|
||||
{
|
||||
"expr": "kafka_consumer_lag",
|
||||
"legendFormat": "{{consumer_group}}",
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 6,
|
||||
"type": "timeseries",
|
||||
"title": "Ingest Latency p50 / p99",
|
||||
"gridPos": { "x": 18, "y": 8, "w": 6, "h": 8 },
|
||||
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||
"targets": [
|
||||
{
|
||||
"expr": "histogram_quantile(0.50, rate(observation_ingest_duration_seconds_bucket[5m]))",
|
||||
"legendFormat": "p50",
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"expr": "histogram_quantile(0.99, rate(observation_ingest_duration_seconds_bucket[5m]))",
|
||||
"legendFormat": "p99",
|
||||
"refId": "B"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 7,
|
||||
"type": "stat",
|
||||
"title": "Escalations Total",
|
||||
"gridPos": { "x": 0, "y": 12, "w": 12, "h": 4 },
|
||||
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||
"targets": [
|
||||
{
|
||||
"expr": "escalations_total",
|
||||
"legendFormat": "escalations",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"options": {
|
||||
"colorMode": "value",
|
||||
"graphMode": "none",
|
||||
"justifyMode": "auto",
|
||||
"reduceOptions": {
|
||||
"values": false,
|
||||
"calcs": ["lastNotNull"],
|
||||
"fields": ""
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 8,
|
||||
"type": "stat",
|
||||
"title": "SIRS Detections Total",
|
||||
"gridPos": { "x": 12, "y": 12, "w": 12, "h": 4 },
|
||||
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||
"targets": [
|
||||
{
|
||||
"expr": "sirs_detections_total",
|
||||
"legendFormat": "sirs",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"options": {
|
||||
"colorMode": "value",
|
||||
"graphMode": "none",
|
||||
"justifyMode": "auto",
|
||||
"reduceOptions": {
|
||||
"values": false,
|
||||
"calcs": ["lastNotNull"],
|
||||
"fields": ""
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,6 @@
|
||||
apiVersion: 1
|
||||
providers:
|
||||
- name: VigilCare
|
||||
type: file
|
||||
options:
|
||||
path: /var/lib/grafana/dashboards
|
||||
@@ -0,0 +1,7 @@
|
||||
apiVersion: 1
|
||||
datasources:
|
||||
- name: Prometheus
|
||||
type: prometheus
|
||||
url: http://prometheus:9090
|
||||
isDefault: true
|
||||
access: proxy
|
||||
@@ -0,0 +1,11 @@
|
||||
global:
|
||||
scrape_interval: 15s
|
||||
evaluation_interval: 15s
|
||||
|
||||
scrape_configs:
|
||||
- job_name: vigilcare_api
|
||||
# host.docker.internal resolves to the host machine from inside Docker.
|
||||
# Replace with the container name and internal port if the API runs in Docker too.
|
||||
static_configs:
|
||||
- targets: ["host.docker.internal:5270"]
|
||||
metrics_path: /metrics
|
||||
Executable
+192
@@ -0,0 +1,192 @@
|
||||
#!/usr/bin/env bash
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
COMPOSE_FILE="${COMPOSE_FILE:-${SCRIPT_DIR}/../docker-compose.yml}"
|
||||
|
||||
BASE_URL="${BASE_URL:-http://localhost:5270}"
|
||||
PROM_URL="${PROM_URL:-http://localhost:9101}"
|
||||
SEQ_URL="${SEQ_URL:-http://localhost:5345}"
|
||||
|
||||
PGHOST="${PGHOST:-localhost}"
|
||||
PGPORT="${PGPORT:-5436}"
|
||||
PGDATABASE="${PGDATABASE:-vigilcare}"
|
||||
PGUSER="${PGUSER:-postgres}"
|
||||
PGPASSWORD="${PGPASSWORD:-password}"
|
||||
|
||||
COLLECTOR_WAIT_SECS="${COLLECTOR_WAIT_SECS:-35}"
|
||||
|
||||
TMP_FILES=()
|
||||
cleanup() {
|
||||
local f
|
||||
for f in "${TMP_FILES[@]}"; do
|
||||
rm -f "$f" "$f.status" 2>/dev/null || true
|
||||
done
|
||||
}
|
||||
trap cleanup EXIT
|
||||
|
||||
need() {
|
||||
command -v "$1" >/dev/null 2>&1 || {
|
||||
echo "Missing dependency: $1"
|
||||
exit 1
|
||||
}
|
||||
}
|
||||
|
||||
need curl
|
||||
need jq
|
||||
|
||||
request() {
|
||||
local method="$1"
|
||||
local url="$2"
|
||||
local body="${3:-}"
|
||||
local tmp
|
||||
tmp="$(mktemp)"
|
||||
TMP_FILES+=("$tmp")
|
||||
local status
|
||||
|
||||
if [[ -n "$body" ]]; then
|
||||
status="$(curl -sS -o "$tmp" -w "%{http_code}" -X "$method" "$url" \
|
||||
-H "Content-Type: application/json" -d "$body")"
|
||||
else
|
||||
status="$(curl -sS -o "$tmp" -w "%{http_code}" -X "$method" "$url")"
|
||||
fi
|
||||
|
||||
echo "$status" > "$tmp.status"
|
||||
echo "$tmp"
|
||||
}
|
||||
|
||||
assert_status() {
|
||||
local expected="$1"
|
||||
local body_file="$2"
|
||||
local status
|
||||
status="$(<"$body_file.status")"
|
||||
if [[ "$status" != "$expected" ]]; then
|
||||
echo "Expected HTTP $expected, got $status"
|
||||
cat "$body_file"
|
||||
echo
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
psql_cmd() {
|
||||
local sql="$1"
|
||||
if command -v psql >/dev/null 2>&1; then
|
||||
PGPASSWORD="$PGPASSWORD" psql -h "$PGHOST" -p "$PGPORT" -U "$PGUSER" -d "$PGDATABASE" -tAc "$sql"
|
||||
elif command -v docker >/dev/null 2>&1 && [[ -f "${COMPOSE_FILE}" ]]; then
|
||||
docker compose -f "${COMPOSE_FILE}" exec -T postgres \
|
||||
psql -U "${PGUSER}" -d "${PGDATABASE}" -tAc "${sql}"
|
||||
else
|
||||
echo "Missing dependency: psql (or docker compose with postgres service)"
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
metric_value() {
|
||||
local name="$1"
|
||||
local body="$2"
|
||||
awk -v m="$name" '
|
||||
$0 ~ "^"m"([ \t]|\\{|$)" {
|
||||
n=split($0, a, /[ \t]+/);
|
||||
if (n >= 2) { print a[n]; exit 0; }
|
||||
}
|
||||
' <<< "$body"
|
||||
}
|
||||
|
||||
echo "Phase 8 verification against ${BASE_URL}"
|
||||
|
||||
echo "[1/9] Preflight API and Prometheus"
|
||||
api_status="$(curl -sS -o /dev/null -w "%{http_code}" "${BASE_URL}/api/v1/alert-thresholds" || true)"
|
||||
prom_status="$(curl -sS -o /dev/null -w "%{http_code}" "${PROM_URL}/-/healthy" || true)"
|
||||
[[ "$api_status" == "200" ]] || { echo "API not ready (${api_status})"; exit 1; }
|
||||
[[ "$prom_status" == "200" ]] || { echo "Prometheus not ready (${prom_status})"; exit 1; }
|
||||
|
||||
echo "[2/9] Ensure Prometheus target vigilcare_api is UP"
|
||||
target_json="$(curl -sS "${PROM_URL}/api/v1/query?query=up%7Bjob%3D%22vigilcare_api%22%7D")"
|
||||
up_val="$(jq -r '.data.result[0].value[1] // "0"' <<< "$target_json")"
|
||||
[[ "$up_val" == "1" ]] || { echo "vigilcare_api target not UP"; exit 1; }
|
||||
|
||||
echo "[3/9] Validate /metrics contains all eight metric families"
|
||||
metrics_body="$(curl -sS "${BASE_URL}/metrics")"
|
||||
required=(
|
||||
observations_ingested_total
|
||||
observation_ingest_duration_seconds
|
||||
clinical_alerts_total
|
||||
alerts_unacknowledged_gauge
|
||||
kafka_consumer_lag
|
||||
outbox_pending_events
|
||||
sirs_detections_total
|
||||
escalations_total
|
||||
)
|
||||
for m in "${required[@]}"; do
|
||||
grep -q "$m" <<< "$metrics_body" || { echo "Missing metric: $m"; exit 1; }
|
||||
done
|
||||
|
||||
echo "[4/9] Verify correlation header on API response"
|
||||
corr="$(curl -sSI "${BASE_URL}/api/v1/alert-thresholds" | awk -F': ' 'tolower($1)=="x-correlation-id"{print $2}' | tr -d '\r')"
|
||||
[[ -n "$corr" ]] || { echo "Missing X-Correlation-Id header"; exit 1; }
|
||||
|
||||
echo "[5/9] Create patient and active encounter"
|
||||
patient_resp="$(request POST "${BASE_URL}/api/v1/patients" '{"firstName":"Phase8","lastName":"Verify","dateOfBirth":"1988-01-10","gender":"F"}')"
|
||||
assert_status "201" "$patient_resp"
|
||||
patient_id="$(jq -r '.data.id' "$patient_resp")"
|
||||
|
||||
enc_resp="$(request POST "${BASE_URL}/api/v1/patients/${patient_id}/encounters" '{"encounterType":"Inpatient","department":"ICU","attendingPhysician":"Dr. Verify"}')"
|
||||
assert_status "201" "$enc_resp"
|
||||
encounter_id="$(jq -r '.data.id' "$enc_resp")"
|
||||
|
||||
transition_resp="$(request PATCH "${BASE_URL}/api/v1/encounters/${encounter_id}/status" '{"status":"Active"}')"
|
||||
transition_status="$(<"$transition_resp.status")"
|
||||
if [[ "$transition_status" != "200" && "$transition_status" != "409" ]]; then
|
||||
echo "Unexpected status transitioning encounter to active: ${transition_status}"
|
||||
cat "$transition_resp"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "[6/9] Create critical alert, backdate 6 minutes"
|
||||
baseline_metrics="$(curl -sS "${BASE_URL}/metrics")"
|
||||
baseline_gauge="$(metric_value "alerts_unacknowledged_gauge" "$baseline_metrics")"
|
||||
baseline_gauge_int="${baseline_gauge%.*}"
|
||||
baseline_gauge_int="${baseline_gauge_int:-0}"
|
||||
|
||||
recorded_at="$(date -u +"%Y-%m-%dT%H:%M:%SZ")"
|
||||
obs_payload="$(jq -nc --arg ts "$recorded_at" \
|
||||
'{observations:[{observationCode:"POTASSIUM_MEQ_L",value:2.1,unit:"mEq/L",source:"LAB",recordedAt:$ts}]}' )"
|
||||
obs_resp="$(request POST "${BASE_URL}/api/v1/encounters/${encounter_id}/observations" "$obs_payload")"
|
||||
assert_status "201" "$obs_resp"
|
||||
alert_id="$(jq -r '.data.alertId // empty' "$obs_resp")"
|
||||
[[ -n "$alert_id" ]] || { echo "Critical observation did not return alertId"; cat "$obs_resp"; exit 1; }
|
||||
|
||||
psql_cmd "UPDATE clinical_alerts SET triggered_at = NOW() - INTERVAL '6 minutes' WHERE id = '${alert_id}';" >/dev/null
|
||||
|
||||
echo "[7/9] Wait collector and verify alerts_unacknowledged_gauge increased"
|
||||
sleep "$COLLECTOR_WAIT_SECS"
|
||||
metrics_body="$(curl -sS "${BASE_URL}/metrics")"
|
||||
gauge="$(metric_value "alerts_unacknowledged_gauge" "$metrics_body")"
|
||||
gauge_int="${gauge%.*}"
|
||||
gauge_int="${gauge_int:-0}"
|
||||
expected_min=$((baseline_gauge_int + 1))
|
||||
[[ -n "$gauge" && "$gauge_int" -ge "$expected_min" ]] || {
|
||||
echo "alerts_unacknowledged_gauge did not increase (baseline=${baseline_gauge_int}, value=${gauge:-missing})"
|
||||
exit 1
|
||||
}
|
||||
|
||||
echo "[8/9] Acknowledge alert and verify gauge returned to baseline"
|
||||
ack_resp="$(request POST "${BASE_URL}/api/v1/alerts/${alert_id}/acknowledge" '{"clinicianId":"DR-VERIFY","note":"phase8 verification"}')"
|
||||
assert_status "200" "$ack_resp"
|
||||
sleep "$COLLECTOR_WAIT_SECS"
|
||||
metrics_body="$(curl -sS "${BASE_URL}/metrics")"
|
||||
gauge="$(metric_value "alerts_unacknowledged_gauge" "$metrics_body")"
|
||||
gauge_int="${gauge%.*}"
|
||||
gauge_int="${gauge_int:-0}"
|
||||
[[ "$gauge_int" -le "$baseline_gauge_int" ]] || {
|
||||
echo "alerts_unacknowledged_gauge did not return to baseline (baseline=${baseline_gauge_int}, value=${gauge:-missing})"
|
||||
exit 1
|
||||
}
|
||||
|
||||
echo "[9/9] Manual Seq check"
|
||||
echo "Open ${SEQ_URL} and filter: EncounterId IS NOT NULL"
|
||||
echo "Confirm logs include EncounterId, PatientId, CorrelationId."
|
||||
|
||||
echo
|
||||
echo "Phase 8 verification checks passed."
|
||||
Reference in New Issue
Block a user