fix: No token refresh or revocation mechanism~

This commit is contained in:
voltsrage
2026-06-25 14:14:20 +08:00
parent a8964381a2
commit fdcc646fae
26 changed files with 3453 additions and 55 deletions
+3 -20
View File
@@ -565,26 +565,9 @@ Any compromised container on the Docker network can read/write/delete clinical d
---
## P3 — No token refresh or revocation mechanism
## ~~P3 — No token refresh or revocation mechanism~~ DONE
### Problem
JWT tokens are issued with a configurable expiration but there is no refresh token flow and no token revocation/blacklist. A compromised token remains valid until natural expiration. There is no `POST /auth/refresh` or `POST /auth/revoke` endpoint.
### Why fix
Clinical sessions may last entire shifts (8-12 hours). Short token lifetimes require frequent re-authentication, disrupting clinical workflows. Long lifetimes without revocation mean a stolen token grants extended access. Compromised accounts cannot be locked out until the token expires.
### How to fix
1. Add refresh token support: issue a long-lived opaque refresh token stored in the database alongside the access token.
2. Add `POST /api/v1/auth/refresh` — validate refresh token, issue new access token.
3. Add `POST /api/v1/auth/revoke` — invalidate refresh token and optionally blacklist the access token (via Redis TTL set matching remaining token lifetime).
4. Add `LastLoginAt` update on token refresh (already exists on `ClinicalUser`).
**Files:** `AuthController.cs`, `AuthService.cs`, `ClinicalUser.cs` (add `RefreshToken`, `RefreshTokenExpiresAt`), migration.
**Dependency:** None.
Implemented: `RefreshToken` entity with DB-backed storage, `POST /api/v1/auth/refresh` (rotate refresh token + issue new access token), `POST /api/v1/auth/logout` (revoke refresh token server-side). Access token reduced to 15 min, refresh token 7 days. Frontend auto-refreshes before expiry, retries on 401, and redirects to login on refresh failure. Logout button in header, sidebar, and mobile nav. Audit logged as `USER_LOGOUT` and `TOKEN_REFRESHED`.
---
@@ -743,7 +726,7 @@ Different hospitals and clinical settings have different protocols. CMS Sepsis S
| 19 | JWT key not validated on startup | P3 | D | Open |
| 20 | No authorization failure audit | P3 | D | Open |
| 21 | Elasticsearch security disabled | P3 | D | Open |
| 22 | No token refresh/revocation | P3 | D | Open |
| 22 | ~~No token refresh/revocation~~ | P3 | D | **Done** |
| 23 | No request timing metrics | P5 | E | Open |
| 24 | Background service error metrics | P5 | E | Open |
| 25 | Thin concurrent/resilience tests | P5 | E | Open |
+346
View File
@@ -0,0 +1,346 @@
# Guide 22: Clinical Scoring Engines (NEWS2, GCS, SOFA, qSOFA)
## What Are Clinical Scoring Systems?
In medicine, a single vital sign (like heart rate = 110) doesn't tell you much on its own — it could be a patient exercising or a patient in septic shock. **Clinical scoring systems** combine multiple vital signs and lab values into a single number that predicts how sick a patient is and whether they need urgent intervention.
Think of it like a weather severity index: temperature alone doesn't tell you if a storm is dangerous, but combining temperature, wind speed, pressure, and humidity gives you a meaningful risk score.
This project implements four clinical scoring systems:
| Score | Full Name | What It Measures | Parameters | Alert Threshold |
|-------|-----------|-----------------|------------|-----------------|
| **NEWS2** | National Early Warning Score 2 | General deterioration risk | 7 vital signs | Score >= 7 → Emergency |
| **GCS** | Glasgow Coma Scale | Level of consciousness | 3 components (Eye, Verbal, Motor) | Total <= 8 → Severe (Critical) |
| **qSOFA** | Quick SOFA | Bedside sepsis screen | 3 criteria | >= 2 criteria met → Screen positive |
| **SOFA** | Sequential Organ Failure Assessment | Organ dysfunction severity | 6 organ systems | Delta >= 2 from baseline → Sepsis |
---
## How Scoring Engines Work (Architecture)
All four scoring engines follow the same pattern: they're Kafka consumers that react to observation events.
```
Observation recorded (HTTP POST)
PostgreSQL + Outbox
▼ (OutboxRelay)
Kafka topic: observation.recorded
├──► SepsisEngineService → QsofaDetector → qSOFA evaluation
├──► News2ScoringService → News2Detector → NEWS2 score
├──► GcsScoringService → GcsDetector → GCS score
│ │
│ Kafka: gcs.scored ◄──┘
│ │
├──► SofaScoringService → SofaDetector → SOFA score
├──► TrendAnalyzerService → TrendDetector → Rate-of-change
└──► WarningAlertService → WarningEvaluator → Warning alerts
```
Each engine is a separate Kafka consumer group, so they process the same observation event independently and in parallel. A single heart rate reading can simultaneously trigger NEWS2 recalculation, qSOFA re-evaluation, trend analysis, and warning threshold checking.
### The Two-Class Pattern: Calculator + Detector
Each scoring system is split into two classes:
- **Calculator** (static, pure logic): Contains the scoring rules — "heart rate 45 scores 3 points in NEWS2." No database, no Redis, no side effects. Easy to unit test.
- **Detector** (stateful, orchestration): Manages Redis state, calls the calculator, persists scores to PostgreSQL, creates alerts, publishes events. Contains all the infrastructure plumbing.
---
## NEWS2 — National Early Warning Score 2
NEWS2 combines 7 vital sign parameters into a single risk score (020+). Higher scores indicate greater deterioration risk.
### The 7 Parameters and Their Scoring
```csharp
public static class News2Calculator
{
public static readonly IReadOnlyList<string> ParameterCodes = new[]
{
"RESP_RATE", "SPO2", "SYSTOLIC_BP", "HEART_RATE",
"AVPU", "TEMP_C", "SUPPLEMENTAL_O2"
};
public static int ScoreRespRate(decimal value) => value switch
{
<= 8 => 3, // Dangerously low
<= 11 => 1,
<= 20 => 0, // Normal range
<= 24 => 2,
_ => 3 // Dangerously high
};
public static int ScoreHeartRate(decimal value) => value switch
{
<= 40 => 3,
<= 50 => 1,
<= 90 => 0, // Normal range
<= 110 => 1,
<= 130 => 2,
_ => 3
};
// ... similar for SpO2, SystolicBp, Temperature, Consciousness, SupplementalO2
}
```
Each parameter scores 03 points. The total score determines the risk level:
| Total Score | Risk Level | Alert |
|-------------|-----------|-------|
| 04 | LOW | No alert |
| 56 or any single param = 3 | MEDIUM / LOW_MEDIUM | `NEWS2_WARNING` (Warning) |
| >= 7 | HIGH | `NEWS2_EMERGENCY` (Critical) |
### How NEWS2 Aggregates Over Time
The 7 parameters rarely arrive simultaneously. A nurse records respiratory rate at 14:01, heart rate at 14:03, blood pressure at 14:05. The `News2Detector` uses Redis to accumulate parameters until all 7 are present:
```
14:01 RESP_RATE=18 → Redis: news2:{enc}:RESP_RATE = {value:18, score:0} (1/7 present)
14:03 HEART_RATE=95 → Redis: news2:{enc}:HEART_RATE = {value:95, score:1} (2/7 present)
14:05 SYSTOLIC_BP=115 → Redis: news2:{enc}:SYSTOLIC_BP = ... (3/7 present)
... (more parameters arrive)
14:12 TEMP_C=37.5 → Redis: all 7 present → compute score = 5 (MEDIUM)
→ persist News2Score to PostgreSQL
→ create NEWS2_WARNING alert
```
Each Redis key has a **4-hour TTL**. If no new respiratory rate arrives within 4 hours, that parameter expires and the next NEWS2 calculation waits for a fresh reading.
### Consciousness Resolution: GCS-First, AVPU-Fallback
The consciousness parameter prefers GCS (Glasgow Coma Scale) over AVPU (a simpler alert/voice/pain/unresponsive scale). If GCS components are cached in Redis, they're used. Otherwise, the AVPU value is used:
```csharp
private async Task<int?> ResolveConsciousnessScoreAsync(Guid encounterId)
{
// Try GCS first
var gcsValues = await cache.StringGetAsync(GcsCalculator.AllComponentKeys(encounterId));
if (gcsValues.All(v => v.HasValue))
{
var total = GcsCalculator.ComputeTotal(eye, verbal, motor)!.Value;
return News2Calculator.ScoreConsciousnessFromGcs(total);
}
// Fallback to AVPU
var avpuVal = await cache.StringGetAsync(
News2Calculator.ParameterKey(encounterId, "AVPU"));
return cached?.Score;
}
```
---
## GCS — Glasgow Coma Scale
GCS measures level of consciousness through 3 components:
| Component | Range | What It Assesses |
|-----------|-------|-----------------|
| Eye (E) | 14 | Eye opening response |
| Verbal (V) | 15 | Best verbal response |
| Motor (M) | 16 | Best motor response |
| **Total** | **315** | **Sum of all three** |
```csharp
public static string ClassifyGcs(int total) => total switch
{
<= 8 => "SEVERE", // Coma — Critical alert
<= 12 => "MODERATE", // Warning alert
_ => "MILD" // 13-15 — Normal or near-normal
};
```
### Component Assembly
Like NEWS2, GCS components may arrive separately. The `GcsDetector` caches each component in Redis and computes the total when all 3 are present:
```
14:00 GCS_EYE=3 → Redis: gcs:{enc}:GCS_EYE = 3 (1/3)
14:00 GCS_VERBAL=4 → Redis: gcs:{enc}:GCS_VERBAL = 4 (2/3)
14:01 GCS_MOTOR=5 → Redis: gcs:{enc}:GCS_MOTOR = 5 (3/3)
→ Total = 12 (MODERATE)
→ Persist GcsScore to PostgreSQL
→ Create GCS_WARNING alert
→ Publish gcs.scored to Kafka (for SOFA CNS rescoring)
→ Re-evaluate qSOFA altered mentation
```
### Downstream Effects
GCS has the most downstream effects of any scoring engine:
1. **GCS alert**: SEVERE (total <= 8) → Critical alert, MODERATE (912) → Warning alert
2. **Kafka `gcs.scored` event**: Triggers SOFA CNS organ re-scoring
3. **NEWS2 consciousness**: Replaces AVPU with GCS-derived score
4. **qSOFA altered mentation**: GCS < 15 counts as one of the 3 qSOFA criteria
---
## qSOFA — Quick SOFA (Sepsis Screen)
qSOFA is a bedside screening tool with 3 binary criteria:
| Criterion | Threshold | Meaning |
|-----------|-----------|---------|
| Respiratory rate | >= 22 /min | Breathing fast |
| Systolic blood pressure | <= 100 mmHg | Blood pressure low |
| Altered mentation | GCS < 15 or AVPU >= 1 | Not fully alert |
If **2 or more criteria** are met simultaneously, a `QSOFA_SCREEN` warning alert fires, recommending SOFA labs be ordered.
### Sliding Window with Redis TTL
Each criterion has a **30-minute TTL** in Redis. If a criterion is met, the key is set with a 30-minute expiry. If not met, the key is deleted immediately:
```csharp
if (QsofaCalculator.MeetsCriterion(observationCode, value))
await cache.StringSetAsync(key, value.ToString(), TimeSpan.FromSeconds(1800));
else
await cache.KeyDeleteAsync(key);
```
This means a patient's respiratory rate of 24 (meets criterion) at 14:00 expires automatically at 14:30 if no new high respiratory rate arrives. The qSOFA score naturally decreases as criteria expire.
### Alert Deduplication
qSOFA uses `INSERT ... WHERE NOT EXISTS` to prevent duplicate screen alerts:
```sql
INSERT INTO clinical_alerts (...)
SELECT ... WHERE NOT EXISTS (
SELECT 1 FROM clinical_alerts
WHERE encounter_id = @encounterId
AND alert_type = 'QSOFA_SCREEN'
AND status IN ('OPEN', 'ESCALATED')
)
```
If an open qSOFA screen already exists for this encounter, no duplicate is created.
---
## SOFA — Sequential Organ Failure Assessment
SOFA is the most complex scoring system. It assesses 6 organ systems, each scored 04:
| Organ System | Data Source | Score 0 | Score 4 |
|-------------|------------|---------|---------|
| Respiratory | PaO2/FiO2 ratio (or SpO2/FiO2 fallback) | >= 400 | < 100 with mechanical ventilation |
| Coagulation | Platelet count | >= 150 k/µL | < 20 k/µL |
| Liver | Bilirubin | < 1.2 mg/dL | >= 12 mg/dL |
| Cardiovascular | MAP and vasopressor dose | MAP >= 70, no vasopressors | High-dose epinephrine/norepinephrine |
| CNS | GCS total (via `gcs.scored` Kafka event) | GCS 15 | GCS < 6 |
| Renal | Creatinine and urine output | Creatinine < 1.2 | Creatinine >= 5.0 or urine < 200 mL/day |
Total SOFA score: 024 (sum of all organ scores).
### Baseline and Delta
SOFA doesn't alert on the absolute score — it alerts on the **change from baseline**:
1. **Baseline established**: When >= 4 organ systems have data, the first score becomes the baseline
2. **Delta calculated**: Every subsequent score computes `delta = current total - baseline total`
3. **Alerts**:
- Delta >= 2 → `SOFA_SEPSIS` (Critical) — indicates acute organ dysfunction, triggers a sepsis bundle
- Delta = 1 → `SOFA_WARNING` (Warning)
This is clinically important because a patient with chronic kidney disease might have a baseline SOFA of 4. A score of 4 is not alarming for them — but a sudden jump to 6 (delta = 2) indicates new organ dysfunction.
### Lab Staleness Tracking
Lab values (platelets, bilirubin, creatinine) arrive infrequently — sometimes only once per day. The `SofaLabCache` tracks how old each value is:
| Age | Status | Behavior |
|-----|--------|----------|
| < 12 hours | Current | Used as-is |
| 1224 hours | Stale | Used but flagged in `staleness_flags` JSONB |
| > 24 hours | Expired | Organ score set to 0 (assume normal) |
### SpO2/FiO2 Fallback
Many ward patients don't have arterial blood gas (PaO2) measurements. When PaO2 is unavailable but SpO2 (pulse oximetry) is available, the SOFA detector uses a validated proxy ratio (Rice et al., 2007):
```csharp
if (pao2 is not null && fio2 is not null)
respiratory = SofaCalculator.ScoreRespiratory(pao2, fio2, onMechanicalVent);
else if (_options.UseSpO2FiO2Fallback && spo2 is not null && fio2 is not null)
respiratory = SofaCalculator.ScoreRespiratoryFromSpo2(spo2, fio2, onMechanicalVent);
```
---
## How the Scores Connect
The scoring engines are not independent — they feed into each other:
```
Observation arrives
├── NEWS2: uses RESP_RATE, SPO2, SYSTOLIC_BP, HEART_RATE, TEMP_C,
│ SUPPLEMENTAL_O2, and consciousness (from GCS or AVPU)
├── qSOFA: uses RESP_RATE, SYSTOLIC_BP, and altered mentation (from GCS)
├── GCS: uses GCS_EYE, GCS_VERBAL, GCS_MOTOR
│ │
│ └── publishes gcs.scored ──► SOFA (CNS organ)
│ ──► NEWS2 (consciousness)
│ ──► qSOFA (altered mentation)
├── SOFA: uses labs + vitals + GCS + medications
│ │
│ └── SOFA_SEPSIS (delta >= 2) ──► Sepsis Bundle (Guide 23)
└── Trend: uses HEART_RATE, RESP_RATE, SYSTOLIC_BP, TEMP_C, SPO2
```
---
## Common Patterns Across All Engines
### Alert Suppression
Warning-level alerts check for suppression before firing:
```csharp
if (alertType == AlertType.News2Warning)
{
var suppression = _services.GetRequiredService<IAlertSuppressionService>();
if (await suppression.IsSuppressedAsync(encounterId, alertType, ct))
return false;
}
```
When a clinician acknowledges a WARNING alert, a Redis suppression key is set for 30 minutes. During that window, the same alert type won't fire again for that encounter. CRITICAL alerts are never suppressed.
### Duplicate Prevention
Every alert creation uses `INSERT ... WHERE NOT EXISTS` to prevent duplicates. If an open alert of the same type already exists for the encounter, no new alert is created.
### Prometheus Metrics
Every engine increments counters and records timing:
```csharp
using var timer = _metrics.SofaScoringDuration.NewTimer();
// ... compute score ...
_metrics.SofaScoresTotal.WithLabels(alertCreated ? "true" : "false").Inc();
```
---
## Key Takeaways
- **Scoring engines are Kafka consumers** — each runs independently, processing the same observation events in parallel
- **Redis aggregates parameters over time** — observations arrive individually; Redis holds partial state until all parameters are present
- **Calculator + Detector separation** — pure scoring logic is testable without infrastructure; orchestration logic handles Redis, PostgreSQL, and Kafka
- **TTL-based expiry prevents stale scores** — parameters automatically expire (30 minutes for qSOFA, 4 hours for NEWS2, 24 hours for SOFA labs)
- **Scoring engines cascade** — GCS feeds into NEWS2, qSOFA, and SOFA. SOFA delta >= 2 triggers sepsis bundles. One observation can ripple through multiple scoring pipelines.
- **Alert suppression prevents alarm fatigue** — warning alerts are silenced for 30 minutes after acknowledgment; critical alerts always fire
+320
View File
@@ -0,0 +1,320 @@
# Guide 23: Sepsis Bundle Automation
## What is a Sepsis Bundle?
**Sepsis** is a life-threatening condition where the body's response to an infection damages its own organs. It's one of the leading causes of death in hospitals, and early treatment dramatically improves survival. The **Surviving Sepsis Campaign** defines a set of mandatory interventions (a "bundle") that must be completed within 1 hour of sepsis recognition:
| Element | What It Is | Why It's Urgent |
|---------|-----------|----------------|
| Blood cultures | Draw blood samples before antibiotics | Identifies the infecting organism so treatment can be targeted |
| Serum lactate | Blood test for lactate level | High lactate indicates tissue damage from inadequate blood flow |
| Broad-spectrum antibiotics | Administer antibiotics immediately | Every hour of delay increases mortality by ~8% |
| IV fluid resuscitation | Administer 30 mL/kg crystalloid fluids | Restores blood volume and organ perfusion |
**What is a "bundle" in software terms?** It's a checklist of 4 orders that the system creates automatically when sepsis is detected. Each element is tracked as PENDING → COMPLETED, and the bundle as a whole is tracked as IN_PROGRESS → COMPLIANT or NON_COMPLIANT based on whether all 4 elements are completed within the 1-hour deadline.
---
## Why Automate Sepsis Bundles?
Without automation, a nurse sees a sepsis alert, mentally recalls the 4-element bundle, manually creates each order, and tracks compliance on paper. In a busy ICU with multiple deteriorating patients, elements get missed or delayed. Automation ensures:
1. **Instant order creation**: All 4 orders are created the moment sepsis is detected — no manual recall needed
2. **Deadline tracking**: The 1-hour clock starts automatically
3. **Compliance monitoring**: A background service checks every 5 minutes for overdue bundles
4. **Audit trail**: Every bundle is recorded with its triggering alert, deadline, and outcome
---
## Architecture Overview
```
SOFA score computed (delta >= 2 from baseline)
SofaDetector creates SOFA_SEPSIS alert
SepsisAlertHandler.OnSepsisAlertCreatedAsync()
SepsisBundleService.TryCreateBundleAsync()
├── Creates SepsisBundle (IN_PROGRESS, deadline = now + 1 hour)
├── Creates 4 SepsisBundleElements (PENDING)
├── Creates 4 Orders (orderedBy: "sepsis-bundle-engine")
└── All in one PostgreSQL transaction (atomic)
... 1 hour passes ...
SepsisBundleMonitorService (every 5 minutes)
├── Finds IN_PROGRESS bundles past deadline
└── Marks as NON_COMPLIANT if elements remain PENDING
```
---
## Trigger: SOFA Delta >= 2
Sepsis bundles are only triggered by `SOFA_SEPSIS` alerts — not by qSOFA screens, NEWS2 scores, or any other alert type:
```csharp
public class SepsisAlertHandler
{
public async Task OnSepsisAlertCreatedAsync(
Guid encounterId, Guid alertId, AlertType alertType, CancellationToken ct)
{
if (alertType != AlertType.SofaSepsis)
return; // Only SOFA_SEPSIS triggers a bundle
var bundle = await _bundleService.TryCreateBundleAsync(
encounterId, alertId, alertType, ct);
if (bundle is not null)
_logger.LogInformation(
"Sepsis bundle {BundleId} created for encounter {EncounterId}",
bundle.Id, encounterId);
}
}
```
**Why only SOFA_SEPSIS?** The Sepsis-3 definition requires evidence of organ dysfunction (SOFA delta >= 2 from baseline). A qSOFA screen (>= 2 criteria) is a bedside screen that recommends ordering SOFA labs — it doesn't confirm sepsis. Creating bundles on qSOFA would produce false positives. The clinical flow is: qSOFA screen → order labs → SOFA computed → if delta >= 2 → sepsis bundle.
---
## Bundle Creation: Atomic Transaction
The bundle, its 4 elements, and the 4 corresponding orders are all created in a single PostgreSQL transaction:
```csharp
public async Task<SepsisBundle?> TryCreateBundleAsync(
Guid encounterId, Guid alertId, AlertType alertType, CancellationToken ct)
{
// Idempotency: only one in-progress bundle per encounter
var existing = await _db.SepsisBundles
.AnyAsync(b => b.EncounterId == encounterId
&& b.ComplianceStatus == SepsisBundleComplianceStatus.InProgress, ct);
if (existing) return null;
await using var tx = await _db.Database.BeginTransactionAsync(ct);
var recognizedAt = DateTimeOffset.UtcNow;
var bundle = new SepsisBundle
{
Id = Guid.NewGuid(),
EncounterId = encounterId,
TriggeringAlertId = alertId,
TriggeringAlertType = alertType.ToDbString(),
RecognizedAt = recognizedAt,
DeadlineAt = recognizedAt.AddHours(1), // 1-hour compliance window
ComplianceStatus = SepsisBundleComplianceStatus.InProgress,
};
_db.SepsisBundles.Add(bundle);
// Create the 4 bundle elements with linked orders
var elements = new[]
{
("BLOOD_CULTURE", "Draw blood cultures (2 sets, aerobic + anaerobic)"),
("SERUM_LACTATE", "Obtain serum lactate level"),
("ANTIBIOTICS", "Administer broad-spectrum antibiotics"),
("IV_FLUIDS", "Begin IV crystalloid fluid resuscitation (30 mL/kg)"),
};
foreach (var (code, description) in elements)
{
var order = new Order
{
Id = Guid.NewGuid(),
EncounterId = encounterId,
OrderType = MapOrderType(code),
Description = description,
Status = OrderStatus.Pending,
OrderedBy = "sepsis-bundle-engine",
OrderedAt = recognizedAt,
};
_db.Orders.Add(order);
_db.SepsisBundleElements.Add(new SepsisBundleElement
{
Id = Guid.NewGuid(),
BundleId = bundle.Id,
ElementCode = code,
OrderId = order.Id,
Status = SepsisBundleElementStatus.Pending,
});
}
// Outbox events for downstream notification
_db.OutboxEvents.Add(/* sepsis.bundle.created event */);
await _db.SaveChangesAsync(ct);
await tx.CommitAsync(ct);
_metrics.SepsisBundleComplianceTotal.WithLabels("CREATED").Inc();
return bundle;
}
```
**Why atomic?** If the bundle is created but one of the orders fails, you'd have a partially-created bundle with missing elements — clinicians would see a checklist with items missing. The transaction ensures all-or-nothing: either all 4 elements and their orders exist, or none do.
**Why `orderedBy: "sepsis-bundle-engine"`?** This identifies auto-created orders vs manually-created ones. Clinicians see that the order was system-generated and can distinguish it from orders they placed themselves.
---
## Bundle Element Lifecycle
Each bundle element starts as PENDING and moves to COMPLETED when the linked order is resulted:
```
PENDING ──(order resulted)──► COMPLETED
```
When a clinician marks an order as "resulted" (e.g., blood cultures drawn, antibiotics administered), the corresponding bundle element is updated. When all 4 elements are COMPLETED before the deadline, the bundle transitions:
```
IN_PROGRESS ──(all 4 elements completed within 1 hour)──► COMPLIANT
```
---
## Compliance Monitoring: SepsisBundleMonitorService
A background service runs every 5 minutes and checks for overdue bundles:
```csharp
public class SepsisBundleMonitorService : BackgroundService
{
private static readonly TimeSpan ScanInterval = TimeSpan.FromMinutes(5);
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
while (!stoppingToken.IsCancellationRequested)
{
try { await ScanOverdueBundlesAsync(stoppingToken); }
catch (Exception ex) when (ex is not OperationCanceledException)
{
_logger.LogError(ex, "Sepsis bundle monitor error — will retry");
}
await Task.Delay(ScanInterval, stoppingToken);
}
}
internal async Task ScanOverdueBundlesAsync(CancellationToken ct)
{
var overdue = await db.SepsisBundles
.Where(b => b.ComplianceStatus == SepsisBundleComplianceStatus.InProgress
&& b.DeadlineAt < DateTimeOffset.UtcNow)
.ToListAsync(ct);
foreach (var bundle in overdue)
{
bundle.ComplianceStatus = SepsisBundleComplianceStatus.NonCompliant;
_metrics.SepsisBundleComplianceTotal
.WithLabels("NON_COMPLIANT").Inc();
var incompleteCount = await db.SepsisBundleElements
.CountAsync(e => e.BundleId == bundle.Id
&& e.Status != SepsisBundleElementStatus.Completed, ct);
_logger.LogWarning(
"Sepsis bundle {BundleId} for encounter {EncounterId} marked NON_COMPLIANT — " +
"deadline {Deadline} passed with {Incomplete} incomplete elements",
bundle.Id, bundle.EncounterId, bundle.DeadlineAt, incompleteCount);
}
if (overdue.Count > 0)
await db.SaveChangesAsync(ct);
}
}
```
The scan finds all bundles that are still IN_PROGRESS but past their deadline, marks them NON_COMPLIANT, and logs which elements were incomplete. The `sepsis_bundle_compliance_total` Prometheus counter tracks compliance outcomes on the Grafana dashboard.
---
## Idempotency: One Bundle Per Encounter
The `TryCreateBundleAsync` method checks for existing in-progress bundles before creating a new one:
```csharp
var existing = await _db.SepsisBundles
.AnyAsync(b => b.EncounterId == encounterId
&& b.ComplianceStatus == SepsisBundleComplianceStatus.InProgress, ct);
if (existing) return null;
```
This prevents multiple bundles from being created if the SOFA score triggers multiple `SOFA_SEPSIS` alerts (e.g., if the score worsens further). Only one bundle can be in progress per encounter at a time.
---
## The Complete Sepsis Detection Timeline
```
t=0:00 qSOFA screen: resp_rate=24, systolic_bp=95 (2/3 criteria)
→ QSOFA_SCREEN warning alert
→ Recommendation: "Order SOFA labs"
t=0:30 Labs drawn: platelets, bilirubin, creatinine
t=1:00 Lab results arrive + vitals recorded
→ SOFA baseline established (total = 3)
t=2:00 Patient deteriorates — new labs + vitals
→ SOFA current = 6, delta = 3 from baseline
→ SOFA_SEPSIS critical alert created
→ SepsisAlertHandler triggers bundle creation
t=2:00 Sepsis bundle created (deadline = t=3:00):
✓ Blood cultures order (PENDING)
✓ Serum lactate order (PENDING)
✓ Antibiotics order (PENDING)
✓ IV fluids order (PENDING)
t=2:10 Nurse draws blood cultures → order resulted → element COMPLETED (1/4)
t=2:15 Lactate result arrives → element COMPLETED (2/4)
t=2:20 Antibiotics administered → element COMPLETED (3/4)
t=2:35 IV fluids initiated → element COMPLETED (4/4)
→ Bundle status: COMPLIANT (within 1-hour deadline)
-- OR --
t=3:00 Deadline passes with 2/4 elements still PENDING
→ SepsisBundleMonitorService marks: NON_COMPLIANT
→ Prometheus counter: sepsis_bundle_compliance_total{status="NON_COMPLIANT"}
```
---
## Database Schema
```
sepsis_bundles
├── id (UUID)
├── encounter_id (FK)
├── triggering_alert_id (FK)
├── triggering_alert_type ("SOFA_SEPSIS")
├── recognized_at (timestamp)
├── deadline_at (recognized_at + 1 hour)
├── compliance_status ("IN_PROGRESS" | "COMPLIANT" | "NON_COMPLIANT")
└── completed_at (nullable)
sepsis_bundle_elements
├── id (UUID)
├── bundle_id (FK → sepsis_bundles)
├── element_code ("BLOOD_CULTURE" | "SERUM_LACTATE" | "ANTIBIOTICS" | "IV_FLUIDS")
├── order_id (FK → orders)
└── status ("PENDING" | "COMPLETED")
```
---
## Key Takeaways
- **Only SOFA_SEPSIS triggers bundles** — qSOFA is a screen, not a confirmation. Bundles require evidence of organ dysfunction (SOFA delta >= 2).
- **Atomic creation ensures completeness** — all 4 elements and orders are created in one transaction; no partially-created bundles
- **1-hour compliance window is automatically enforced** — the deadline is set at creation time and checked every 5 minutes
- **One bundle per encounter at a time** — prevents duplicate bundles from repeated SOFA alerts during deterioration
- **Auto-created orders are labeled** — `orderedBy: "sepsis-bundle-engine"` distinguishes automated from manual orders
- **Compliance is tracked as a Prometheus metric** — trends in COMPLIANT vs NON_COMPLIANT rates are visible on the dashboard for quality improvement
@@ -0,0 +1,287 @@
# Guide 24: Trend Detection (Rate-of-Change Analysis)
## What is Trend Detection?
Traditional threshold alerts fire when a vital sign crosses a fixed boundary — "heart rate above 130, alert." But what about a heart rate that's at 85, then 95, then 105, then 115 — all within 30 minutes? No single reading crosses the threshold, but the patient is clearly deteriorating rapidly.
**Trend detection** (also called rate-of-change analysis) watches the _speed_ at which a vital sign is changing over time. It calculates the **velocity** — how many units per minute the value is rising or falling — and fires an alert when the velocity exceeds a threshold.
Think of it like a speedometer vs a position marker. A threshold alert says "you're past the speed limit." A trend alert says "you're accelerating dangerously fast and will hit the speed limit soon."
```
Heart Rate over 30 minutes:
130 ┤ ← Threshold (CRITICAL_HEART_RATE)
│ ╱
120 ┤ ╱╱
│ ╱╱
110 ┤ ╱╱
│ ╱╱
100 ┤ ╱╱
│╱╱
90 ┤ ← No individual reading crosses 130...
│ but the RATE (0.8 bpm/min) exceeds the trend threshold (0.5)
└──────────────────────────────────────
0 5 10 15 20 25 30 min
→ RAPID_DETERIORATION alert fires at ~20 minutes
```
---
## How Trend Detection Works
### The 5 Tracked Parameters
```csharp
public static class TrendCalculator
{
public static readonly IReadOnlyList<string> TrendCodes = new[]
{
"HEART_RATE", "RESP_RATE", "SYSTOLIC_BP", "TEMP_C", "SPO2"
};
}
```
Each parameter has a configured velocity threshold (from `appsettings.json`):
```json
{
"TrendDetection": {
"WindowMinutes": 30,
"MaxHistoryEntries": 10,
"HistoryTtlSeconds": 7200,
"RateThresholdsPerMinute": {
"HEART_RATE": 0.5,
"RESP_RATE": 0.3,
"SYSTOLIC_BP": 1.0,
"TEMP_C": 0.05,
"SPO2": 0.2
}
}
}
```
| Parameter | Threshold | Meaning |
|-----------|-----------|---------|
| HEART_RATE | 0.5 /min/min | Heart rate rising by 0.5 bpm each minute (15 bpm over 30 min) |
| RESP_RATE | 0.3 /min/min | Respiratory rate rising by 0.3/min each minute |
| SYSTOLIC_BP | 1.0 mmHg/min | Blood pressure dropping by 1 mmHg each minute (30 mmHg in 30 min) |
| TEMP_C | 0.05 °C/min | Temperature rising by 0.05°C each minute (1.5°C in 30 min) |
| SPO2 | 0.2 %/min | Oxygen saturation dropping by 0.2% each minute |
### Direction Matters
For HEART_RATE, RESP_RATE, and TEMP_C, rising values are concerning (tachycardia, tachypnea, fever). For SPO2 and SYSTOLIC_BP, falling values are concerning (desaturation, hypotension). The calculator handles this:
```csharp
public static bool ExceedsThreshold(
string observationCode, decimal ratePerMinute, decimal thresholdPerMinute) =>
observationCode switch
{
"SPO2" or "SYSTOLIC_BP" => ratePerMinute <= -thresholdPerMinute,
_ => ratePerMinute >= thresholdPerMinute
};
```
For SPO2: a rate of -0.3 %/min (dropping) exceeds the threshold of 0.2 (because |-0.3| > 0.2). For HEART_RATE: a rate of +0.7 bpm/min (rising) exceeds the threshold of 0.5.
---
## The Trend Detection Pipeline
```
Observation arrives (Kafka consumer: trend-analyzer)
Is it a trend code? (HEART_RATE, RESP_RATE, etc.)
│ no → return NotTrendCode
│ yes
Read history from Redis: trend:{encounterId}:{code}
Append new entry, trim to window (30 min, max 10 entries)
Write updated history back to Redis (2-hour TTL)
Enough history? (need >= 2 data points)
│ no → return InsufficientHistory
│ yes
Compute velocity: (newest value - oldest value) / time difference
Exceeds threshold?
│ no → return Stable
│ yes
Create RAPID_DETERIORATION alert (if not already open)
```
### Redis History Storage
The trend detector stores a list of recent readings in a single Redis key as a JSON array:
```csharp
var cache = _redis.GetDatabase();
var key = TrendCalculator.HistoryKey(encounterId, observationCode);
// Read existing history
var historyJson = await cache.StringGetAsync(key);
var history = historyJson.HasValue
? JsonSerializer.Deserialize<List<TrendHistoryEntry>>(historyJson!)
: new List<TrendHistoryEntry>();
// Append new entry
history.Add(new TrendHistoryEntry(value, recordedAt));
// Trim: remove entries outside window, keep max N entries
var cutoff = recordedAt.AddMinutes(-_options.WindowMinutes);
history = history
.Where(e => e.RecordedAt >= cutoff)
.TakeLast(_options.MaxHistoryEntries) // max 10
.ToList();
// Write back with TTL
await cache.StringSetAsync(
key, JsonSerializer.Serialize(history),
TimeSpan.FromSeconds(_options.HistoryTtlSeconds)); // 2 hours
```
**Why a JSON list instead of a Redis list?** Redis lists support push/pop operations, but trimming by time range (remove entries older than 30 minutes) requires scanning the entire list. Storing the entire history as a JSON string allows the application to deserialize, filter, and reserialize in one read-write cycle. At max 10 entries, the overhead is negligible.
### Velocity Calculation
```csharp
public static decimal? ComputeRatePerMinute(
IReadOnlyList<TrendHistoryEntry> entries, int windowMinutes)
{
if (entries.Count < 2) return null;
var newest = entries[^1]; // last entry
var oldest = entries[0]; // first entry
var deltaMinutes = (newest.RecordedAt - oldest.RecordedAt).TotalMinutes;
if (deltaMinutes <= 0 || deltaMinutes > windowMinutes) return null;
return (newest.Value - oldest.Value) / (decimal)deltaMinutes;
}
```
**Simple linear velocity**: The rate is `(newest - oldest) / time elapsed`. This is intentionally simple — no weighted averages, no curve fitting. For clinical safety, a simple slope between the oldest and newest readings in the window is sufficient and easy to reason about.
**Why require deltaMinutes > 0?** Two readings with identical timestamps would produce division by zero. Why check `> windowMinutes`? If the oldest and newest readings span more than the window (e.g., a stale entry wasn't properly trimmed), the rate would be artificially diluted.
---
## Alert Creation
When the velocity exceeds the threshold, a `RAPID_DETERIORATION` alert is created:
```csharp
var details = TrendCalculator.DescribeTrend(observationCode, rate.Value, value);
// "Rapid rise: HEART_RATE rising at 0.72/min (current 118)"
// "Rapid decline: SPO2 falling at 0.31/min (current 91)"
var affected = await db.Database.ExecuteSqlInterpolatedAsync($"""
INSERT INTO clinical_alerts
(id, encounter_id, patient_id, alert_type, severity, details,
observation_code, status, triggered_at)
SELECT {alertId}, {encounterId}, {patientId},
'RAPID_DETERIORATION', 'WARNING', {fullDetails},
{observationCode}, 'OPEN', {triggeredAt}
WHERE NOT EXISTS (
SELECT 1 FROM clinical_alerts
WHERE encounter_id = {encounterId}
AND alert_type = 'RAPID_DETERIORATION'
AND observation_code = {observationCode}
AND status IN ('OPEN', 'ESCALATED')
)
""", ct);
```
Key design decisions:
- **Alert severity is WARNING**, not CRITICAL — trend detection is predictive ("the patient may deteriorate"), not confirmatory ("the patient has a dangerous vital sign"). The actual threshold breach alert (CRITICAL_HEART_RATE, CRITICAL_SPO2) fires separately when the value crosses the absolute threshold.
- **`observation_code` in the WHERE clause** — a patient can have simultaneous trend alerts for different parameters (rising heart rate AND falling SpO2), but only one per parameter.
- **Outbox event for downstream consumers** — the alert appears on the dashboard, triggers Kafka consumers (ES indexer, notification publisher), and may eventually escalate through RabbitMQ if unacknowledged.
---
## The Kafka Consumer: TrendAnalyzerService
```csharp
public class TrendAnalyzerService : BackgroundService
{
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
var config = new ConsumerConfig
{
BootstrapServers = _kafkaOptions.BootstrapServers,
GroupId = "trend-analyzer",
AutoOffsetReset = AutoOffsetReset.Earliest,
EnableAutoCommit = false
};
using var consumer = new ConsumerBuilder<string, string>(config).Build();
consumer.Subscribe(_kafkaOptions.Topics.ObservationRecorded);
var guard = new PoisonPillGuard("trend-analyzer",
_kafkaOptions.MaxPoisonRetries, _logger);
while (!stoppingToken.IsCancellationRequested)
{
var result = consumer.Consume(stoppingToken);
var evt = JsonSerializer.Deserialize<TrendObservationEvent>(result.Message.Value)!;
using var scope = _services.CreateScope();
var detector = scope.ServiceProvider.GetRequiredService<TrendDetector>();
var outcome = await detector.ProcessObservationAsync(
evt.EncounterId, evt.PatientId,
evt.ObservationCode, evt.Value, evt.RecordedAt, stoppingToken);
if (outcome.Outcome == TrendOutcome.RapidDeterioration)
_logger.LogWarning(
"RAPID_DETERIORATION for {Code} in encounter {EncounterId}",
evt.ObservationCode, evt.EncounterId);
consumer.Commit(result);
guard.OnSuccess();
}
}
}
```
This follows the standard Kafka consumer pattern from Guide 12: consume → process → commit → repeat.
---
## Example Scenario
A patient in the ICU has these heart rate readings over 20 minutes:
| Time | Heart Rate | History Window | Velocity |
|------|-----------|---------------|----------|
| 14:00 | 82 | [82] | — (need >= 2) |
| 14:05 | 88 | [82, 88] | +1.2/min (exceeds 0.5) |
| 14:10 | 95 | [82, 88, 95] | +1.3/min |
| 14:15 | 103 | [82, 88, 95, 103] | +1.4/min |
| 14:20 | 112 | [82, 88, 95, 103, 112] | +1.5/min |
At 14:05, the velocity (1.2/min) already exceeds the threshold (0.5/min). A `RAPID_DETERIORATION` alert fires. The alert details: "Rapid rise: HEART_RATE rising at 1.20/min (current 88) — velocity 1.20/min over 30min window."
The individual readings (82, 88, 95...) are all normal — none cross the critical threshold of 130. But the trend detector catches the rapid acceleration before any threshold is breached.
---
## Key Takeaways
- **Trend detection catches deterioration early** — before absolute thresholds are breached, alerting clinicians to accelerating decline
- **Velocity is simple and interpretable** — (newest - oldest) / time. No complex statistics. Clinicians can understand "heart rate rising at 0.7 bpm/min."
- **Direction-aware thresholds** — rising heart rate is bad, but rising SpO2 is good. Falling SpO2 is bad, but falling heart rate may be fine. Each parameter knows which direction to watch.
- **Redis history with JSON lists** — lightweight storage for sliding-window data with automatic TTL expiry
- **WARNING severity for predictive alerts** — trend alerts warn about future risk; threshold alerts confirm current danger. Both are needed.
- **One trend alert per parameter per encounter** — deduplication prevents alert storms when a patient is continuously deteriorating