287 lines
14 KiB
Markdown
287 lines
14 KiB
Markdown
# Guide 8: Elasticsearch as a CQRS Read Store
|
|
|
|
## What is Elasticsearch?
|
|
|
|
**Elasticsearch** is a distributed search and analytics engine. While PostgreSQL excels at transactional operations (INSERT, UPDATE, JOIN), Elasticsearch excels at searching across large amounts of data — full-text search, filtering, aggregations (sums, averages, counts grouped by category), and fuzzy matching.
|
|
|
|
Elasticsearch stores data in **indexes** (similar to database tables). Each index contains **documents** (similar to rows) in JSON format. Unlike a relational database, Elasticsearch doesn't require you to define a rigid schema upfront — though you should define **mappings** (field types) for predictable behavior.
|
|
|
|
Key terminology:
|
|
- **Index**: A collection of documents (like a table). Example: `patient_encounters`, `observations`.
|
|
- **Document**: A single JSON record (like a row). Identified by an ID.
|
|
- **Mapping**: The schema definition for an index — which fields exist and what type they are (`keyword` for exact match, `text` for full-text search, `date` for timestamps, etc.).
|
|
- **Keyword vs Text**: A `keyword` field stores the value as-is for exact matching and sorting ("ICU" must match exactly). A `text` field is analyzed (split into tokens, lowercased) for full-text search ("John Smith" matches a search for "john").
|
|
- **Aggregation**: A computation across documents — like SQL's `GROUP BY`, `COUNT`, `AVG`. Elasticsearch can aggregate millions of documents in milliseconds.
|
|
|
|
## What is CQRS?
|
|
|
|
**CQRS** stands for **Command Query Responsibility Segregation**. The core idea: use different data stores (or different models) for writes and reads.
|
|
|
|
- **Command side** (writes): Your application writes to PostgreSQL — it handles transactions, constraints, and data integrity.
|
|
- **Query side** (reads): Your application reads from Elasticsearch — it handles search, filtering, and analytics.
|
|
|
|
Why separate them? Because the ideal data structure for writing (normalized tables with foreign keys and constraints) is different from the ideal structure for reading (denormalized documents with all related data in one place). A single read from Elasticsearch can return a patient's name, encounter status, department, alert count, and latest NEWS2 score — without any JOINs.
|
|
|
|
The bridge between the two is an event stream. When data changes in PostgreSQL, an event is published to Kafka. A consumer reads that event and updates Elasticsearch. This means Elasticsearch is **eventually consistent** — there's a small delay (usually under a second) between a write to PostgreSQL and the data appearing in Elasticsearch.
|
|
|
|
---
|
|
|
|
## Why Elasticsearch in This Project?
|
|
|
|
The dashboard needs to search patients by name or MRN, filter encounters by department, see alert counts by severity, and display observation trends — all in near-real-time. These are analytics and search queries that Elasticsearch handles orders of magnitude faster than PostgreSQL with JOINs across large tables.
|
|
|
|
---
|
|
|
|
## Architecture Overview
|
|
|
|
```
|
|
Write Path (PostgreSQL) Read Path (Elasticsearch)
|
|
┌──────────────┐ ┌──────────────────────┐
|
|
│ API writes │ │ Dashboard queries │
|
|
│ observations,│ │ AnalyticsService │
|
|
│ alerts, │ │ SearchPatientsAsync │
|
|
│ encounters │ │ GetAlertSummaryAsync │
|
|
└──────┬───────┘ └──────────┬───────────┘
|
|
│ │
|
|
▼ ▼
|
|
PostgreSQL ──outbox──► Kafka ──es-indexer──► Elasticsearch
|
|
│
|
|
┌──────────┴──────────┐
|
|
│ patient_encounters │
|
|
│ observations │
|
|
│ clinical_alerts │
|
|
└─────────────────────┘
|
|
```
|
|
|
|
---
|
|
|
|
## Index Provisioning
|
|
|
|
Indexes and their mappings are created on application startup by `ElasticIndexProvisioner`:
|
|
|
|
```csharp
|
|
public class ElasticIndexProvisioner : IHostedService
|
|
{
|
|
public async Task StartAsync(CancellationToken cancellationToken)
|
|
{
|
|
await EnsureIndexAsync<PatientEncounterDocument>(
|
|
_options.Indices.PatientEncounters, BuildPatientEncountersMapping());
|
|
await EnsureIndexAsync<ObservationDocument>(
|
|
_options.Indices.Observations, BuildObservationsMapping());
|
|
await EnsureIndexAsync<ClinicalAlertDocument>(
|
|
_options.Indices.ClinicalAlerts, BuildClinicalAlertsMapping());
|
|
}
|
|
|
|
private async Task EnsureIndexAsync<T>(string indexName,
|
|
Action<CreateIndexRequestDescriptor<T>> configure) where T : class
|
|
{
|
|
var exists = await _elastic.Indices.ExistsAsync(indexName);
|
|
if (exists.Exists) return; // idempotent — skip if already exists
|
|
|
|
var resp = await _elastic.Indices.CreateAsync<T>(indexName, configure);
|
|
if (!resp.IsValidResponse)
|
|
throw new InvalidOperationException(
|
|
$"Failed to create index '{indexName}': {resp.DebugInformation}");
|
|
}
|
|
}
|
|
```
|
|
|
|
### The Patient Encounters Mapping
|
|
|
|
```csharp
|
|
private Action<CreateIndexRequestDescriptor<PatientEncounterDocument>>
|
|
BuildPatientEncountersMapping() =>
|
|
d => d.Mappings(m => m.Properties(p => p
|
|
.Keyword(k => k.EncounterId)
|
|
.Keyword(k => k.PatientId)
|
|
.Keyword(k => k.Mrn)
|
|
// text for full-text search + keyword sub-field for exact sort/filter
|
|
.Text(t => t.PatientName, tf => tf
|
|
.Fields(f => f.Keyword(k => k.PatientName)))
|
|
.Keyword(k => k.Department)
|
|
.Keyword(k => k.Status)
|
|
.IntegerNumber(i => i.News2Score!)
|
|
.Keyword(k => k.News2RiskLevel!)
|
|
.Date(d => d.AdmittedAt)
|
|
.IntegerNumber(i => i.OpenAlertCount)
|
|
.Date(d => d.LastObservationAt!)
|
|
));
|
|
```
|
|
|
|
The `PatientName` field has a **multi-field mapping**: it's stored as both `text` (for full-text search — searching "john" matches "John Smith") and `keyword` (for exact sorting and filtering). This is a common Elasticsearch pattern for fields that need both search and sort capabilities.
|
|
|
|
---
|
|
|
|
## The ES Indexer Service (Kafka → Elasticsearch)
|
|
|
|
`EsIndexerService` is the bridge between the write side (PostgreSQL/Kafka) and the read side (Elasticsearch). It's a Kafka consumer in the `es-indexer` consumer group that subscribes to 5 topics and routes each message to the appropriate handler:
|
|
|
|
```csharp
|
|
consumer.Subscribe(new[]
|
|
{
|
|
_kafkaOptions.Topics.ObservationRecorded,
|
|
_kafkaOptions.Topics.AlertGenerated,
|
|
_kafkaOptions.Topics.EncounterStatusChanged,
|
|
_kafkaOptions.Topics.SepsisBundleCreated,
|
|
_kafkaOptions.Topics.SepsisBundleUpdated
|
|
});
|
|
|
|
private Task DispatchAsync(string topic, string payload, CancellationToken ct)
|
|
=> topic switch
|
|
{
|
|
var t when t == _kafkaOptions.Topics.EncounterStatusChanged
|
|
=> HandleEncounterStatusChangedAsync(payload, ct),
|
|
var t when t == _kafkaOptions.Topics.ObservationRecorded
|
|
=> HandleObservationRecordedAsync(payload, ct),
|
|
var t when t == _kafkaOptions.Topics.AlertGenerated
|
|
=> HandleAlertGeneratedAsync(payload, ct),
|
|
// ... sepsis bundle topics
|
|
};
|
|
```
|
|
|
|
### Upsert for Encounters
|
|
|
|
**What is an upsert?** "Update or insert" — if the document exists, update it. If it doesn't exist, create it. This makes the operation **idempotent**: processing the same event twice produces the same result.
|
|
|
|
```csharp
|
|
var resp = await _elastic.UpdateAsync<PatientEncounterDocument, PatientEncounterDocument>(
|
|
_esOptions.Indices.PatientEncounters,
|
|
evt.EncounterId.ToString(),
|
|
u => u.Doc(doc).DocAsUpsert(true), // create if missing, replace if exists
|
|
ct);
|
|
```
|
|
|
|
### Append-Only for Observations
|
|
|
|
Observations are indexed by their unique `observationId`. Since observation IDs never repeat, this is naturally idempotent:
|
|
|
|
```csharp
|
|
await _elastic.IndexAsync(
|
|
doc,
|
|
i => i.Index(_esOptions.Indices.Observations).Id(doc.ObservationId),
|
|
ct);
|
|
```
|
|
|
|
The indexer also updates `lastObservationAt` on the parent encounter document using a **Painless script** (Elasticsearch's built-in scripting language). The script handles out-of-order delivery — an older observation reprocessed after a newer one won't overwrite the timestamp:
|
|
|
|
```painless
|
|
if (ctx._source.lastObservationAt == null ||
|
|
params.recordedAt > ctx._source.lastObservationAt) {
|
|
ctx._source.lastObservationAt = params.recordedAt;
|
|
}
|
|
```
|
|
|
|
### Denormalization for Alerts
|
|
|
|
When an alert fires, the indexer both creates the alert document AND updates the parent encounter document:
|
|
|
|
```csharp
|
|
// 1. Index the alert
|
|
await _elastic.IndexAsync(alertDoc, i => i.Index("clinical_alerts").Id(alertDoc.AlertId), ct);
|
|
|
|
// 2. Increment openAlertCount on the encounter + stamp NEWS2 score if present
|
|
await _elastic.UpdateAsync<PatientEncounterDocument, object>(
|
|
"patient_encounters", evt.EncounterId.ToString(),
|
|
u => u.Script(new Script(new InlineScript
|
|
{
|
|
Source = "ctx._source.openAlertCount += 1",
|
|
Language = ScriptLanguage.Painless
|
|
})).RetryOnConflict(3), ct);
|
|
```
|
|
|
|
**What is denormalization?** In a relational database, you'd JOIN alerts to encounters to get the count. In Elasticsearch, you store the count directly on the encounter document. This eliminates JOINs (which Elasticsearch doesn't support well) but means you must keep the denormalized data in sync through your event handlers.
|
|
|
|
**`RetryOnConflict(3)`** handles optimistic concurrency — if two events update the same encounter document simultaneously, Elasticsearch retries the update up to 3 times.
|
|
|
|
---
|
|
|
|
## Querying Elasticsearch — The Analytics Service
|
|
|
|
The `AnalyticsService` demonstrates common Elasticsearch query patterns.
|
|
|
|
### Patient Search (Full-Text + Filters)
|
|
|
|
```csharp
|
|
public async Task<object> SearchPatientsAsync(string? q, string? department,
|
|
string? status, int page, int pageSize)
|
|
{
|
|
var resp = await _elastic.SearchAsync<PatientEncounterDocument>(s => s
|
|
.Indices(_options.Indices.PatientEncounters)
|
|
.Query(q2 => q2.Bool(b =>
|
|
{
|
|
if (!string.IsNullOrWhiteSpace(q))
|
|
{
|
|
b.Should(
|
|
// MRN: exact keyword match — boosted x3 because MRN lookup
|
|
// is the most common search for clinicians
|
|
s2 => s2.Term(t => t.Field(p => p.Mrn).Value(q).Boost(3)),
|
|
// Patient name: full-text search
|
|
s2 => s2.Match(m => m.Field(p => p.PatientName).Query(q)),
|
|
// Department: exact match
|
|
s2 => s2.Term(t => t.Field(p => p.Department).Value(q))
|
|
);
|
|
b.MinimumShouldMatch(1);
|
|
}
|
|
|
|
if (filters.Count > 0)
|
|
b.Filter(filters.ToArray());
|
|
}))
|
|
.From((page - 1) * pageSize)
|
|
.Size(pageSize));
|
|
|
|
return new { total = resp.Total, page, pageSize, data = resp.Documents };
|
|
}
|
|
```
|
|
|
|
**What is boosting?** `Boost(3)` means an MRN match is weighted 3x higher than a name match in relevance scoring. If a clinician searches for "12345" and that matches both a patient's MRN and part of their phone number, the MRN match ranks higher.
|
|
|
|
### Alert Summary with Aggregations
|
|
|
|
```csharp
|
|
var resp = await _elastic.SearchAsync<ClinicalAlertDocument>(s => s
|
|
.Indices(_options.Indices.ClinicalAlerts)
|
|
.Query(q => q.Bool(b => b.Filter(filters.ToArray())))
|
|
.Aggregations(a => a
|
|
.Add("by_department", agg => agg.Terms(t => t
|
|
.Field(a => a.Department).Size(50)))
|
|
)
|
|
.Size(0)); // no raw documents — aggregation result only
|
|
```
|
|
|
|
`.Size(0)` tells Elasticsearch "I don't need the actual documents, just the aggregation results." This is much faster when you only need counts or statistics.
|
|
|
|
### Population Query with Cardinality
|
|
|
|
```csharp
|
|
var resp = await _elastic.SearchAsync<ObservationDocument>(s => s
|
|
.Indices(_options.Indices.Observations)
|
|
.Query(q => q.Bool(b => b.Filter(filters.ToArray())))
|
|
.Aggregations(a => a
|
|
.Add("unique_patients", agg => agg
|
|
.Cardinality(c => c.Field(o => o.PatientId)))
|
|
)
|
|
.Size(0));
|
|
```
|
|
|
|
**What is cardinality?** It counts distinct values — like SQL's `COUNT(DISTINCT patient_id)`. "How many unique patients had a heart rate above 120?" Two observations from the same patient count as one patient.
|
|
|
|
---
|
|
|
|
## Index Summary
|
|
|
|
| Index | Document ID | Write Pattern | Use Case |
|
|
|-------|-------------|---------------|----------|
|
|
| `patient_encounters` | `encounterId` | Upsert (create or replace) | Dashboard ward table, patient search |
|
|
| `observations` | `observationId` | Append (index by unique ID) | Observation trend queries, population analytics |
|
|
| `clinical_alerts` | `alertId` | Append (index by unique ID) | Alert volume by department/severity |
|
|
|
|
---
|
|
|
|
## Key Takeaways
|
|
|
|
- **CQRS separates concerns**: PostgreSQL handles writes with transactions and constraints. Elasticsearch handles reads with search and aggregations. Each store is optimized for its purpose.
|
|
- **Events are the bridge**: Kafka events flow from the write side to the read side. The ES indexer consumer keeps Elasticsearch in sync with PostgreSQL.
|
|
- **Idempotency matters**: Upserts and unique document IDs ensure that reprocessing the same event (after a crash or retry) doesn't corrupt the data.
|
|
- **Denormalization eliminates JOINs**: The encounter document contains patient name, alert count, NEWS2 score, and sepsis bundle status — all in one document, readable in one query.
|
|
- **Eventual consistency is the tradeoff**: There's a small delay between writing to PostgreSQL and the data appearing in Elasticsearch. For a dashboard that refreshes every 5-10 seconds, this is imperceptible.
|