# Guide 7: RabbitMQ for Notification Queuing

## What is RabbitMQ?

**RabbitMQ** is a message broker — a middleman that accepts, routes, and delivers messages between parts of your application. Think of it like a post office: producers drop off letters (messages), the post office sorts them into the right mailboxes (queues), and consumers pick them up.

Key concepts:

- **Queue**: A named buffer that stores messages until a consumer processes them. Unlike Kafka (where messages persist for all consumer groups), RabbitMQ queues are point-to-point by default — once a consumer acknowledges a message, it's removed from the queue.
- **Exchange**: A routing layer that sits in front of queues. Producers send messages to an exchange (not directly to a queue), and the exchange decides which queue(s) to route each message to based on routing rules.
- **Routing Key**: A string attached to each message that the exchange uses to decide routing. With a **direct exchange**, the routing key must exactly match the queue's binding key.
- **Binding**: A rule that connects an exchange to a queue with a specific routing key. "Route messages with key `alerts.paging` to the queue `alerts.paging.queue`."
- **ACK (Acknowledge)**: When a consumer successfully processes a message, it sends an ACK back to RabbitMQ, which removes the message from the queue.
- **NACK (Negative Acknowledge)**: When a consumer fails to process a message, it sends a NACK. The message can either be requeued (try again) or sent to a dead-letter queue.
- **Dead-Letter Queue (DLQ)**: A special queue where "rejected" messages go. Instead of losing failed messages, they're stored in the DLQ for later analysis or automatic retry.

**How is RabbitMQ different from Kafka?** Kafka is designed for high-throughput event streaming where many consumers read the same stream independently. RabbitMQ is designed for task distribution — each message is processed by exactly one consumer, and the broker manages acknowledgments and retries. In this project, Kafka handles the "fan-out" (one event → 9 consumers), while RabbitMQ handles point-to-point workflows (page a physician, generate a discharge summary).

---

## Why RabbitMQ in This Project?

When a critical alert fires, someone needs to be paged. If they don't respond, the alert must escalate to a backup. This requires a workflow with timeouts, acknowledgment tracking, and dead-letter routing — exactly what RabbitMQ excels at. Kafka handles the high-volume event streaming; RabbitMQ handles the delivery-guaranteed notification workflows.

---

## Architecture Overview

```
Kafka Consumer                            RabbitMQ
(NotificationPublisher)                   ┌─────────────────────────────────────┐
        │                                 │  clinical.notifications.exchange    │
        │  BasicPublish                   │         (direct exchange)           │
        ▼                                 └──────┬──────┬──────┬──────┬────────┘
                                                 │      │      │      │
                routing key:               alerts. alerts. notif. notif.
                                           paging  escal. disch. recon.
                                                 │      │      │      │
                                                 ▼      ▼      ▼      ▼
                                          ┌────────┐ ┌──────┐ ┌─────┐ ┌──────┐
                                          │paging  │ │escal.│ │disch│ │recon.│
                                          │.queue  │ │.queue│ │.queue│ │.queue│
                                          └───┬────┘ └──────┘ └─────┘ └──────┘
                                              │         ▲
                                    NACK      │         │  after TTL expires
                                    (timeout) ▼         │
                                          ┌────────┐    │
                                          │paging  │────┘
                                          │.dlq    │  (dead-letter re-routes
                                          └────────┘   to escalation queue)
```

---

## Configuration

```csharp
public sealed class RabbitMqOptions
{
    public const string Section = "RabbitMq";
    public string Host { get; init; } = "localhost";
    public int Port { get; init; } = 5674;
    public string Username { get; init; } = "guest";
    public string Password { get; init; } = "guest";
    public string VirtualHost { get; init; } = "/";
    public int PagingAckTimeoutMs { get; init; } = 300000;  // 5 minutes
}
```

```json
{
  "RabbitMq": {
    "Host": "localhost",
    "Port": 5674,
    "Username": "guest",
    "Password": "guest",
    "PagingAckTimeoutMs": 300000
  }
}
```

The `PagingAckTimeoutMs` drives two things: how long the paging worker waits for a physician to acknowledge, and the TTL on the dead-letter queue.

### Connection Factory

```csharp
public static class RabbitMqConnectionFactory
{
    public static ConnectionFactory Create(RabbitMqOptions o, bool dispatchConsumersAsync = false)
        => new()
    {
        HostName               = o.Host,
        Port                   = o.Port,
        UserName               = o.Username,
        Password               = o.Password,
        VirtualHost            = o.VirtualHost,
        DispatchConsumersAsync = dispatchConsumersAsync,
    };
}
```

**What is `DispatchConsumersAsync`?** By default, the RabbitMQ .NET client delivers messages to consumers on a synchronous thread. Setting this to `true` lets consumers use `async/await` in their message handlers — necessary when the handler needs to call the database or other async services.

---

## Topology Provisioning

**What is topology?** In RabbitMQ, "topology" means the structure of exchanges, queues, and bindings. Before any message can flow, these must exist. The `RabbitMqTopologyProvisioner` creates them on application startup:

```csharp
public sealed class RabbitMqTopologyProvisioner : IHostedService
{
    public const string Exchange     = "clinical.notifications.exchange";
    public const string PagingKey    = "alerts.paging";
    public const string EscalKey     = "alerts.escalation";
    public const string DischargeKey = "notifications.discharge";

    public Task StartAsync(CancellationToken ct)
    {
        // Create the exchange
        channel.ExchangeDeclare(Exchange, ExchangeType.Direct, durable: true);

        // Create queues and bind them to the exchange with routing keys
        channel.QueueDeclare("alerts.paging.queue", durable: true, ...);
        channel.QueueBind("alerts.paging.queue", Exchange, PagingKey);

        channel.QueueDeclare("alerts.escalation.queue", durable: true, ...);
        channel.QueueBind("alerts.escalation.queue", Exchange, EscalKey);

        channel.QueueDeclare("notifications.discharge.queue", durable: true, ...);
        channel.QueueBind("notifications.discharge.queue", Exchange, DischargeKey);

        // ... more queues
    }
}
```

**`durable: true`** means the queue survives a RabbitMQ restart. Without this, restarting the broker would delete the queue and all its messages.

All declare operations are **idempotent** — if the queue already exists with the same settings, RabbitMQ does nothing. This makes startup safe to run multiple times.

### The Paging Queue and Dead-Letter Chain

The paging queue has special arguments that set up automatic escalation:

```csharp
// alerts.paging.queue — dead-letters unacknowledged messages to the DLQ
channel.QueueDeclare(
    queue: "alerts.paging.queue",
    durable: true,
    arguments: new Dictionary<string, object>
    {
        ["x-dead-letter-exchange"]    = "",                    // default exchange
        ["x-dead-letter-routing-key"] = "alerts.paging.dlq",  // DLQ queue name
    });

// alerts.paging.dlq — messages expire after PagingAckTimeoutMs, then re-route to escalation
channel.QueueDeclare(
    queue: "alerts.paging.dlq",
    durable: true,
    arguments: new Dictionary<string, object>
    {
        ["x-message-ttl"]             = opts.PagingAckTimeoutMs,  // 5 minutes
        ["x-dead-letter-exchange"]    = Exchange,                 // back to main exchange
        ["x-dead-letter-routing-key"] = EscalKey,                 // → alerts.escalation.queue
    });
```

**What is `x-message-ttl`?** A queue-level TTL (time-to-live). Any message sitting in this queue for longer than this duration is automatically removed. Combined with `x-dead-letter-exchange`, removed messages are re-routed instead of deleted.

This creates a chain: paging queue → NACK → DLQ → wait 5 minutes → escalation queue. No application code manages the delay — RabbitMQ handles it automatically.

**Immutable TTL gotcha**: `x-message-ttl` cannot be changed after a queue is created. If you need to change the timeout (e.g., from 5 minutes to 5 seconds for testing), the provisioner must delete and recreate the DLQ. The code handles this with a try/catch:

```csharp
try
{
    channel.QueueDeclare(dlq, durable: true, arguments: args);
}
catch (OperationInterruptedException ex)
{
    // TTL mismatch — delete and recreate
    cleanup.QueueDelete(dlq);
    cleanup.QueueDeclare(dlq, durable: true, arguments: args);
}
```

---

## The Paging Worker

The paging worker is the core notification workflow. It consumes messages from `alerts.paging.queue`, pages the attending physician, then polls the database waiting for acknowledgment:

```csharp
public sealed class PagingWorkerService : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        var factory = RabbitMqConnectionFactory.Create(o, dispatchConsumersAsync: true);
        using var connection = factory.CreateConnection("paging-worker");
        using var channel    = connection.CreateModel();

        // Process one message at a time
        channel.BasicQos(prefetchSize: 0, prefetchCount: 1, global: false);

        var consumer = new AsyncEventingBasicConsumer(channel);
        consumer.Received += async (sender, ea) =>
        {
            try
            {
                await HandlePageAsync(channel, ea, stoppingToken);
            }
            catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested)
            {
                // App shutting down — requeue so restart doesn't false-escalate
                channel.BasicNack(ea.DeliveryTag, multiple: false, requeue: true);
            }
            catch (Exception ex)
            {
                // Unexpected error — send to DLQ for escalation
                channel.BasicNack(ea.DeliveryTag, multiple: false, requeue: false);
            }
        };

        channel.BasicConsume("alerts.paging.queue", autoAck: false, consumer);
        await Task.Delay(Timeout.Infinite, stoppingToken);
    }
}
```

**What is `BasicQos` (prefetch)?** Prefetch controls how many unacknowledged messages RabbitMQ sends to the consumer at once. `prefetchCount: 1` means "send me one message, wait for my ACK before sending the next one." For paging, this ensures the worker handles one alert at a time — you don't want to be simultaneously waiting on acknowledgments for 10 different alerts.

**What is `autoAck: false`?** When true, RabbitMQ considers the message acknowledged the moment it's delivered to the consumer. When false (manual acknowledgment), the consumer must explicitly ACK or NACK. Manual mode is safer because if the consumer crashes before finishing, the message is re-delivered.

### The Paging and Acknowledgment Loop

```csharp
private async Task HandlePageAsync(IModel channel, BasicDeliverEventArgs ea, CancellationToken ct)
{
    var alertId = Guid.Parse(doc.RootElement.GetProperty("alertId").GetString()!);

    _logger.LogWarning(
        "[PAGE] Paging attending physician '{Physician}' for encounter {EncounterId}",
        physician, encounterId);

    var deadline = DateTimeOffset.UtcNow.AddMilliseconds(o.PagingAckTimeoutMs);

    // Poll the database every 2 seconds waiting for acknowledgment
    while (DateTimeOffset.UtcNow < deadline && !ct.IsCancellationRequested)
    {
        await Task.Delay(2_000, ct);

        var acknowledged = await IsAlertAcknowledgedAsync(alertId, ct);
        if (acknowledged)
        {
            // Physician acknowledged — ACK the message (removes from queue)
            channel.BasicAck(ea.DeliveryTag, multiple: false);
            return;
        }
    }

    // Timeout: NACK with requeue=false → message goes to DLQ
    // After x-message-ttl expires on the DLQ, it re-routes to escalation queue
    channel.BasicNack(ea.DeliveryTag, multiple: false, requeue: false);
}
```

The flow:
1. **Page sent** — log the page to the attending physician
2. **Poll loop** — check the database every 2 seconds for up to 5 minutes
3. **If acknowledged** — `BasicAck` removes the message. Done.
4. **If timeout** — `BasicNack(requeue: false)` sends the message to the DLQ. After `x-message-ttl` expires, RabbitMQ automatically routes it to the escalation queue.

### Graceful Shutdown

When the application is stopping, in-flight messages are requeued (`requeue: true`) instead of NACKed to the DLQ. This prevents false escalations caused by application restarts.

---

## The Escalation Worker

The escalation worker consumes from `alerts.escalation.queue` — messages that reached here because no one acknowledged the page within the timeout:

```csharp
private async Task HandleEscalationAsync(IModel channel, BasicDeliverEventArgs ea, CancellationToken ct)
{
    var alertId = Guid.Parse(doc.RootElement.GetProperty("alertId").GetString()!);

    _logger.LogCritical("[ESCALATION] Paging on-call backup for alert {AlertId}", alertId);

    // Update the alert status to "Escalated" in the database
    var escalated = await UpdateAlertStatusEscalatedAsync(alertId, ct);
    if (escalated)
        _metrics.EscalationsTotal.Inc();

    channel.BasicAck(ea.DeliveryTag, multiple: false);
}

private async Task<bool> UpdateAlertStatusEscalatedAsync(Guid alertId, CancellationToken ct)
{
    var alert = await db.ClinicalAlerts.FindAsync(new object[] { alertId }, ct);
    if (alert is null) return false;

    // Only escalate if still open — if acknowledged between NACK and TTL expiry, leave it
    if (alert.Status != AlertStatus.Open) return false;

    alert.Status = AlertStatus.Escalated;
    await db.SaveChangesAsync(ct);
    return true;
}
```

The `status != AlertStatus.Open` check handles a race condition: if the physician acknowledges the alert while the message sits in the DLQ, the escalation worker should not overwrite the acknowledgment.

---

## The Discharge Summary Worker

When a patient is discharged, a message arrives on `notifications.discharge.queue`. The worker generates a text summary and uploads it to MinIO:

```csharp
private async Task HandleDischargeSummaryAsync(IModel channel, BasicDeliverEventArgs ea, CancellationToken ct)
{
    var encounterId = Guid.Parse(doc.RootElement.GetProperty("encounterId").GetString()!);

    var summary = await BuildSummaryAsync(encounterId, ct);
    await UploadToMinioAsync(encounterId, summary, ct);

    channel.BasicAck(ea.DeliveryTag, multiple: false);
}
```

On success: ACK. On failure: NACK with `requeue: true` (retry later — maybe MinIO was temporarily unreachable).

---

## Queue Summary

| Queue | Routing Key | Consumer | Purpose |
|-------|-------------|----------|---------|
| `alerts.paging.queue` | `alerts.paging` | `PagingWorkerService` | Page attending physician, wait for ACK |
| `alerts.paging.dlq` | (dead-letter from paging queue) | (none — auto-routes after TTL) | Hold unacknowledged pages before escalation |
| `alerts.escalation.queue` | `alerts.escalation` | `EscalationWorkerService` | Page on-call backup, mark alert escalated |
| `notifications.discharge.queue` | `notifications.discharge` | `DischargeSummaryWorkerService` | Generate and upload discharge summary |
| `notifications.reconciliation.queue` | `notifications.reconciliation` | `ReconciliationScheduler` | Safety findings (unacked alerts, pending orders) |
| `clinical.sync.batch_received` | `sync.batch_received` | `ClinicalSyncBatchConsumer` | Process gateway sync batches |

---

## The Complete Escalation Timeline

```
t=0:00   Critical alert fires
         → OutboxRelay → Kafka (alert.generated)
         → NotificationPublisher → RabbitMQ (alerts.paging.queue)

t=0:00   PagingWorker picks up message
         → Logs "[PAGE] Paging attending physician..."
         → Starts polling database every 2 seconds

t=0:00   IF physician acknowledges within 5 minutes:
  to       → PagingWorker sees ACK in database
t=5:00     → BasicAck — message removed from queue
           → Flow complete ✓

t=5:00   IF no acknowledgment after 5 minutes:
         → PagingWorker BasicNack(requeue: false)
         → Message routes to alerts.paging.dlq
         → DLQ holds message for PagingAckTimeoutMs (another 5 min)

t=10:00  DLQ x-message-ttl expires
         → Message auto-routes to alerts.escalation.queue

t=10:00  EscalationWorker picks up message
         → Logs "[ESCALATION] Paging on-call backup..."
         → Sets alert status → Escalated
         → Increments escalations_total metric
         → BasicAck — flow complete
```

---

## Health Check

```csharp
public sealed class RabbitMqHealthCheck : IHealthCheck
{
    public async Task<HealthCheckResult> CheckHealthAsync(
        HealthCheckContext context, CancellationToken cancellationToken)
    {
        var factory = RabbitMqConnectionFactory.Create(_options);
        using var connection = await Task.Run(
            () => factory.CreateConnection(), cancellationToken);
        return HealthCheckResult.Healthy(
            data: new Dictionary<string, object>
                { ["endpoint"] = connection.Endpoint.ToString() });
    }
}
```

Opens a connection to verify RabbitMQ is reachable. Used by the `/health/ready` endpoint.

---

## Key Takeaways

- **RabbitMQ is for workflows, Kafka is for streaming** — use RabbitMQ when you need acknowledgments, timeouts, and dead-letter routing. Use Kafka when you need one event consumed by many independent groups.
- **Dead-letter queues + TTL = delayed retry** — RabbitMQ's built-in features handle the escalation timer without any application-level scheduling code.
- **Manual acknowledgment is essential** — `autoAck: false` ensures messages are only removed after successful processing, preventing data loss on crashes.
- **Prefetch controls concurrency** — `prefetchCount: 1` on the paging worker ensures one alert is paged at a time. The escalation worker uses `prefetchCount: 5` because escalation handling is fast (just a DB update).
- **Topology provisioning is idempotent** — declare operations safely re-run on every application restart. The exception is immutable queue arguments like `x-message-ttl`, which require delete-and-recreate.
