vigilcare-clinical/docs/guides/07-rabbitmq-notification-queuing.md

Guide 7: RabbitMQ for Notification Queuing

What is RabbitMQ?

RabbitMQ is a message broker — a middleman that accepts, routes, and delivers messages between parts of your application. Think of it like a post office: producers drop off letters (messages), the post office sorts them into the right mailboxes (queues), and consumers pick them up.

Key concepts:

• Queue: A named buffer that stores messages until a consumer processes them. Unlike Kafka (where messages persist for all consumer groups), RabbitMQ queues are point-to-point by default — once a consumer acknowledges a message, it's removed from the queue.
• Exchange: A routing layer that sits in front of queues. Producers send messages to an exchange (not directly to a queue), and the exchange decides which queue(s) to route each message to based on routing rules.
• Routing Key: A string attached to each message that the exchange uses to decide routing. With a direct exchange, the routing key must exactly match the queue's binding key.
• Binding: A rule that connects an exchange to a queue with a specific routing key. "Route messages with key alerts.paging to the queue alerts.paging.queue."
• ACK (Acknowledge): When a consumer successfully processes a message, it sends an ACK back to RabbitMQ, which removes the message from the queue.
• NACK (Negative Acknowledge): When a consumer fails to process a message, it sends a NACK. The message can either be requeued (try again) or sent to a dead-letter queue.
• Dead-Letter Queue (DLQ): A special queue where "rejected" messages go. Instead of losing failed messages, they're stored in the DLQ for later analysis or automatic retry.

How is RabbitMQ different from Kafka? Kafka is designed for high-throughput event streaming where many consumers read the same stream independently. RabbitMQ is designed for task distribution — each message is processed by exactly one consumer, and the broker manages acknowledgments and retries. In this project, Kafka handles the "fan-out" (one event → 9 consumers), while RabbitMQ handles point-to-point workflows (page a physician, generate a discharge summary).

---

Why RabbitMQ in This Project?

When a critical alert fires, someone needs to be paged. If they don't respond, the alert must escalate to a backup. This requires a workflow with timeouts, acknowledgment tracking, and dead-letter routing — exactly what RabbitMQ excels at. Kafka handles the high-volume event streaming; RabbitMQ handles the delivery-guaranteed notification workflows.

---

Architecture Overview

Kafka Consumer                            RabbitMQ
(NotificationPublisher)                   ┌─────────────────────────────────────┐
        │                                 │  clinical.notifications.exchange    │
        │  BasicPublish                   │         (direct exchange)           │
        ▼                                 └──────┬──────┬──────┬──────┬────────┘
                                                 │      │      │      │
                routing key:               alerts. alerts. notif. notif.
                                           paging  escal. disch. recon.
                                                 │      │      │      │
                                                 ▼      ▼      ▼      ▼
                                          ┌────────┐ ┌──────┐ ┌─────┐ ┌──────┐
                                          │paging  │ │escal.│ │disch│ │recon.│
                                          │.queue  │ │.queue│ │.queue│ │.queue│
                                          └───┬────┘ └──────┘ └─────┘ └──────┘
                                              │         ▲
                                    NACK      │         │  after TTL expires
                                    (timeout) ▼         │
                                          ┌────────┐    │
                                          │paging  │────┘
                                          │.dlq    │  (dead-letter re-routes
                                          └────────┘   to escalation queue)

---

Configuration

public sealed class RabbitMqOptions
{
    public const string Section = "RabbitMq";
    public string Host { get; init; } = "localhost";
    public int Port { get; init; } = 5674;
    public string Username { get; init; } = "guest";
    public string Password { get; init; } = "guest";
    public string VirtualHost { get; init; } = "/";
    public int PagingAckTimeoutMs { get; init; } = 300000;  // 5 minutes
}

{
  "RabbitMq": {
    "Host": "localhost",
    "Port": 5674,
    "Username": "guest",
    "Password": "guest",
    "PagingAckTimeoutMs": 300000
  }
}

The PagingAckTimeoutMs drives two things: how long the paging worker waits for a physician to acknowledge, and the TTL on the dead-letter queue.

Connection Factory

public static class RabbitMqConnectionFactory
{
    public static ConnectionFactory Create(RabbitMqOptions o, bool dispatchConsumersAsync = false)
        => new()
    {
        HostName               = o.Host,
        Port                   = o.Port,
        UserName               = o.Username,
        Password               = o.Password,
        VirtualHost            = o.VirtualHost,
        DispatchConsumersAsync = dispatchConsumersAsync,
    };
}

What is DispatchConsumersAsync? By default, the RabbitMQ .NET client delivers messages to consumers on a synchronous thread. Setting this to true lets consumers use async/await in their message handlers — necessary when the handler needs to call the database or other async services.

---

Topology Provisioning

What is topology? In RabbitMQ, "topology" means the structure of exchanges, queues, and bindings. Before any message can flow, these must exist. The RabbitMqTopologyProvisioner creates them on application startup:

public sealed class RabbitMqTopologyProvisioner : IHostedService
{
    public const string Exchange     = "clinical.notifications.exchange";
    public const string PagingKey    = "alerts.paging";
    public const string EscalKey     = "alerts.escalation";
    public const string DischargeKey = "notifications.discharge";

    public Task StartAsync(CancellationToken ct)
    {
        // Create the exchange
        channel.ExchangeDeclare(Exchange, ExchangeType.Direct, durable: true);

        // Create queues and bind them to the exchange with routing keys
        channel.QueueDeclare("alerts.paging.queue", durable: true, ...);
        channel.QueueBind("alerts.paging.queue", Exchange, PagingKey);

        channel.QueueDeclare("alerts.escalation.queue", durable: true, ...);
        channel.QueueBind("alerts.escalation.queue", Exchange, EscalKey);

        channel.QueueDeclare("notifications.discharge.queue", durable: true, ...);
        channel.QueueBind("notifications.discharge.queue", Exchange, DischargeKey);

        // ... more queues
    }
}

durable: true means the queue survives a RabbitMQ restart. Without this, restarting the broker would delete the queue and all its messages.

All declare operations are idempotent — if the queue already exists with the same settings, RabbitMQ does nothing. This makes startup safe to run multiple times.

The Paging Queue and Dead-Letter Chain

The paging queue has special arguments that set up automatic escalation:

// alerts.paging.queue — dead-letters unacknowledged messages to the DLQ
channel.QueueDeclare(
    queue: "alerts.paging.queue",
    durable: true,
    arguments: new Dictionary<string, object>
    {
        ["x-dead-letter-exchange"]    = "",                    // default exchange
        ["x-dead-letter-routing-key"] = "alerts.paging.dlq",  // DLQ queue name
    });

// alerts.paging.dlq — messages expire after PagingAckTimeoutMs, then re-route to escalation
channel.QueueDeclare(
    queue: "alerts.paging.dlq",
    durable: true,
    arguments: new Dictionary<string, object>
    {
        ["x-message-ttl"]             = opts.PagingAckTimeoutMs,  // 5 minutes
        ["x-dead-letter-exchange"]    = Exchange,                 // back to main exchange
        ["x-dead-letter-routing-key"] = EscalKey,                 // → alerts.escalation.queue
    });

What is x-message-ttl? A queue-level TTL (time-to-live). Any message sitting in this queue for longer than this duration is automatically removed. Combined with x-dead-letter-exchange, removed messages are re-routed instead of deleted.

This creates a chain: paging queue → NACK → DLQ → wait 5 minutes → escalation queue. No application code manages the delay — RabbitMQ handles it automatically.

Immutable TTL gotcha: x-message-ttl cannot be changed after a queue is created. If you need to change the timeout (e.g., from 5 minutes to 5 seconds for testing), the provisioner must delete and recreate the DLQ. The code handles this with a try/catch:

try
{
    channel.QueueDeclare(dlq, durable: true, arguments: args);
}
catch (OperationInterruptedException ex)
{
    // TTL mismatch — delete and recreate
    cleanup.QueueDelete(dlq);
    cleanup.QueueDeclare(dlq, durable: true, arguments: args);
}

---

The Paging Worker

The paging worker is the core notification workflow. It consumes messages from alerts.paging.queue, pages the attending physician, then polls the database waiting for acknowledgment:

public sealed class PagingWorkerService : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        var factory = RabbitMqConnectionFactory.Create(o, dispatchConsumersAsync: true);
        using var connection = factory.CreateConnection("paging-worker");
        using var channel    = connection.CreateModel();

        // Process one message at a time
        channel.BasicQos(prefetchSize: 0, prefetchCount: 1, global: false);

        var consumer = new AsyncEventingBasicConsumer(channel);
        consumer.Received += async (sender, ea) =>
        {
            try
            {
                await HandlePageAsync(channel, ea, stoppingToken);
            }
            catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested)
            {
                // App shutting down — requeue so restart doesn't false-escalate
                channel.BasicNack(ea.DeliveryTag, multiple: false, requeue: true);
            }
            catch (Exception ex)
            {
                // Unexpected error — send to DLQ for escalation
                channel.BasicNack(ea.DeliveryTag, multiple: false, requeue: false);
            }
        };

        channel.BasicConsume("alerts.paging.queue", autoAck: false, consumer);
        await Task.Delay(Timeout.Infinite, stoppingToken);
    }
}

What is BasicQos (prefetch)? Prefetch controls how many unacknowledged messages RabbitMQ sends to the consumer at once. prefetchCount: 1 means "send me one message, wait for my ACK before sending the next one." For paging, this ensures the worker handles one alert at a time — you don't want to be simultaneously waiting on acknowledgments for 10 different alerts.

What is autoAck: false? When true, RabbitMQ considers the message acknowledged the moment it's delivered to the consumer. When false (manual acknowledgment), the consumer must explicitly ACK or NACK. Manual mode is safer because if the consumer crashes before finishing, the message is re-delivered.

The Paging and Acknowledgment Loop

private async Task HandlePageAsync(IModel channel, BasicDeliverEventArgs ea, CancellationToken ct)
{
    var alertId = Guid.Parse(doc.RootElement.GetProperty("alertId").GetString()!);

    _logger.LogWarning(
        "[PAGE] Paging attending physician '{Physician}' for encounter {EncounterId}",
        physician, encounterId);

    var deadline = DateTimeOffset.UtcNow.AddMilliseconds(o.PagingAckTimeoutMs);

    // Poll the database every 2 seconds waiting for acknowledgment
    while (DateTimeOffset.UtcNow < deadline && !ct.IsCancellationRequested)
    {
        await Task.Delay(2_000, ct);

        var acknowledged = await IsAlertAcknowledgedAsync(alertId, ct);
        if (acknowledged)
        {
            // Physician acknowledged — ACK the message (removes from queue)
            channel.BasicAck(ea.DeliveryTag, multiple: false);
            return;
        }
    }

    // Timeout: NACK with requeue=false → message goes to DLQ
    // After x-message-ttl expires on the DLQ, it re-routes to escalation queue
    channel.BasicNack(ea.DeliveryTag, multiple: false, requeue: false);
}

The flow:

1. Page sent — log the page to the attending physician
2. Poll loop — check the database every 2 seconds for up to 5 minutes
3. If acknowledged — BasicAck removes the message. Done.
4. If timeout — BasicNack(requeue: false) sends the message to the DLQ. After x-message-ttl expires, RabbitMQ automatically routes it to the escalation queue.

Graceful Shutdown

When the application is stopping, in-flight messages are requeued (requeue: true) instead of NACKed to the DLQ. This prevents false escalations caused by application restarts.

---

The Escalation Worker

The escalation worker consumes from alerts.escalation.queue — messages that reached here because no one acknowledged the page within the timeout:

private async Task HandleEscalationAsync(IModel channel, BasicDeliverEventArgs ea, CancellationToken ct)
{
    var alertId = Guid.Parse(doc.RootElement.GetProperty("alertId").GetString()!);

    _logger.LogCritical("[ESCALATION] Paging on-call backup for alert {AlertId}", alertId);

    // Update the alert status to "Escalated" in the database
    var escalated = await UpdateAlertStatusEscalatedAsync(alertId, ct);
    if (escalated)
        _metrics.EscalationsTotal.Inc();

    channel.BasicAck(ea.DeliveryTag, multiple: false);
}

private async Task<bool> UpdateAlertStatusEscalatedAsync(Guid alertId, CancellationToken ct)
{
    var alert = await db.ClinicalAlerts.FindAsync(new object[] { alertId }, ct);
    if (alert is null) return false;

    // Only escalate if still open — if acknowledged between NACK and TTL expiry, leave it
    if (alert.Status != AlertStatus.Open) return false;

    alert.Status = AlertStatus.Escalated;
    await db.SaveChangesAsync(ct);
    return true;
}

The status != AlertStatus.Open check handles a race condition: if the physician acknowledges the alert while the message sits in the DLQ, the escalation worker should not overwrite the acknowledgment.

---

The Discharge Summary Worker

When a patient is discharged, a message arrives on notifications.discharge.queue. The worker generates a text summary and uploads it to MinIO:

private async Task HandleDischargeSummaryAsync(IModel channel, BasicDeliverEventArgs ea, CancellationToken ct)
{
    var encounterId = Guid.Parse(doc.RootElement.GetProperty("encounterId").GetString()!);

    var summary = await BuildSummaryAsync(encounterId, ct);
    await UploadToMinioAsync(encounterId, summary, ct);

    channel.BasicAck(ea.DeliveryTag, multiple: false);
}

On success: ACK. On failure: NACK with requeue: true (retry later — maybe MinIO was temporarily unreachable).

---

Queue Summary

Queue	Routing Key	Consumer	Purpose
alerts.paging.queue	alerts.paging	PagingWorkerService	Page attending physician, wait for ACK
alerts.paging.dlq	(dead-letter from paging queue)	(none — auto-routes after TTL)	Hold unacknowledged pages before escalation
alerts.escalation.queue	alerts.escalation	EscalationWorkerService	Page on-call backup, mark alert escalated
notifications.discharge.queue	notifications.discharge	DischargeSummaryWorkerService	Generate and upload discharge summary
notifications.reconciliation.queue	notifications.reconciliation	ReconciliationScheduler	Safety findings (unacked alerts, pending orders)
clinical.sync.batch_received	sync.batch_received	ClinicalSyncBatchConsumer	Process gateway sync batches

---

The Complete Escalation Timeline

t=0:00   Critical alert fires
         → OutboxRelay → Kafka (alert.generated)
         → NotificationPublisher → RabbitMQ (alerts.paging.queue)

t=0:00   PagingWorker picks up message
         → Logs "[PAGE] Paging attending physician..."
         → Starts polling database every 2 seconds

t=0:00   IF physician acknowledges within 5 minutes:
  to       → PagingWorker sees ACK in database
t=5:00     → BasicAck — message removed from queue
           → Flow complete ✓

t=5:00   IF no acknowledgment after 5 minutes:
         → PagingWorker BasicNack(requeue: false)
         → Message routes to alerts.paging.dlq
         → DLQ holds message for PagingAckTimeoutMs (another 5 min)

t=10:00  DLQ x-message-ttl expires
         → Message auto-routes to alerts.escalation.queue

t=10:00  EscalationWorker picks up message
         → Logs "[ESCALATION] Paging on-call backup..."
         → Sets alert status → Escalated
         → Increments escalations_total metric
         → BasicAck — flow complete

---

Health Check

public sealed class RabbitMqHealthCheck : IHealthCheck
{
    public async Task<HealthCheckResult> CheckHealthAsync(
        HealthCheckContext context, CancellationToken cancellationToken)
    {
        var factory = RabbitMqConnectionFactory.Create(_options);
        using var connection = await Task.Run(
            () => factory.CreateConnection(), cancellationToken);
        return HealthCheckResult.Healthy(
            data: new Dictionary<string, object>
                { ["endpoint"] = connection.Endpoint.ToString() });
    }
}

Opens a connection to verify RabbitMQ is reachable. Used by the /health/ready endpoint.

---

Key Takeaways

• RabbitMQ is for workflows, Kafka is for streaming — use RabbitMQ when you need acknowledgments, timeouts, and dead-letter routing. Use Kafka when you need one event consumed by many independent groups.
• Dead-letter queues + TTL = delayed retry — RabbitMQ's built-in features handle the escalation timer without any application-level scheduling code.
• Manual acknowledgment is essential — autoAck: false ensures messages are only removed after successful processing, preventing data loss on crashes.
• Prefetch controls concurrency — prefetchCount: 1 on the paging worker ensures one alert is paged at a time. The escalation worker uses prefetchCount: 5 because escalation handling is fast (just a DB update).
• Topology provisioning is idempotent — declare operations safely re-run on every application restart. The exception is immutable queue arguments like x-message-ttl, which require delete-and-recreate.