fix: Missing input validators + No patient update endpoint + Pagination inconsistencies + Missing list/get endpoints

docs/patient-encounter-api-lifecycle.md

@@ -10,6 +10,7 @@
 - [Medication correlation design](decisions/medication-correlation-design.md) — how drug context annotates warning/NEWS2 alerts
 - [Simulator guide](simulator-guide.md) — replay JSON scenarios against this lifecycle
 - [Dashboard guide](dashboard-guide.md) — which endpoints the Vue ward UI polls
+- [Mirth FHIR integration](integration/mirth-fhir-channels.md) — HL7v2 → FHIR R4 ingest via integration engines
 
 ---
 
@@ -81,13 +82,17 @@ When a nurse records a heart rate of 95 bpm, here is what happens under the hood
 3. COMMIT — everything saved atomically
 
 4. OutboxRelayService (runs every 500ms):
-   - Reads unsent outbox events from PostgreSQL
-   - Sends them to the correct Kafka topic
+   - Reads unsent outbox events from PostgreSQL (using FOR UPDATE SKIP LOCKED for safe concurrency)
+   - Sends them to the correct Kafka topic via an idempotent producer (broker deduplicates retries)
    - Marks them as processed
+   - If Kafka produce fails: increments retry counter and tries again on next poll
+   - After 10 failed attempts: marks the event as permanently failed (dead-lettered) — stops blocking other events
 ```
 
 **Why this pattern?** It guarantees that if the observation is saved, the message to Kafka will also be sent — even if Kafka is temporarily down. The database acts as a reliable staging area. This is called the "transactional outbox pattern."
 
+**What about permanently failed events?** If a message truly cannot be delivered after 10 attempts (corrupted payload, topic deleted, etc.), it is marked `FailedAt` and excluded from future relay polls. This prevents a single poisoned event from blocking all subsequent messages. Operators can query failed events via `SELECT * FROM outbox_events WHERE failed_at IS NOT NULL` for manual investigation.
+
 ### Kafka topics — the conveyor belts
 
 Each Kafka topic carries a specific type of message. Think of them as labeled conveyor belts in a factory:
@@ -103,6 +108,8 @@ Each Kafka topic carries a specific type of message. Think of them as labeled co
 
 **Key detail:** One message on `observation.recorded` is consumed by up to 8 different workers independently. Each worker has its own "consumer group" — Kafka tracks where each group left off, so every worker gets every message even if they process at different speeds.
 
+**Poison pill protection:** If a consumer receives a malformed or un-processable message (corrupted JSON, invalid format), the system does not get stuck retrying it forever. A "poison pill guard" detects permanent errors and skips them immediately. For transient errors (temporary network issues), it retries up to 5 times before giving up on that message. Skipped messages are logged and tracked by a Prometheus metric (`kafka_poison_pills_skipped_total`) so operators know if something is wrong with the data flowing through the system.
+
 ### Redis — the fast-lookup layer
 
 Redis stores data that needs to be checked on every observation ingest or scoring calculation. Reading from Redis takes less than 1 millisecond, compared to 5–50ms for a database query.
@@ -157,7 +164,7 @@ MinIO stores files organized by path, like a file system in the cloud:
 | Path pattern | What is stored | How it gets there |
 |---|---|---|
 | `discharge-summaries/{encounterId}/summary.pdf` | Discharge summary for each encounter | Generated by the discharge worker (RabbitMQ) when a patient is discharged |
-| `observations/{YYYY}/{MM}/{DD}/partition-{p}-offset-{o}.parquet` | Daily observation data in Parquet format | Written by the data lake writer (Kafka consumer) in batches of 1000 events or every 5 minutes |
+| `observations/{YYYY}/{MM}/{DD}/partition-{p}-offset-{o}.parquet` | Daily observation data in Parquet format | Written by the data lake writer (Kafka consumer) in batches of 1000 events or every 5 minutes; Kafka offsets are only committed for partitions where the upload succeeded — failed partitions are retried on the next flush |
 | `alerts/{YYYY}/{MM}/{DD}/partition-{p}-offset-{o}.parquet` | Daily alert data in Parquet format | Same as above |
 | `encounters/{YYYY}/{MM}/{DD}/partition-{p}-offset-{o}.parquet` | Daily encounter status changes in Parquet format | Same as above |
 
@@ -179,6 +186,7 @@ Prometheus scrapes the `/metrics` endpoint every 15 seconds and records operatio
 | `alerts_unacknowledged_gauge` | Gauge | Current count of critical alerts open for more than 5 minutes |
 | `outbox_pending_events` | Gauge | Unprocessed outbox events (if this grows, the relay is falling behind) |
 | `kafka_consumer_lag` | Gauge | How far behind each Kafka consumer is (if this grows, scoring is delayed) |
+| `kafka_poison_pills_skipped_total` | Counter | Messages skipped as un-processable (labeled by consumer group and topic). If this grows, investigate the source of malformed messages |
 
 Grafana displays these metrics as charts, and operators can set up alerts (system alerts, not clinical alerts) if things like consumer lag or pending outbox events climb too high.
 
@@ -199,6 +207,10 @@ All services run in Docker containers defined in `docker-compose.yml`:
 
 Starting the full stack: `docker compose up -d` brings up all 8 services. The .NET API runs separately (outside Docker) and connects to these services on the ports listed above.
 
+**Health check endpoints:** The API exposes two health endpoints (no authentication required):
+- `GET /health/live` — liveness probe. Returns 200 if the process is running. Use this for Kubernetes liveness probes or load balancer checks.
+- `GET /health/ready` — readiness probe. Checks connectivity to all five infrastructure services (PostgreSQL, Redis, Kafka, RabbitMQ, Elasticsearch). Returns 200 only when all services are reachable. Use this for readiness gates — if it returns unhealthy, the API cannot process requests properly.
+
 ---
 
 ## Conventions
@@ -208,6 +220,8 @@ Starting the full stack: `docker compose up -d` brings up all 8 services. The .N
 | Base path | `/api/v1` |
 | Response envelope | `{ success, statusCode, data, error }` |
 | Correlation | Optional `X-Correlation-Id` request header; echoed on the response |
+| Authentication | JWT bearer token required on all clinical endpoints; obtain via `POST /api/v1/auth/login` |
+| Health checks | `GET /health/live` (liveness) and `GET /health/ready` (readiness) — no authentication required |
 | Active encounter guard | `POST` observations, medications, and orders return **409** (`ENCOUNTER_NOT_ACTIVE`) when the encounter is discharged or cancelled |
 | Async latency | Warning alerts, NEWS2 scores, qSOFA, SOFA, trend alerts, and sepsis bundles are created by background workers — allow a few seconds after recording a measurement before polling for results |
 
@@ -496,8 +510,8 @@ These are typically set up once at deploy time, not called per patient. This is
 
 | What | How | Technology involved | Why it matters |
 |---|---|---|---|
-| Alert thresholds | Seeded in PostgreSQL; `ThresholdCacheLoader` copies them into Redis at startup; manageable via `POST/GET/PUT /alert-thresholds` | PostgreSQL → Redis | Every measurement is validated against thresholds from Redis (fast) to decide if alerts fire. Without this cache, every observation ingest would need a database query |
-| Kafka topics | `KafkaTopicProvisioner` creates 7 topics on startup (6 partitions each, auto-creation disabled) | Kafka | `observation.recorded`, `alert.generated`, `encounter.status.changed`, `sepsis.bundle.created`, `sepsis.bundle.updated`, `gcs.scored`. These must exist before any messages can flow |
+| Alert thresholds | Seeded in PostgreSQL; `ThresholdCacheLoader` copies them into Redis at startup (retries up to 3 times with exponential backoff if Redis is unavailable; application starts without cache if Redis remains down — falls back to PostgreSQL queries); manageable via `POST/GET/PUT /alert-thresholds` | PostgreSQL → Redis | Every measurement is validated against thresholds from Redis (fast) to decide if alerts fire. Without this cache, every observation ingest would need a database query |
+| Kafka topics | `KafkaTopicProvisioner` creates 7 topics on startup (6 partitions each, configurable replication factor — default 3, auto-creation disabled) | Kafka | `observation.recorded`, `alert.generated`, `encounter.status.changed`, `sepsis.bundle.created`, `sepsis.bundle.updated`, `gcs.scored`. These must exist before any messages can flow |
 | Elasticsearch indices | `ElasticIndexProvisioner` creates 3 indices with proper field mappings | Elasticsearch | `patient_encounters`, `observations`, `clinical_alerts` — must exist before data can be indexed |
 | RabbitMQ topology | `RabbitMqTopologyProvisioner` declares the exchange and 5 queues | RabbitMQ | Sets up the `clinical.notifications.exchange` and all queues (paging, DLQ, escalation, discharge, reconciliation) with proper routing and dead-letter configuration |
 | MinIO bucket | Created if it does not exist | MinIO | The `vigilcare` bucket must exist before discharge summaries or data lake files can be written |
@@ -518,7 +532,7 @@ POST /api/v1/patients
 | `firstName`, `lastName`, `dateOfBirth`, `gender` | yes | |
 | `bloodType`, `allergies`, `emergencyContactName`, `emergencyContactPhone` | no | Stored for ward context |
 
-**Response:** `201 Created` — `data` includes a system-generated `mrn` (e.g. `MRN-000042`) and `id` (UUID). Save both; the MRN is what nurses see on wristbands, the UUID is used in all API paths.
+**Response:** `201 Created` — `data` includes a system-generated `mrn` (e.g. `MRN-000042`) and `id` (UUID). Save both; the MRN is what nurses see on wristbands, the UUID is used in all API paths. MRNs are generated from a PostgreSQL sequence (`mrn_seq`), guaranteeing uniqueness even under concurrent registration.
 
 **Later lookups:**
 
@@ -811,7 +825,7 @@ GET /api/v1/sepsis-bundles/{id}
 
 **Compliance flow:**
 
-1. Bundle is created with `complianceStatus: IN_PROGRESS` and `deadlineAt` set to 1 hour from detection
+1. Bundle is created with `complianceStatus: IN_PROGRESS` and `deadlineAt` set to 1 hour from detection. A database constraint ensures only one in-progress bundle can exist per encounter at any time — preventing duplicate bundles from race conditions.
 2. Clinicians complete orders via `PATCH /orders/{id}/result`
 3. Each completed order marks its bundle element `COMPLETED`
 4. When all 4 elements are done:
@@ -912,7 +926,7 @@ A patient may later return and receive a new encounter via `POST /patients/{id}/
 | | `GET` | `/analytics/patients`, `/analytics/observations/trend`, `/analytics/alerts/summary`, `/analytics/population` |
 | **9 — Discharge** | `PATCH` | `/encounters/{id}/status` |
 
-**Operational (not encounter-scoped):** `GET /metrics` (Prometheus), Swagger UI (development only).
+**Operational (not encounter-scoped):** `GET /metrics` (Prometheus), `GET /health/live` (liveness probe), `GET /health/ready` (readiness probe — checks PostgreSQL, Redis, Kafka, RabbitMQ, Elasticsearch), Swagger UI (development only).
 
 ---
 
@@ -984,7 +998,7 @@ The Vue ward dashboard (`vigilcare-dashboard/`) implements this lifecycle throug
 | Trend alert | No | PostgreSQL → Kafka → trend analyzer (reads Redis history) → PostgreSQL | `GET .../alerts` |
 | Medication annotation on alert | No — applied at alert creation | Kafka consumer reads recent meds from PostgreSQL | `GET .../alerts` → read `details` field |
 | Search index updated | No | PostgreSQL → Kafka → ES indexer → Elasticsearch | `GET /analytics/...` endpoints |
-| Data lake file written | No | PostgreSQL → Kafka → data lake writer → MinIO (Parquet) | MinIO bucket `vigilcare` |
+| Data lake file written | No | PostgreSQL → Kafka → data lake writer → MinIO (Parquet); offsets only committed for successful partition uploads | MinIO bucket `vigilcare` |
 | Clinician paged | No | Kafka → notification publisher → RabbitMQ paging queue | Alert `status` field |
 | Alert escalated | No | RabbitMQ paging queue → DLQ → escalation queue → PostgreSQL | Alert `status` = `escalated` |
 | Discharge summary PDF | No | Kafka → RabbitMQ discharge queue → worker → MinIO | MinIO `/discharge-summaries/{encounterId}/summary.pdf` |