Add deployment

docs/ops/phi-encryption-runbook.md

@@ -31,7 +31,14 @@ Configuration lives in `appsettings.json` under `PhiEncryption` and `DataProtect
 | Environment | Data Protection key ring | Search HMAC key (`SearchTokenKey`) |
 |---|---|---|
 | Development | `./data-protection-keys/` on disk (`DataProtection:KeyPath`) | `appsettings.json` (dev placeholder only) |
-| Production | Azure Key Vault XML blob or AWS KMS-backed store | Key Vault / Secrets Manager secret — **not** appsettings |
+| Production | Docker named volume `vigilcare_dp_keys` mounted at `/app/data-protection-keys` (`docker-compose.prod.yml` uses `name: vigilcare`) | Host `.env` → `PHI_SEARCH_TOKEN_KEY` / secrets manager — **not** appsettings |
+
+**Production custody rules (Phase 36 Step 10):**
+
+1. The `dp_keys` volume is mandatory. Losing it makes every encrypted patient column permanently unreadable — a database backup alone cannot recover PHI.
+2. Back up the keyring **separately** from PostgreSQL, daily, and replicate off-host.
+3. Single API instance only with filesystem keys. Scaling past one replica requires moving to `PersistKeysToDbContext<AppDbContext>()` (or a shared store) so all instances share the ring.
+4. Keys on the volume are not encrypted at rest; ensure the host filesystem / volume store is encrypted, or add `ProtectKeysWithCertificate()` later.
 
 ---
 
@@ -122,6 +129,77 @@ Prometheus metric: `phi_access_logs_total{access_type="VIEW|LIST|SEARCH|CREATE|U
 
 Rotating one key without the other does not require touching the other, but both rotations need a full patient re-save.
 
+**`SearchTokenKey` is effectively permanent in production.** Rotating it invalidates every stored `name_search_token` and breaks patient name search until a full `encrypt-phi` re-tokenization pass completes. Prefer treating it like a root secret: generate once, store in the secrets system, never rotate casually.
+
+---
+
+## Production keyring backup
+
+On the deploy host (after the API has started at least once and written keys into the volume):
+
+```bash
+# From a checkout that includes scripts/, or copy the script to /opt/vigilcare/scripts/
+./scripts/backup-dp-keys.sh
+```
+
+- Volume: `vigilcare_dp_keys` (from compose `name: vigilcare` + volume `dp_keys`)
+- Default destination: `/var/backups/vigilcare/dp-keys/dp-keys-<UTC>.tar.gz` (mode `0600`)
+- Retention: 30 days inside that directory
+- Override destination for off-host sync: `BACKUP_DIR=/mnt/offsite/vigilcare/dp-keys ./scripts/backup-dp-keys.sh`
+
+Suggested cron (daily 02:15 UTC):
+
+```
+15 2 * * * /opt/vigilcare/scripts/backup-dp-keys.sh >> /var/log/vigilcare-dp-backup.log 2>&1
+```
+
+Replicate `/var/backups/vigilcare/dp-keys/` (or `BACKUP_DIR`) to a second site. A backup that only lives on the same disk as the volume is not a disaster-recovery backup.
+
+---
+
+## Keyring restore
+
+Use when the volume is empty/corrupt, the host was rebuilt, or PHI decrypt fails after a redeploy.
+
+```bash
+./scripts/restore-dp-keys.sh /var/backups/vigilcare/dp-keys/dp-keys-YYYYMMDDThhmmssZ.tar.gz
+```
+
+The script stops the `api` service (if compose is present), extracts the archive into `vigilcare_dp_keys`, then prints the bring-up steps:
+
+```bash
+docker compose -f /opt/vigilcare/docker-compose.prod.yml --env-file /opt/vigilcare/.env up -d api
+curl -fsS http://localhost:5270/health/ready
+# Then fetch a known patient and confirm firstName/lastName decrypt to plaintext.
+```
+
+**Do not** invent a new empty keyring and restart the API against an existing encrypted database — that permanently orphans ciphertext.
+
+---
+
+## Test-restore cadence
+
+A backup that has never been restored is not a backup. Cadence:
+
+| Cadence | Action |
+|---|---|
+| After first production deploy | Take an immediate backup; restore into a **throwaway** Docker volume on a non-prod host (or a second named volume); start an API against a DB snapshot and decrypt one patient |
+| Quarterly | Repeat the throwaway restore drill; record date, archive name, operator, and pass/fail in the ops log |
+| Before any host migration / disk replacement | Fresh backup, then restore drill on the target host before cutting traffic |
+
+Throwaway restore sketch (does not touch production volume):
+
+```bash
+docker volume create vigilcare_dp_keys_drill
+docker run --rm \
+  -v vigilcare_dp_keys_drill:/keys \
+  -v /var/backups/vigilcare/dp-keys:/backup:ro \
+  alpine tar xzf /backup/dp-keys-<stamp>.tar.gz -C /keys
+# Point a staging API at DataProtection__KeyPath=/app/data-protection-keys
+# with -v vigilcare_dp_keys_drill:/app/data-protection-keys and a DB clone.
+docker volume rm vigilcare_dp_keys_drill
+```
+
 ---
 
 ## PHI access log retention (HIPAA)
@@ -156,10 +234,11 @@ These paths are **not** covered by column encryption in PostgreSQL:
 ### `CryptographicException` / cannot decrypt patient
 
 - Key ring missing or wrong `DataProtection:KeyPath`
-- App deployed to new host without copying `data-protection-keys/`
+- App deployed to new host without restoring `vigilcare_dp_keys` (or copying `data-protection-keys/`)
+- Production compose ran without the `dp_keys` volume — container regenerated a new empty ring
 - `ProtectorPurpose` changed without re-running `encrypt-phi`
 
-**Fix:** Restore key ring from backup. Do not delete old keys until all data is re-encrypted.
+**Fix:** Restore key ring from backup (`scripts/restore-dp-keys.sh`). Do not delete old keys until all data is re-encrypted.
 
 ### Name search returns no results
 
@@ -194,5 +273,9 @@ Migration `WidenPhiEncryptedColumns` widens `first_name`, `last_name`, and emerg
 | `VigilCareClinicalAPI/Commands/EncryptPhiCommand.cs` | Bulk re-save CLI |
 | `VigilCareClinicalAPI/BackgroundServices/PatientPhiMigrationService.cs` | Startup token backfill |
 | `scripts/encrypt-existing-patient-phi.sh` | Wrapper for encrypt CLI |
+| `scripts/backup-dp-keys.sh` | Daily backup of `vigilcare_dp_keys` |
+| `scripts/restore-dp-keys.sh` | Restore keyring archive into the Docker volume |
 | `scripts/run-phase32-verification.sh` | End-to-end verification |
+| `docker-compose.prod.yml` | Mounts `dp_keys` → `/app/data-protection-keys` |
 | `docs/plans/phase-32-plan.md` | Implementation plan and design rationale |
+| `docs/plans/vigilcare-clinical-deployment-plan.md` | Phase 36 deployment (Step 10) |

docs/ops/production-observability.md

@@ -0,0 +1,60 @@
+# Production observability wiring (Phase 36 Step 9)
+#
+# Local-dev Prometheus/Grafana/Seq remain in docker-compose.yml + infra/.
+# Production uses *existing external* instances — do not start those services
+# from docker-compose.prod.yml.
+
+## Prometheus
+
+1. Keep [infra/prometheus/prometheus.yml](../infra/prometheus/prometheus.yml) as the
+   local-dev scrape of `host.docker.internal:5270`. Do not add production jobs there.
+2. On the external Prometheus host, merge
+   [infra/prometheus/production/scrape-vigilcare.yml](../infra/prometheus/production/scrape-vigilcare.yml)
+   into `scrape_configs`.
+3. Load
+   [infra/prometheus/production/alert-rules.yml](../infra/prometheus/production/alert-rules.yml)
+   via `rule_files`.
+4. Optionally enable the commented blackbox probe for `/health/ready` (same scrape file).
+
+### `/metrics` access
+
+The Clinical API uses a global `FallbackPolicy` that requires an authenticated user.
+`/metrics` is explicitly `.AllowAnonymous()` so Prometheus can scrape without a JWT.
+**Restrict at the network layer** — firewall so only the Prometheus host (and
+operators) can reach `API_PORT` /metrics. Metric labels include department and
+gateway identifiers.
+
+## Grafana
+
+Prefer file provisioning over UI import (UI imports are lost on Grafana redeploy):
+
+1. Copy datasources:
+   [infra/grafana/production/datasources.yml](../infra/grafana/production/datasources.yml)
+   — keep `uid: prometheus` so existing dashboards keep working.
+2. Copy dashboards from [infra/grafana/dashboards/](../infra/grafana/dashboards/) to the
+   path referenced by
+   [infra/grafana/production/dashboards-provider.yml](../infra/grafana/production/dashboards-provider.yml).
+3. Confirm panels render against the external Prometheus (job labels
+   `vigilcare_api_prod` / `environment=production`).
+
+Minimum production alerts (also in Prometheus rules): API down, outbox backlog,
+Kafka consumer lag, ready probe failing, gateway offline, unacked CRITICAL alerts.
+
+## Seq
+
+1. Point production at Seq via env (compose already sets these):
+   - `Seq__ServerUrl` / `Serilog__WriteTo__1__Args__serverUrl` → `SEQ_URL`
+   - `Serilog__WriteTo__1__Args__apiKey` → `SEQ_API_KEY` (ingest-only key)
+2. `appsettings.Production.json` sets
+   `Microsoft.EntityFrameworkCore.Database.Command` to **Warning** so SQL with
+   patient identifiers is not shipped to Seq.
+3. Serilog `Properties:Application` is set to `VigilCareClinicalAPI` /
+   `VigilCare.WardGateway` for filtering.
+4. In Seq, create a signal (or shared dashboard) approximately:
+
+   ```
+   Application = 'VigilCareClinicalAPI' and @Level in ['Error', 'Fatal']
+   ```
+
+   Optionally a second signal for the ward gateway with
+   `Application = 'VigilCare.WardGateway'`.