vigilcare-clinical/docs/ops/phi-encryption-runbook.md

# PHI Encryption & Access Logging — Operations Runbook

Phase 32 encrypts sensitive patient demographics at rest and records who accessed PHI. This runbook covers key management, migration, rotation, compliance, and troubleshooting.

---

## Architecture summary

| Layer | Mechanism |
|---|---|
| Encryption at rest | ASP.NET Data Protection API (AES-256-GCM) via EF Core value converters |
| Encrypted columns | `first_name`, `last_name`, `date_of_birth`, `allergies`, `emergency_contact_name`, `emergency_contact_phone` |
| Plaintext (by design) | `mrn` — exact-match lookup only |
| Name search | HMAC-SHA256 `name_search_token` index — search without decrypting all rows |
| Access audit | Append-only `phi_access_logs` table; written by `PhiAccessLogService` on patient Get/List/Register/Update |

Configuration lives in `appsettings.json` under `PhiEncryption` and `DataProtection`. Implementation: `PhiEncryptionService`, `PatientPhiConverterConfigurator`, `PatientService`.

---

## Secrets and key storage

**Never commit:**

- `data-protection-keys/` (Data Protection key ring)
- Production `PhiEncryption:SearchTokenKey`
- Any Key Vault / KMS credentials

`data-protection-keys/` is listed in `.gitignore`. Treat loss of this directory as **unrecoverable data loss** for encrypted PHI columns.

| Environment | Data Protection key ring | Search HMAC key (`SearchTokenKey`) |
|---|---|---|
| Development | `./data-protection-keys/` on disk (`DataProtection:KeyPath`) | `appsettings.json` (dev placeholder only) |
| Production | Docker named volume `vigilcare_dp_keys` mounted at `/app/data-protection-keys` (`docker-compose.prod.yml` uses `name: vigilcare`) | Host `.env` → `PHI_SEARCH_TOKEN_KEY` / secrets manager — **not** appsettings |

**Production custody rules (Phase 36 Step 10):**

1. The `dp_keys` volume is mandatory. Losing it makes every encrypted patient column permanently unreadable — a database backup alone cannot recover PHI.
2. Back up the keyring **separately** from PostgreSQL, daily, and replicate off-host.
3. Single API instance only with filesystem keys. Scaling past one replica requires moving to `PersistKeysToDbContext<AppDbContext>()` (or a shared store) so all instances share the ring.
4. Keys on the volume are not encrypted at rest; ensure the host filesystem / volume store is encrypted, or add `ProtectKeysWithCertificate()` later.

---

## Initial deployment / encrypting existing rows

New patients are encrypted automatically on save. Existing plaintext rows need a one-time re-save.

### Option A — CLI (recommended)

```bash
./scripts/encrypt-existing-patient-phi.sh
```

Or directly:

```bash
dotnet run --project VigilCareClinicalAPI -- encrypt-phi
```

Loads every patient through EF, applies value converters, and recomputes `name_search_token`.

### Option B — Startup hosted service

`PatientPhiMigrationService` runs on application start and backfills search tokens for rows that are not yet encrypted. It is registered in `Program.cs`. Disable or remove after the first successful production deploy to avoid redundant work on every restart.

---

## Verifying encryption

### Automated tests

```bash
dotnet test VigilCareClinicalAPI.Tests --filter "FullyQualifiedName~PhiEncryption"
```

### Live stack (API + Postgres running)

```bash
./scripts/run-phase32-verification.sh
```

### Manual DB check

Encrypted `first_name` values will **not** equal the patient's plaintext name:

```sql
SELECT id, mrn, left(first_name, 30) AS encrypted_prefix
FROM patients
LIMIT 5;
```

Prefix often starts with `CfDJ8` (Data Protection default).

### PHI access logs

```bash
# Admin token required (AuditRead permission)
curl -H "Authorization: Bearer $TOKEN" \
  "http://localhost:5270/api/v1/phi-access-logs?patientId=<uuid>"
```

Prometheus metric: `phi_access_logs_total{access_type="VIEW|LIST|SEARCH|CREATE|UPDATE"}`.

---

## Key rotation

**Always back up the current key ring before rotating.**

### Encryption keys (Data Protection)

1. Snapshot `data-protection-keys/` (or export Key Vault blob).
2. Deploy new key ring **alongside** the old one (Data Protection supports multiple keys; newest encrypts, all decrypt).
3. Change `PhiEncryption:ProtectorPurpose` to a new version string (e.g. `VigilCare.PatientPhi.v2`).
4. Re-run encrypt command to re-encrypt all patient rows with the new protector:
   ```bash
   dotnet run --project VigilCareClinicalAPI -- encrypt-phi
   ```
5. Verify API reads and raw DB ciphertext look correct.
6. Retire old keys only after confirming all rows decrypt successfully.

### Search HMAC key (`SearchTokenKey`)

1. Store new key in secrets manager.
2. Update `PhiEncryption:SearchTokenKey` in configuration.
3. Re-run `encrypt-phi` (recomputes all `name_search_token` values).
4. Confirm name search still works (`GET /api/v1/patients?q=First+Last`).

Rotating one key without the other does not require touching the other, but both rotations need a full patient re-save.

**`SearchTokenKey` is effectively permanent in production.** Rotating it invalidates every stored `name_search_token` and breaks patient name search until a full `encrypt-phi` re-tokenization pass completes. Prefer treating it like a root secret: generate once, store in the secrets system, never rotate casually.

---

## Production keyring backup

On the deploy host (after the API has started at least once and written keys into the volume):

```bash
# From a checkout that includes scripts/, or copy the script to /opt/vigilcare/scripts/
./scripts/backup-dp-keys.sh
```

- Volume: `vigilcare_dp_keys` (from compose `name: vigilcare` + volume `dp_keys`)
- Default destination: `/var/backups/vigilcare/dp-keys/dp-keys-<UTC>.tar.gz` (mode `0600`)
- Retention: 30 days inside that directory
- Override destination for off-host sync: `BACKUP_DIR=/mnt/offsite/vigilcare/dp-keys ./scripts/backup-dp-keys.sh`

Suggested cron (daily 02:15 UTC):

```
15 2 * * * /opt/vigilcare/scripts/backup-dp-keys.sh >> /var/log/vigilcare-dp-backup.log 2>&1
```

Replicate `/var/backups/vigilcare/dp-keys/` (or `BACKUP_DIR`) to a second site. A backup that only lives on the same disk as the volume is not a disaster-recovery backup.

---

## Keyring restore

Use when the volume is empty/corrupt, the host was rebuilt, or PHI decrypt fails after a redeploy.

```bash
./scripts/restore-dp-keys.sh /var/backups/vigilcare/dp-keys/dp-keys-YYYYMMDDThhmmssZ.tar.gz
```

The script stops the `api` service (if compose is present), extracts the archive into `vigilcare_dp_keys`, then prints the bring-up steps:

```bash
docker compose -f /opt/vigilcare/docker-compose.prod.yml --env-file /opt/vigilcare/.env up -d api
curl -fsS http://localhost:5270/health/ready
# Then fetch a known patient and confirm firstName/lastName decrypt to plaintext.
```

**Do not** invent a new empty keyring and restart the API against an existing encrypted database — that permanently orphans ciphertext.

---

## Test-restore cadence

A backup that has never been restored is not a backup. Cadence:

| Cadence | Action |
|---|---|
| After first production deploy | Take an immediate backup; restore into a **throwaway** Docker volume on a non-prod host (or a second named volume); start an API against a DB snapshot and decrypt one patient |
| Quarterly | Repeat the throwaway restore drill; record date, archive name, operator, and pass/fail in the ops log |
| Before any host migration / disk replacement | Fresh backup, then restore drill on the target host before cutting traffic |

Throwaway restore sketch (does not touch production volume):

```bash
docker volume create vigilcare_dp_keys_drill
docker run --rm \
  -v vigilcare_dp_keys_drill:/keys \
  -v /var/backups/vigilcare/dp-keys:/backup:ro \
  alpine tar xzf /backup/dp-keys-<stamp>.tar.gz -C /keys
# Point a staging API at DataProtection__KeyPath=/app/data-protection-keys
# with -v vigilcare_dp_keys_drill:/app/data-protection-keys and a DB clone.
docker volume rm vigilcare_dp_keys_drill
```

---

## PHI access log retention (HIPAA)

`phi_access_logs` records who accessed patient demographics, when, from which path, and (hashed) search terms. HIPAA requires audit controls; retention guidance is **minimum 6 years**.

This phase creates the table and query API (`GET /api/v1/phi-access-logs`). Automated retention/archival policy is **out of scope** — plan for Phase 33+ (partitioning, cold storage, or purge job with legal review).

Query endpoints:

| Endpoint | Permission |
|---|---|
| `GET /api/v1/phi-access-logs` | `AuditRead` |
| `GET /api/v1/phi-access-logs/patients/{id}` | `PatientsRead` |

---

## Cross-system PHI (out of scope)

These paths are **not** covered by column encryption in PostgreSQL:

| System | Risk | Action |
|---|---|---|
| Elasticsearch patient documents | May index decrypted names at index time | Exclude PHI fields or accept decrypted-at-index policy |
| Kafka / data lake Parquet events | `patientName` in outbox payloads (e.g. encounter events) | Review lake schemas; redact or tokenize in a future phase |
| Application logs | Serilog must not log PHI fields | Audit log configuration |

---

## Troubleshooting

### `CryptographicException` / cannot decrypt patient

- Key ring missing or wrong `DataProtection:KeyPath`
- App deployed to new host without restoring `vigilcare_dp_keys` (or copying `data-protection-keys/`)
- Production compose ran without the `dp_keys` volume — container regenerated a new empty ring
- `ProtectorPurpose` changed without re-running `encrypt-phi`

**Fix:** Restore key ring from backup (`scripts/restore-dp-keys.sh`). Do not delete old keys until all data is re-encrypted.

### Name search returns no results

- `name_search_token` is null on older rows → run `encrypt-phi`
- `SearchTokenKey` changed without token recompute
- Query format: two-term search uses `First Last` (space-separated); single term matches first or last name only

### PHI access logs empty

- Caller not authenticated (`PhiAccessLogService` skips unauthenticated requests)
- `PhiEncryption:LogListAccess` is `false` (list/search aggregate logs suppressed)
- Integration/FHIR paths must still authenticate (Phase 31 Integration role)

### Column length errors on encrypt

Migration `WidenPhiEncryptedColumns` widens `first_name`, `last_name`, and emergency contact columns to `text`. Ciphertext is longer than plaintext. If new columns are added to encryption, ensure DB column types accommodate protected payload size.

### Tests fail with 500 on patient register

- Confirm migrations applied (`AddPatientNameSearchToken`, `AddPhiAccessLogs`, `WidenPhiEncryptedColumns`)
- Confirm `PhiEncryption:SearchTokenKey` is set in configuration
- Confirm `data-protection-keys/` is writable in the API working directory

---

## Related files

| File | Purpose |
|---|---|
| `VigilCareClinicalAPI/Services/PhiEncryptionService.cs` | Encrypt/decrypt + HMAC tokens |
| `VigilCareClinicalAPI/Services/PhiAccessLogService.cs` | Access audit writes |
| `VigilCareClinicalAPI/Commands/EncryptPhiCommand.cs` | Bulk re-save CLI |
| `VigilCareClinicalAPI/BackgroundServices/PatientPhiMigrationService.cs` | Startup token backfill |
| `scripts/encrypt-existing-patient-phi.sh` | Wrapper for encrypt CLI |
| `scripts/backup-dp-keys.sh` | Daily backup of `vigilcare_dp_keys` |
| `scripts/restore-dp-keys.sh` | Restore keyring archive into the Docker volume |
| `scripts/run-phase32-verification.sh` | End-to-end verification |
| `docker-compose.prod.yml` | Mounts `dp_keys` → `/app/data-protection-keys` |
| `docs/plans/phase-32-plan.md` | Implementation plan and design rationale |
| `docs/plans/vigilcare-clinical-deployment-plan.md` | Phase 36 deployment (Step 10) |