vigilcare-clinical/docs/ops/phi-encryption-runbook.md

PHI Encryption & Access Logging — Operations Runbook

Phase 32 encrypts sensitive patient demographics at rest and records who accessed PHI. This runbook covers key management, migration, rotation, compliance, and troubleshooting.

---

Architecture summary

Layer	Mechanism
Encryption at rest	ASP.NET Data Protection API (AES-256-GCM) via EF Core value converters
Encrypted columns	first_name, last_name, date_of_birth, allergies, emergency_contact_name, emergency_contact_phone
Plaintext (by design)	mrn — exact-match lookup only
Name search	HMAC-SHA256 name_search_token index — search without decrypting all rows
Access audit	Append-only phi_access_logs table; written by PhiAccessLogService on patient Get/List/Register/Update

Configuration lives in appsettings.json under PhiEncryption and DataProtection. Implementation: PhiEncryptionService, PatientPhiConverterConfigurator, PatientService.

---

Secrets and key storage

Never commit:

• data-protection-keys/ (Data Protection key ring)
• Production PhiEncryption:SearchTokenKey
• Any Key Vault / KMS credentials

data-protection-keys/ is listed in .gitignore. Treat loss of this directory as unrecoverable data loss for encrypted PHI columns.

Environment	Data Protection key ring	Search HMAC key (SearchTokenKey)
Development	./data-protection-keys/ on disk (DataProtection:KeyPath)	appsettings.json (dev placeholder only)
Production	Docker named volume vigilcare_dp_keys mounted at /app/data-protection-keys (docker-compose.prod.yml uses name: vigilcare)	Host .env → PHI_SEARCH_TOKEN_KEY / secrets manager — not appsettings

Production custody rules (Phase 36 Step 10):

1. The dp_keys volume is mandatory. Losing it makes every encrypted patient column permanently unreadable — a database backup alone cannot recover PHI.
2. Back up the keyring separately from PostgreSQL, daily, and replicate off-host.
3. Single API instance only with filesystem keys. Scaling past one replica requires moving to PersistKeysToDbContext<AppDbContext>() (or a shared store) so all instances share the ring.
4. Keys on the volume are not encrypted at rest; ensure the host filesystem / volume store is encrypted, or add ProtectKeysWithCertificate() later.

---

Initial deployment / encrypting existing rows

New patients are encrypted automatically on save. Existing plaintext rows need a one-time re-save.

Option A — CLI (recommended)

./scripts/encrypt-existing-patient-phi.sh

Or directly:

dotnet run --project VigilCareClinicalAPI -- encrypt-phi

Loads every patient through EF, applies value converters, and recomputes name_search_token.

Option B — Startup hosted service

PatientPhiMigrationService runs on application start and backfills search tokens for rows that are not yet encrypted. It is registered in Program.cs. Disable or remove after the first successful production deploy to avoid redundant work on every restart.

---

Verifying encryption

Automated tests

dotnet test VigilCareClinicalAPI.Tests --filter "FullyQualifiedName~PhiEncryption"

Live stack (API + Postgres running)

./scripts/run-phase32-verification.sh

Manual DB check

Encrypted first_name values will not equal the patient's plaintext name:

SELECT id, mrn, left(first_name, 30) AS encrypted_prefix
FROM patients
LIMIT 5;

Prefix often starts with CfDJ8 (Data Protection default).

PHI access logs

# Admin token required (AuditRead permission)
curl -H "Authorization: Bearer $TOKEN" \
  "http://localhost:5270/api/v1/phi-access-logs?patientId=<uuid>"

Prometheus metric: phi_access_logs_total{access_type="VIEW|LIST|SEARCH|CREATE|UPDATE"}.

---

Key rotation

Always back up the current key ring before rotating.

Encryption keys (Data Protection)

1. Snapshot data-protection-keys/ (or export Key Vault blob).
2. Deploy new key ring alongside the old one (Data Protection supports multiple keys; newest encrypts, all decrypt).
3. Change PhiEncryption:ProtectorPurpose to a new version string (e.g. VigilCare.PatientPhi.v2).
4. Re-run encrypt command to re-encrypt all patient rows with the new protector:

   dotnet run --project VigilCareClinicalAPI -- encrypt-phi
   

5. Verify API reads and raw DB ciphertext look correct.
6. Retire old keys only after confirming all rows decrypt successfully.

Search HMAC key (SearchTokenKey)

1. Store new key in secrets manager.
2. Update PhiEncryption:SearchTokenKey in configuration.
3. Re-run encrypt-phi (recomputes all name_search_token values).
4. Confirm name search still works (GET /api/v1/patients?q=First+Last).

Rotating one key without the other does not require touching the other, but both rotations need a full patient re-save.

SearchTokenKey is effectively permanent in production. Rotating it invalidates every stored name_search_token and breaks patient name search until a full encrypt-phi re-tokenization pass completes. Prefer treating it like a root secret: generate once, store in the secrets system, never rotate casually.

---

Production keyring backup

On the deploy host (after the API has started at least once and written keys into the volume):

# From a checkout that includes scripts/, or copy the script to /opt/vigilcare/scripts/
./scripts/backup-dp-keys.sh

• Volume: vigilcare_dp_keys (from compose name: vigilcare + volume dp_keys)
• Default destination: /var/backups/vigilcare/dp-keys/dp-keys-<UTC>.tar.gz (mode 0600)
• Retention: 30 days inside that directory
• Override destination for off-host sync: BACKUP_DIR=/mnt/offsite/vigilcare/dp-keys ./scripts/backup-dp-keys.sh

Suggested cron (daily 02:15 UTC):

15 2 * * * /opt/vigilcare/scripts/backup-dp-keys.sh >> /var/log/vigilcare-dp-backup.log 2>&1

Replicate /var/backups/vigilcare/dp-keys/ (or BACKUP_DIR) to a second site. A backup that only lives on the same disk as the volume is not a disaster-recovery backup.

---

Keyring restore

Use when the volume is empty/corrupt, the host was rebuilt, or PHI decrypt fails after a redeploy.

./scripts/restore-dp-keys.sh /var/backups/vigilcare/dp-keys/dp-keys-YYYYMMDDThhmmssZ.tar.gz

The script stops the api service (if compose is present), extracts the archive into vigilcare_dp_keys, then prints the bring-up steps:

docker compose -f /opt/vigilcare/docker-compose.prod.yml --env-file /opt/vigilcare/.env up -d api
curl -fsS http://localhost:5270/health/ready
# Then fetch a known patient and confirm firstName/lastName decrypt to plaintext.

Do not invent a new empty keyring and restart the API against an existing encrypted database — that permanently orphans ciphertext.

---

Test-restore cadence

A backup that has never been restored is not a backup. Cadence:

Cadence	Action
After first production deploy	Take an immediate backup; restore into a throwaway Docker volume on a non-prod host (or a second named volume); start an API against a DB snapshot and decrypt one patient
Quarterly	Repeat the throwaway restore drill; record date, archive name, operator, and pass/fail in the ops log
Before any host migration / disk replacement	Fresh backup, then restore drill on the target host before cutting traffic

Throwaway restore sketch (does not touch production volume):

docker volume create vigilcare_dp_keys_drill
docker run --rm \
  -v vigilcare_dp_keys_drill:/keys \
  -v /var/backups/vigilcare/dp-keys:/backup:ro \
  alpine tar xzf /backup/dp-keys-<stamp>.tar.gz -C /keys
# Point a staging API at DataProtection__KeyPath=/app/data-protection-keys
# with -v vigilcare_dp_keys_drill:/app/data-protection-keys and a DB clone.
docker volume rm vigilcare_dp_keys_drill

---

PHI access log retention (HIPAA)

phi_access_logs records who accessed patient demographics, when, from which path, and (hashed) search terms. HIPAA requires audit controls; retention guidance is minimum 6 years.

This phase creates the table and query API (GET /api/v1/phi-access-logs). Automated retention/archival policy is out of scope — plan for Phase 33+ (partitioning, cold storage, or purge job with legal review).

Query endpoints:

Endpoint	Permission
GET /api/v1/phi-access-logs	AuditRead
GET /api/v1/phi-access-logs/patients/{id}	PatientsRead

---

Cross-system PHI (out of scope)

These paths are not covered by column encryption in PostgreSQL:

System	Risk	Action
Elasticsearch patient documents	May index decrypted names at index time	Exclude PHI fields or accept decrypted-at-index policy
Kafka / data lake Parquet events	patientName in outbox payloads (e.g. encounter events)	Review lake schemas; redact or tokenize in a future phase
Application logs	Serilog must not log PHI fields	Audit log configuration

---

Troubleshooting

CryptographicException / cannot decrypt patient

• Key ring missing or wrong DataProtection:KeyPath
• App deployed to new host without restoring vigilcare_dp_keys (or copying data-protection-keys/)
• Production compose ran without the dp_keys volume — container regenerated a new empty ring
• ProtectorPurpose changed without re-running encrypt-phi

Fix: Restore key ring from backup (scripts/restore-dp-keys.sh). Do not delete old keys until all data is re-encrypted.

Name search returns no results

• name_search_token is null on older rows → run encrypt-phi
• SearchTokenKey changed without token recompute
• Query format: two-term search uses First Last (space-separated); single term matches first or last name only

PHI access logs empty

• Caller not authenticated (PhiAccessLogService skips unauthenticated requests)
• PhiEncryption:LogListAccess is false (list/search aggregate logs suppressed)
• Integration/FHIR paths must still authenticate (Phase 31 Integration role)

Column length errors on encrypt

Migration WidenPhiEncryptedColumns widens first_name, last_name, and emergency contact columns to text. Ciphertext is longer than plaintext. If new columns are added to encryption, ensure DB column types accommodate protected payload size.

Tests fail with 500 on patient register

• Confirm migrations applied (AddPatientNameSearchToken, AddPhiAccessLogs, WidenPhiEncryptedColumns)
• Confirm PhiEncryption:SearchTokenKey is set in configuration
• Confirm data-protection-keys/ is writable in the API working directory

---

Related files

File	Purpose
VigilCareClinicalAPI/Services/PhiEncryptionService.cs	Encrypt/decrypt + HMAC tokens
VigilCareClinicalAPI/Services/PhiAccessLogService.cs	Access audit writes
VigilCareClinicalAPI/Commands/EncryptPhiCommand.cs	Bulk re-save CLI
VigilCareClinicalAPI/BackgroundServices/PatientPhiMigrationService.cs	Startup token backfill
scripts/encrypt-existing-patient-phi.sh	Wrapper for encrypt CLI
scripts/backup-dp-keys.sh	Daily backup of vigilcare_dp_keys
scripts/restore-dp-keys.sh	Restore keyring archive into the Docker volume
scripts/run-phase32-verification.sh	End-to-end verification
docker-compose.prod.yml	Mounts dp_keys → /app/data-protection-keys
docs/plans/phase-32-plan.md	Implementation plan and design rationale
docs/plans/vigilcare-clinical-deployment-plan.md	Phase 36 deployment (Step 10)