Incident (2026-06-16, server-room power outage)
A power outage rebooted the lab. krg-prod came back but Grafana, Authentik, Guacamole, Temporal, and Vaultwarden were all down/degraded. Single root cause:
- OpenBao (
krg-vault) seals on every reboot (Shamir, 5 shares, threshold 3). After the outage it came up Sealed: true, Unseal Progress 0/3.
openbao-agent.service on krg-prod sat in activating for 40+ min, looping on approle/login → 503 "Vault is sealed" (backoff ~5 min), so it rendered zero secrets to /run/krg/....- The entire
krg-prod compose project is requires=openbao-agent.service (fail-closed), so krg-prod.service never started this boot (queued, inactive (dead)). Grafana exited and stayed down; Authentik/Guacamole/Temporal-UI crash-looped on missing /run secrets and on each other.
The lab stayed down until an operator manually retrieved the unseal keys and unsealed krg-vault.
The real finding: the unseal keys are not distributed safely
The 3-of-5 Shamir unseal keys were stored inside Vaultwarden — which runs behind the very stack OpenBao gates. That is a circular root-of-trust: you can't unseal the vault without the keys, and the keys live behind the thing the vault unlocks.
We recovered only by luck: Vaultwarden's master-password login path is independent of both OpenBao and Authentik (its data is on-disk SQLite; SSO_ONLY is unset). Its SSO was 404ing, but master-password still worked. Change one thing — Vaultwarden needing a rendered /run secret to boot, or sharing the stack's Postgres — and the lab is bricked.
Proposed work
- Distribute the Shamir shares correctly. One share per admin (chris, dzuberi, shperry, …), held out-of-band in stores that do not depend on
krg-vault or the krg-prod stack (personal password manager not behind this SSO, age/GPG-encrypted blob, hardware token, sealed paper). Invariant: ≥ threshold (3) shares must be retrievable with krg-vault AND krg-prod both down. No single store may hold ≥3.
- Add auto-unseal so a power-cycle self-heals without a human: a
seal stanza in the krg-vault Nix config (transit auto-unseal against a second bao, cloud KMS, or a TPM-backed key). The auto-unseal root of trust must live outside this blast radius. Keep the distributed Shamir shares as recovery keys for break-glass.
- Document custody (who holds which share, how to convene 3) in the recovery runbook (see companion issue).
Acceptance criteria
Related: companion issues for the recovery runbook and temporal-ui startup resilience; #184 (secret rendering, already done) is the consumer side of this. This is the OpenBao SPOF analogue of the krg-ldap "second DC" item.
Incident (2026-06-16, server-room power outage)
A power outage rebooted the lab.
krg-prodcame back but Grafana, Authentik, Guacamole, Temporal, and Vaultwarden were all down/degraded. Single root cause:krg-vault) seals on every reboot (Shamir, 5 shares, threshold 3). After the outage it came upSealed: true,Unseal Progress 0/3.openbao-agent.serviceonkrg-prodsat inactivatingfor 40+ min, looping onapprole/login → 503 "Vault is sealed"(backoff ~5 min), so it rendered zero secrets to/run/krg/....krg-prodcompose project isrequires=openbao-agent.service(fail-closed), sokrg-prod.servicenever started this boot (queued,inactive (dead)). Grafana exited and stayed down; Authentik/Guacamole/Temporal-UI crash-looped on missing/runsecrets and on each other.The lab stayed down until an operator manually retrieved the unseal keys and unsealed
krg-vault.The real finding: the unseal keys are not distributed safely
The 3-of-5 Shamir unseal keys were stored inside Vaultwarden — which runs behind the very stack OpenBao gates. That is a circular root-of-trust: you can't unseal the vault without the keys, and the keys live behind the thing the vault unlocks.
We recovered only by luck: Vaultwarden's master-password login path is independent of both OpenBao and Authentik (its data is on-disk SQLite;
SSO_ONLYis unset). Its SSO was 404ing, but master-password still worked. Change one thing — Vaultwarden needing a rendered/runsecret to boot, or sharing the stack's Postgres — and the lab is bricked.Proposed work
krg-vaultor thekrg-prodstack (personal password manager not behind this SSO, age/GPG-encrypted blob, hardware token, sealed paper). Invariant: ≥ threshold (3) shares must be retrievable withkrg-vaultANDkrg-prodboth down. No single store may hold ≥3.sealstanza in thekrg-vaultNix config (transit auto-unseal against a second bao, cloud KMS, or a TPM-backed key). The auto-unseal root of trust must live outside this blast radius. Keep the distributed Shamir shares as recovery keys for break-glass.Acceptance criteria
operator rekeyif shares were exposed).krg-vault, validated by a reboot that comes back unsealed with no human.Related: companion issues for the recovery runbook and
temporal-uistartup resilience; #184 (secret rendering, already done) is the consumer side of this. This is the OpenBao SPOF analogue of the krg-ldap "second DC" item.