kms: add client certificate (mTLS) support to the vault KMS type

[sergeykuperman]

Feature Request

Summary

The vault KMS type supports server certificate verification via vaultCAFromSecret
but has no way to present a client certificate. We are requesting support for
vaultClientCertFromSecret and vaultClientCertKeyFromSecret config options,
using the same K8s Secret lookup mechanism already implemented in the vaulttokens
type.

A reference implementation is available in PR #6375.

---

Background and motivation

We run Rook-Ceph for persistent storage with LUKS encryption. We are upgrading
from Rook v1.16.6 to v1.19.6. Starting in versions after 1.16, the CSI node
plugin DaemonSet enforces hostNetwork: true (required to avoid disrupting existing
mounts on pod restart). This is a hard constraint we cannot change.

Our cluster runs Istio with PeerAuthentication: STRICT enforced globally as a
corporate security requirement — all in-cluster service-to-service communication
must use mTLS. No exceptions are permitted.

With hostNetwork: true, Istio cannot inject a sidecar into the CSI node plugin pod.
Without a sidecar, the node plugin's outbound connections bypass Istio's transparent
mTLS entirely. This means the node plugin can no longer reach our encryption key
service in a compliant way.

---

Our setup

We use a custom vault-compatible API to store LUKS volume encryption keys in our
corporate credential store. It exposes a Vault-compatible KV API
(/v1/auth/kubernetes/login, GET/PUT/DELETE /v1/<mount>/<path>/<handle>) backed
by our corporate secret management infrastructure rather than HashiCorp Vault itself.

The key-manager sits behind Istio's service mesh with PeerAuthentication: STRICT
and an AuthorizationPolicy that permits access only from the CSI plugin service
accounts. Prior to the upgrade, the CSI node plugin ran with hostNetwork: false
(possible in Rook v1.16.6) and received an Istio sidecar automatically, so
transparent mTLS was handled without any application-level configuration.

---

KMS configuration before the upgrade (working, Rook v1.16.6)

{
  "appstudio-cmk": {
    "encryptionKMSType": "vault",
    "vaultAddress": "http://key-manager.webide-system.svc.cluster.local:80",
    "vaultAuthPath": "/v1/auth/kubernetes/login",
    "vaultBackend": "kv",
    "vaultPassphraseRoot": "/v1/ceph-csi-overlay",
    "vaultPassphrasePath": "ceph-csi/",
    "vaultRole": "ceph-csi"
  }
}

With hostNetwork: false and Istio sidecar injection, this worked without any TLS
configuration in the KMS config — Istio handled mTLS transparently at the network layer.

---

The problem after the upgrade (Rook v1.19.6)

With hostNetwork: true enforced:

1. Istio sidecar injection is not possible on the node plugin pods
2. The key-manager's PeerAuthentication: STRICT rejects plain HTTP connections
3. The corporate security policy does not allow relaxing mTLS to PERMISSIVE

The only viable path is application-level mTLS: the key-manager listens on a
dedicated TLS port (bypassed from Istio's iptables interception), and the CSI plugin
connects using a client certificate issued by the cluster's Istio CA.

---

Why the vault type specifically

We need to keep using the vault KMS type (Kubernetes SA token authentication via
/v1/auth/kubernetes/login) rather than switching to vaulttokens. The vaulttokens
type requires a static Vault token in a Kubernetes Secret in each tenant namespace
(the PVC owner's namespace). Our cluster creates thousands of workspace namespaces
dynamically; managing a token Secret in every namespace is operationally impractical.

The vault type uses the CSI plugin's Kubernetes service account JWT for
authentication — no per-namespace Secrets required.

---

What already works in vaulttokens

The vaulttokens and vaulttenantsa types already support mTLS client certificates
via vaultClientCertFromSecret and vaultClientCertKeyFromSecret. These config
values are Kubernetes Secret names; the implementation calls k8s.GetSecret() to
read the PEM-encoded certificate and key from named fields ("cert" and "key")
within those Secrets, writes them to temp files, and passes them to the Vault SDK via
api.EnvVaultClientCert / api.EnvVaultClientKey.

The same mechanism already handles vaultCAFromSecret in the vaulttokens type.

---

Requested change

Extend vaultConnection.initCertificates() in internal/kms/vault.go to support
the same three config options as vaulttokens, using k8s.GetSecret() for all three:

Config key	Type	Description
vaultCAFromSecret	K8s Secret name	Already present in vault; reads field "cert"
vaultClientCertFromSecret	K8s Secret name	New: reads field "cert" from named Secret
vaultClientCertKeyFromSecret	K8s Secret name	New: reads field "key" from named Secret

Secret lookup namespace: the CSI pod namespace (POD_NAMESPACE env var), consistent
with the fallback behaviour in vaulttokens.

Note on vaultCAFromSecret in vault today: the current vault implementation
reads vaultCAFromSecret as a key name inside args.Secrets (the node-stage secret
map from the StorageClass) rather than as a K8s Secret name. For consistency with
vaulttokens and to allow all three cert options to be configured uniformly, the
implementation should also migrate vaultCAFromSecret to use k8s.GetSecret(). This
is a behaviour change for existing users of vaultCAFromSecret in the vault type
and should be clearly documented in the changelog.

---

Desired KMS configuration after the change

{
  "appstudio-cmk": {
    "encryptionKMSType": "vault",
    "vaultAddress": "https://key-manager.webide-system.svc.cluster.local:8443",
    "vaultAuthPath": "/v1/auth/kubernetes/login",
    "vaultBackend": "kv",
    "vaultPassphraseRoot": "/v1/ceph-csi-overlay",
    "vaultPassphrasePath": "ceph-csi/",
    "vaultRole": "ceph-csi",
    "vaultCAFromSecret": "istio-ca-root-cert-secret",
    "vaultClientCertFromSecret": "csi-rbdplugin-vault-client-tls",
    "vaultClientCertKeyFromSecret": "csi-rbdplugin-vault-client-tls"
  }
}

Where:

• istio-ca-root-cert-secret is a K8s Secret in rook-ceph containing the Istio
  cluster CA root certificate under field "cert"
• csi-rbdplugin-vault-client-tls is a K8s Secret in rook-ceph containing the
  client certificate under field "cert" and private key under field "key",
  issued by the Istio CA with SPIFFE URI SAN
  spiffe://cluster.local/ns/rook-ceph/sa/rook-csi-rbd-plugin-sa

---

Alternatives considered

1. vaulttokens with per-namespace token Secrets — rejected: thousands of
   dynamic workspace namespaces make per-namespace Secret management impractical
2. hostNetwork: false — rejected: upstream enforces hostNetwork: true
   in Rook v1.17+ to prevent disrupting existing mounts on node plugin restart
   (see feat: make nodePlugin hostNetwork configurable ceph-csi-operator#520)
3. Relaxing PeerAuthentication to PERMISSIVE — rejected: corporate
   security policy requires STRICT mTLS for all in-cluster communication
4. Server-only TLS (no client cert) — rejected: corporate policy requires
   mutual TLS, not one-way TLS

---

Implementation notes

The change is contained entirely within internal/kms/vault.go:

• vaultConnection.initCertificates(): add vaultClientCertFromSecret and
  vaultClientCertKeyFromSecret handling using k8s.GetSecret()
• vaultConnection.Destroy(): extend to also remove client cert and key temp files

No changes needed to ProviderInitArgs, GetKMS, callers, or vendor files.
api.EnvVaultClientCert and api.EnvVaultClientKey are already defined in the
vendored Vault SDK (vendor/github.com/hashicorp/vault/api/client.go) and already
consumed by ConfigureTLS().

Reference implementation: PR #6375

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.