cephfs: Add KMIP support to CephFS fscrypt encryption

[Greenpepper15]

Hello,

CephFS encryption with KMIP as the KMS backend doesn't work. I'd like to add this support and am opening this issue to get community input on which of two design directions to take: either let KMIP keep using envelope encryption (where the actual encryption key is stored in the volume itself and unwrapped by KMIP on demand), or have KMIP store the encryption key directly (similar to how the existing Vault integration works). Before writing code I'd like to confirm my understanding of the problem and hear which approach the maintainers prefer.

What is the value to the end user? (why is it a priority?)

KMIP (Key Management Interoperability Protocol) is a vendor-neutral standard for applications to interact with key management systems, and is the interface most HSM appliances expose. ceph-csi already supports KMIP for RBD volume encryption, but the same KMS cannot back CephFS encryption today. Users whose only key management option is a KMIP-fronted HSM are therefore forced to either drop CephFS encryption or run HashiCorp Vault as a KMIP passthrough.

Context

All KMS implementations in ceph-csi are either of the type DEKStoreIntegrated or DEKStoreMetadata. The following code comments explain what these types mean.

// ceph-csi internal/kms/kms.go

// DEKStoreType describes what DEKStore needs to be configured when using a
// particular KMS. A KMS might support different DEKStores depending on its
// configuration.
type DEKStoreType string

const (
    // DEKStoreIntegrated indicates that the KMS itself supports storing
    // DEKs.
    DEKStoreIntegrated = DEKStoreType("")
    // DEKStoreMetadata indicates that the KMS should be configured to
    // store the DEK in the metadata of the volume.
    DEKStoreMetadata = DEKStoreType("metadata")
)

Is-State

As you can see in the following code snippet the ConfigureEncryption() function anticipates that some KMS implementations do not implement the GetSecret() function which is needed to use fscrypt (kernel module which implements file-level encryption — block storage equivalent is dmcrypt).

// ceph-csi internal/cephfs/store/volumeoptions.go

// ConfigureEncryption initializes the Ceph CSI key management from
// kmsID and credentials. Sets vo.Encryption on success.
func (vo *VolumeOptions) ConfigureEncryption(
    ctx context.Context,
    kmsID string,
    credentials map[string]string,
) error {
    kms, err := kmsapi.GetKMS(vo.Owner, kmsID, credentials)
    if err != nil {
        log.ErrorLog(ctx, "get KMS failed %+v: %v", vo, err)

        return err
    }
    // With cephfs selecting the cipher is not possible
    vo.Encryption, err = util.NewVolumeEncryption(kmsID, kms, nil)

    if errors.Is(err, util.ErrDEKStoreNeeded) {
        // fscrypt uses secrets directly from the KMS.
        // Therefore we do not support an additional DEK
        // store. Since not all "metadata" KMS support
        // GetSecret, test for support here. Postpone any
        // other error handling
        _, err := vo.Encryption.KMS.GetSecret(ctx, "")
        if errors.Is(err, kmsapi.ErrGetSecretUnsupported) {
            return err
        }
    }

    return nil
}

When we look for the KMIP KMS implementation we can clearly see that the GetSecret() function is not implemented hence the ConfigureEncryption() function will fail if KMIP is the KMS backend for ceph-csi cephfs encryption.

// ceph-csi internal/kms/kmip.go
func (kms *kmipKMS) RequiresDEKStore() DEKStoreType {
    return DEKStoreMetadata
}

func (kms *kmipKMS) GetSecret(ctx context.Context, volumeID string) (string, error) {
    return "", ErrGetSecretUnsupported
}

The kmipKMS struct is of the DekStoreType DEKStoreMetadata which "indicates that the KMS should be configured to store the DEK in the metadata of the volume." However kmipKMS never actually needs to retrieve the secret via GetSecret(), instead getting the secret is done via the DecryptDEK() implementation as seen in the following code snippet.

// ceph-csi internal/kms/kmip.go

// DecryptDEK calls either GET or DECRYPT operation on the KMIP kms to decrypt the DEK.
// The operation to call depends upon the config value `USE_CRYPTO_RPC`.
//
// If USE_CRYPTO_RPC == false : GET operation is used.
// If USE_CRYPTO_RPC == true : DECRYPT operation is used.
func (kms *kmipKMS) DecryptDEK(ctx context.Context, volumeID, encryptedDEK string) (string, error) {
    if kms.useCryptoRPC {
        return kms.decryptDEKUsingDecryptRPC(ctx, volumeID, encryptedDEK)
    }

    return kms.decryptDEKUsingRemoteKey(ctx, volumeID, encryptedDEK)
}

Thus the GetSecret() function did not need to be implemented to retrieve the secret because the DecryptDEK() retrieves the secret and decrypts it in one go.

Should-State

Practically there are two ways KMIP could work in concert with fscrypt. The following code snippet shows the core interaction between the ceph-csi fscrypt logic and the KMS backend.

// ceph-csi internal/util/fscrypt/fscrypt.go
switch encryption.KMS.RequiresDEKStore() {
case kms.DEKStoreIntegrated:
    passphrase, err = encryption.GetCryptoPassphrase(ctx, volID)
case kms.DEKStoreMetadata:
    passphrase, err = encryption.KMS.GetSecret(ctx, volID)
}

So to make KMIP work with ceph-csi fscrypt, one of two things needs to happen:

1. Keep KMIP as a DEKStoreMetadata and implement its GetSecret() method, or
2. Add a new (or extended) KMIP struct of type DEKStoreIntegrated which can use GetCryptoPassphrase() (analogous to how the vaultKMS implementation works).

Questions

I understand that from an fscrypt perspective there is no KEK, DEK (envelope encryption) needed. The KMS system just needs to store a key which can directly be used by fscrypt. This would correspond to the DEKStoreIntegrated semantics.

1. Do we want to implement this key model for ceph-csi fscrypt with KMIP — i.e. implement a new KMIP struct that is a DEKStoreIntegrated key store?

2. Alternatively, we could store the fscrypt DEK as an encrypted DEK on cephfs metadata and decrypt it in the KMIP KMS. This would match the current DEKStoreType the KMIP struct implements (DEKStoreMetadata). Would this be a better approach?

[Greenpepper15]

@Madhu-1 Hi — friendly bump on this. I realize the original write-up turned out longer than necessary, so here's the TL;DR 🥲

Problem: CephFS encryption in ceph-csi doesn't work when KMIP is used as the KMS backend.

Two proposed fixes: either the ceph-csi KMIP backend fetches the fscrypt key directly from the keystore, or the KMIP backend uses envelope encryption (a DEK wrapped by a KEK that stays in the KMS).

Happy to draft a PR for whichever direction the maintainers prefer.

[Madhu-1]

@Greenpepper15 Thanks for the write up! @black-dragon74 @Rakshith-R PTAL

[Greenpepper15]

Friendly bump 🙂 — @black-dragon74 @Rakshith-R , when you get a chance I'd really appreciate a steer on this one.

The only thing blocking me from starting is which of the two directions you'd prefer:

1. Keep KMIP as DEKStoreMetadata (envelope encryption): implement the currently-stubbed GetSecret() so the fscrypt key is stored wrapped as a DEK in the volume metadata and unwrapped by KMIP on demand (reusing its existing DecryptDEK path).
2. Add a DEKStoreIntegrated path: KMIP stores the fscrypt key directly, with no envelope encryption, exactly as the existing Vault integration already does for fscrypt.

Happy to open a draft PR for whichever you'd rather see. Just let me know which approach you prefer and I'll get started.

[Rakshith-R]

Hey @Greenpepper15 ,

Sorry for the delay.
Either or both implementations works.
You can directly open a pr.