Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
66 changes: 66 additions & 0 deletions e2e/nvmeof.go
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,7 @@ var _ = ginkgo.Describe("nvmeof", func() {
ginkgo.Context("Test NVMe CSI", ginkgo.Ordered, func() {

pvcPath := nvmeofExamplePath + "pvc.yaml"
pvcClonePath := nvmeofExamplePath + "pvc-clone.yaml"
appPath := nvmeofExamplePath + "pod.yaml"
rawPvcPath := nvmeofExamplePath + "raw-block-pvc.yaml"
rawAppPath := nvmeofExamplePath + "raw-block-pod.yaml"
Expand Down Expand Up @@ -432,5 +433,70 @@ var _ = ginkgo.Describe("nvmeof", func() {
validateRBDImageCount(f, 0, nvmeofPool)
validateOmapCount(f, 0, rbdType, nvmeofPool, volumesType)
})

ginkgo.It("Cloning nvmeof PVC", func() {
// This test validates cloning of NVMe-oF PVCs.
//
// Test flow:
// 1. Create a source PVC and bind it to an app
// 2. Create a clone of the source PVC and bind it to an app

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clone from snapshot source is not added to e2e?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gadididi is this something you still want to add?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nixpanic sure, I can do it, but is ok to make nvmeof e2e test longer than now?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@iPraveenParihar Hi! I will add snapshot tests in separated PR 🙂

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

okay, create a tracker for this.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

// 3. Delete the clone app and PVC
// 4. Delete the source app and PVC

ginkgo.By("Creating a source PVC")
sourcePVC, err := loadPVC(pvcPath)
Expect(err).ShouldNot(HaveOccurred())

sourcePVC.Namespace = f.UniqueName
sourcePVC.Spec.StorageClassName = &nvmeofStorageClass

err = createPVCAndvalidatePV(f.ClientSet, sourcePVC, deployTimeout)
Expect(err).ShouldNot(HaveOccurred())

ginkgo.By("Binding source PVC to an application")

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you don't need to bind the source volume to an app, that is validated in other tests already too.

@gadididi gadididi Jun 8, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ohh ok I can remove it. I thought you want both to be bind to app .
I will remove it after the test will be finish

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Having both is ok, but not required. The cloned volume needs complete testing.

The current PR is fine by me, no need to update it unless there is something else you want to improve too.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, I will keep it as is.
looks like the cloning test passed!

  �[1mSTEP:�[0m Creating a clone of the source PVC �[38;5;243m@ 06/08/26 13:16:31.494�[0m
  I0608 13:16:31.498376   79259 pvc.go:61] Waiting up to &PersistentVolumeClaim{ObjectMeta:{nvmeof-pvc-clone  nvmeof-3536    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:VolumeResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{67108864 0} {<nil>}  BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*e2e-nvmeof-7592-sc,VolumeMode:nil,DataSource:&TypedLocalObjectReference{APIGroup:nil,Kind:PersistentVolumeClaim,Name:nvmeof-pvc,},DataSourceRef:nil,VolumeAttributesClassName:nil,},Status:PersistentVolumeClaimStatus{Phase:,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},AllocatedResources:ResourceList{},AllocatedResourceStatuses:map[ResourceName]ClaimResourceStatus{},CurrentVolumeAttributesClassName:nil,ModifyVolumeStatus:nil,},} to be in Bound state
  I0608 13:16:31.498423   79259 pvc.go:64] waiting for PVC nvmeof-pvc-clone (0 seconds elapsed)
  I0608 13:16:33.499329   79259 pvc.go:64] waiting for PVC nvmeof-pvc-clone (2 seconds elapsed)
  I0608 13:16:33.503706   79259 pvc.go:86] PVC nvmeof-pvc-clone Event: Provisioning - External provisioner is provisioning volume for claim "nvmeof-3536/nvmeof-pvc-clone"
  I0608 13:16:33.503736   79259 pvc.go:86] PVC nvmeof-pvc-clone Event: ExternalProvisioning - Waiting for a volume to be created either by the external provisioner 'nvmeof.csi.ceph.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  I0608 13:16:35.499332   79259 pvc.go:64] waiting for PVC nvmeof-pvc-clone (4 seconds elapsed)
  I0608 13:16:35.503453   79259 pv.go:468] Waiting for PV pvc-ff99cd28-102e-47fc-8d14-8aaec7c9a280 to bind to PVC nvmeof-pvc-clone
  I0608 13:16:35.503551   79259 pv.go:790] Waiting up to timeout=10m0s for PersistentVolumeClaims [nvmeof-pvc-clone] to have phase Bound
  I0608 13:16:35.505343   79259 pv.go:801] PersistentVolumeClaim nvmeof-pvc-clone found and phase=Bound (1.770669ms)
  I0608 13:16:35.505377   79259 pv.go:763] Waiting up to 10m0s for PersistentVolume pvc-ff99cd28-102e-47fc-8d14-8aaec7c9a280 to have phase Bound
  I0608 13:16:35.507083   79259 pv.go:771] PersistentVolume pvc-ff99cd28-102e-47fc-8d14-8aaec7c9a280 found and phase=Bound (1.688721ms)
  �[1mSTEP:�[0m Binding clone PVC to an application �[38;5;243m@ 06/08/26 13:16:35.51�[0m
  I0608 13:16:35.514333   79259 warnings.go:107] "Warning: would violate PodSecurity \"restricted:latest\": allowPrivilegeEscalation != false (container \"web-server\" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container \"web-server\" must set securityContext.capabilities.drop=[\"ALL\"]), runAsNonRoot != true (pod or container \"web-server\" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container \"web-server\" must set securityContext.seccompProfile.type to \"RuntimeDefault\" or \"Localhost\")"
  I0608 13:16:35.514428   79259 pod.go:377] Waiting up to 10m0s for csi-nvmeof-demo-pod-clone to be in Running state
  �[1mSTEP:�[0m Deleting the clone application and PVC �[38;5;243m@ 06/08/26 13:16:41.518�[0m

sourceApp, err := loadApp(appPath)
Expect(err).ShouldNot(HaveOccurred())

sourceApp.Namespace = f.UniqueName
sourceApp.Spec.Volumes[0].PersistentVolumeClaim.ClaimName = sourcePVC.Name

err = createApp(f.ClientSet, sourceApp, deployTimeout)
Expect(err).ShouldNot(HaveOccurred())

ginkgo.By("Creating a clone of the source PVC")
// Load clone PVC template with DataSource already configured
clonePVC, err := loadPVC(pvcClonePath)
Expect(err).ShouldNot(HaveOccurred())

clonePVC.Namespace = f.UniqueName
clonePVC.Spec.StorageClassName = &nvmeofStorageClass
clonePVC.Spec.DataSource.Name = sourcePVC.Name

err = createPVCAndvalidatePV(f.ClientSet, clonePVC, deployTimeout)
Expect(err).ShouldNot(HaveOccurred())

ginkgo.By("Binding clone PVC to an application")
cloneApp, err := loadApp(appPath)
Expect(err).ShouldNot(HaveOccurred())

cloneApp.Name = sourceApp.Name + "-clone"
cloneApp.Namespace = f.UniqueName
cloneApp.Spec.Volumes[0].PersistentVolumeClaim.ClaimName = clonePVC.Name

err = createApp(f.ClientSet, cloneApp, deployTimeout)
Expect(err).ShouldNot(HaveOccurred())

ginkgo.By("Deleting the clone application and PVC")
err = deletePVCAndApp("", f, clonePVC, cloneApp)
Expect(err).ShouldNot(HaveOccurred())

ginkgo.By("Deleting the source application and PVC")
err = deletePVCAndApp("", f, sourcePVC, sourceApp)
Expect(err).ShouldNot(HaveOccurred())

// validate all backend rbd images are cleaned up
validateRBDImageCount(f, 0, nvmeofPool)
validateOmapCount(f, 0, rbdType, nvmeofPool, volumesType)
})
})
})
15 changes: 15 additions & 0 deletions examples/nvmeof/pvc-clone.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: nvmeof-pvc-clone
spec:
storageClassName: csi-nvmeof-sc
dataSource:
name: nvmeof-pvc
kind: PersistentVolumeClaim
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 64Mi
30 changes: 30 additions & 0 deletions internal/nvmeof/controller/controllerserver.go
Original file line number Diff line number Diff line change
Expand Up @@ -118,7 +118,37 @@ func (cs *Server) CreateVolume(
}
defer cs.volumeLocks.Release(req.GetName())

// Extract clone source if present (for locking)
// Note- there is no need for snapshot source locking because snapshots are not
Comment thread
gadididi marked this conversation as resolved.
// used in the NVMe-oF gateway and do not affect the NVMe-oF resources.
var sourceVolumeID, sourceSnapshotID string
if contentSource := req.GetVolumeContentSource(); contentSource != nil {
switch contentSource.GetType().(type) {
case *csi.VolumeContentSource_Snapshot:
if snapshot := contentSource.GetSnapshot(); snapshot != nil {
sourceSnapshotID = snapshot.GetSnapshotId()
log.DebugLog(ctx, "Creating volume from snapshot: %s", sourceSnapshotID)
}
case *csi.VolumeContentSource_Volume:
if vol := contentSource.GetVolume(); vol != nil {
sourceVolumeID = vol.GetVolumeId()
log.DebugLog(ctx, "Creating volume clone from volume: %s", sourceVolumeID)
}
}
}

// Lock source volume to prevent concurrent deletion
if sourceVolumeID != "" {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lock on snapshot is not required?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @Madhu-1 , I missed your comment here.
There is no need lock at nvmeof level for snapshot because the nvmeof snapshot call directly to rbd snapshot function..(and the rbd call has lock for that here- #6322). there are no more ops.
However, the cloning (which is running in CreateVolume()) call has nvmeof ops , so here need to use lock, prevent cloning + delete race

if acquired := cs.volumeLocks.TryAcquire(sourceVolumeID); !acquired {
log.ErrorLog(ctx, util.VolumeOperationAlreadyExistsFmt, sourceVolumeID)

return nil, status.Errorf(codes.Aborted, util.VolumeOperationAlreadyExistsFmt, sourceVolumeID)
}
defer cs.volumeLocks.Release(sourceVolumeID)
}

// Step 1: Create RBD volume through backend. if exists, it is ok.
// RBD backend automatically handles cloning when VolumeContentSource is present.
res, err := cs.backendServer.CreateVolume(ctx, req)
if err != nil {
log.ErrorLog(ctx, "failed to create RBD volume: %v", err)
Expand Down
1 change: 1 addition & 0 deletions internal/nvmeof/driver/driver.go
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,7 @@ func (d *nvmeofDriver) Run(conf *util.Config) {
csi.ControllerServiceCapability_RPC_MODIFY_VOLUME,
csi.ControllerServiceCapability_RPC_EXPAND_VOLUME,
csi.ControllerServiceCapability_RPC_CREATE_DELETE_SNAPSHOT,
csi.ControllerServiceCapability_RPC_CLONE_VOLUME,

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will add later.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please include an e2e test for cloning with this PR, thanks!

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I missed that. I am on it!

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

})

cd.AddVolumeCapabilityAccessModes([]csi.VolumeCapability_AccessMode_Mode{
Expand Down
Loading