e2e: add nvmeof driver testing support with k8s-e2e-external-storage by nixpanic · Pull Request #6214 · ceph/ceph-csi

nixpanic · 2026-03-31T11:31:03Z

Add configuration files and scripts to enable NVMeoF driver testing in the
k8s-e2e-external-storage CI job. This includes the driver manifest,
StorageClass template, and VolumeSnapshotClass template for NVMeoF.

Also deploy the NVMe-oF gateway with Rook.

Depends-on: #5672 for NVMe-oF gateway support

nixpanic · 2026-03-31T11:32:02Z

/test ci/centos/k8s-e2e-external-storage/1.35

nixpanic · 2026-03-31T11:36:37Z

/test ci/centos/k8s-e2e-external-storage/1.35

Job is still running after addressing the MarkDown linter failures.

nixpanic · 2026-04-08T16:40:24Z

@Mergifyio rebase

nixpanic · 2026-04-08T16:41:39Z

/test ci/centos/k8s-e2e-external-storage/1.35

mergify · 2026-04-08T16:41:41Z

Deprecation notice: This pull request comes from a fork and was rebased using bot_account impersonation. This capability will be removed on July 1, 2026. After this date, the rebase action will no longer be able to rebase fork pull requests with this configuration. Please switch to the update action/command to ensure compatibility going forward.

mergify · 2026-04-08T16:41:42Z

rebase

✅ Branch has been successfully rebased

nixpanic · 2026-04-09T08:04:45Z

/test ci/centos/k8s-e2e-external-storage/1.35

nixpanic · 2026-04-09T08:24:32Z

/test ci/centos/k8s-e2e-external-storage/1.35

nixpanic · 2026-04-09T08:50:42Z

/test ci/centos/k8s-e2e-external-storage/1.35

nixpanic · 2026-04-09T09:07:13Z

/test ci/centos/k8s-e2e-external-storage/1.35

nixpanic · 2026-04-09T09:25:09Z

/test ci/centos/k8s-e2e-external-storage/1.35

nixpanic · 2026-04-09T09:51:21Z

job is running need to inspect the result of the nvmeof driver once done.

/cc @gadididi

nixpanic · 2026-04-09T13:03:25Z

/test ci/centos/k8s-e2e-external-storage/1.35

nixpanic · 2026-04-09T13:52:49Z

/test ci/centos/k8s-e2e-external-storage/1.35

nixpanic · 2026-04-09T15:00:16Z

/test ci/centos/k8s-e2e-external-storage/1.35

nixpanic · 2026-04-10T06:45:17Z

/test ci/centos/k8s-e2e-external-storage/1.35

nixpanic · 2026-04-10T08:52:01Z

/test ci/centos/k8s-e2e-external-storage/1.35

ceph-csi-bot · 2026-07-20T17:27:57Z

/test ci/centos/k8s-e2e-external-storage/1.35

ceph-csi-bot · 2026-07-20T17:27:57Z

/test ci/centos/k8s-e2e-external-storage/1.36

ceph-csi-bot · 2026-07-20T17:27:58Z

/test ci/centos/upgrade-tests-cephfs

ceph-csi-bot · 2026-07-20T17:27:58Z

/test ci/centos/k8s-e2e-external-storage/1.34

ceph-csi-bot · 2026-07-20T17:27:58Z

/test ci/centos/mini-e2e-helm/k8s-1.35

ceph-csi-bot · 2026-07-20T17:27:58Z

/test ci/centos/mini-e2e-helm/k8s-1.36

ceph-csi-bot · 2026-07-20T17:27:58Z

/test ci/centos/upgrade-tests-rbd

ceph-csi-bot · 2026-07-20T17:27:58Z

/test ci/centos/mini-e2e-helm/k8s-1.34

ceph-csi-bot · 2026-07-20T17:27:58Z

/test ci/centos/mini-e2e/k8s-1.35

ceph-csi-bot · 2026-07-20T17:27:58Z

/test ci/centos/mini-e2e/k8s-1.36

ceph-csi-bot · 2026-07-20T17:27:59Z

/test ci/centos/mini-e2e/k8s-1.34

mergify · 2026-07-21T06:50:22Z

Merge Queue Status

🟠 Waiting for queue conditions
⏳ Enter queue
⏳ Run checks
⏳ Merge

Waiting for any of

All conditions

nixpanic · 2026-07-21T16:18:50Z

/test ci/centos/mini-e2e/k8s-1.36

nixpanic · 2026-07-21T16:19:58Z

/test ci/centos/mini-e2e/k8s-1.36

All CI jobs failed some Vault KMS testing. This PR should not affect that, but an other PR passed the same tests today. Maybe there was a temporary issue?

nixpanic · 2026-07-22T11:11:28Z

/test ci/centos/mini-e2e/k8s-1.36

Failed during NFS VolumeGroupSnapshot:

  I0721 17:21:25.559823   83701 pvc.go:86] PVC pvc-1-clone-2 Event: ProvisioningFailed - rpc error: code = Unknown desc = rados: ret=-22, Invalid argument: "Traceback (most recent call last):\n  File \"/usr/share/ceph/mgr/volumes/fs/operations/versions/subvolume_base.py\", line 393, in discover\n    self.fs.stat(self.base_path)\n  File \"cephfs.pyx\", line 1992, in cephfs.LibCephFS.stat\ncephfs.ObjectNotFound: error in stat: /volumes/csi/nfs-export-f080d6e1-b2b0-4e8a-8de7-97514a402a58: No such file or directory [Errno 2]\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/usr/share/ceph/mgr/volumes/fs/volume.py\", line 969, in _clone_subvolume_snapshot\n    with open_subvol(self.mgr, fs_handle, self.volspec, target_group, target_subvolname, SubvolumeOpType.CLONE_CREATE):\n  File \"/usr/lib64/python3.9/contextlib.py\", line 119, in __enter__\n    return next(self.gen)\n  File \"/usr/share/ceph/mgr/volumes/fs/operations/subvolume.py\", line 76, in open_subvol\n    subvolume = loaded_subvolumes.get_subvolume_object(mgr, fs, vol_spec, group, subvolname)\n  File \"/usr/share/ceph/mgr/volumes/fs/operations/versions/__init__.py\", line 110, in get_subvolume_object\n    subvolume.discover()\n  File \"/usr/share/ceph/mgr/volumes/fs/operations/versions/subvolume_base.py\", line 413, in discover\n    raise (VolumeException(-errno.ENOENT,\nvolumes.fs.exception.VolumeException: -2 (subvolume 'nfs-export-f080d6e1-b2b0-4e8a-8de7-97514a402a58' does not exist)\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/usr/share/ceph/mgr/mgr_module.py\", line 1995, in _handle_command\n    return self.handle_command(inbuf, cmd)\n  File \"/usr/share/ceph/mgr/volumes/module.py\", line 696, in handle_command\n    return handler(inbuf, cmd)\n  File \"/usr/share/ceph/mgr/volumes/module.py\", line 40, in wrap\n    return f(self, inbuf, cmd)\n  File \"/usr/share/ceph/mgr/volumes/module.py\", line 1073, in _cmd_fs_subvolume_snapshot_clone\n    return self.vc.clone_subvolume_snapshot(\n  File \"/usr/share/ceph/mgr/volumes/fs/volume.py\", line 992, in clone_subvolume_snapshot\n    self._clone_subvolume_snapshot(fs_handle, volname, s_group, s_subvolume, **kwargs)\n  File \"/usr/share/ceph/mgr/volumes/fs/volume.py\", line 973, in _clone_subvolume_snapshot\n    self._prepare_clone_subvolume(fs_handle, volname, s_subvolume, s_snapname,\n  File \"/usr/share/ceph/mgr/volumes/fs/volume.py\", line 949, in _prepare_clone_subvolume\n    self.clone_progress_reporter.initiate_reporting()\n  File \"/usr/share/ceph/mgr/volumes/fs/stats_util.py\", line 157, in initiate_reporting\n    self.update_task.start()\n  File \"/usr/lib64/python3.9/threading.py\", line 899, in start\n    _start_new_thread(self._bootstrap, ())\nRuntimeError: can't start new thread\n"

Traceback (most recent call last):
  File \"/usr/share/ceph/mgr/volumes/fs/operations/versions/subvolume_base.py\", line 393, in discover
    self.fs.stat(self.base_path)
  File \"cephfs.pyx\", line 1992, in cephfs.LibCephFS.stat
cephfs.ObjectNotFound: error in stat: /volumes/csi/nfs-export-f080d6e1-b2b0-4e8a-8de7-97514a402a58: No such file or directory [Errno 2]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File \"/usr/share/ceph/mgr/volumes/fs/volume.py\", line 969, in _clone_subvolume_snapshot
    with open_subvol(self.mgr, fs_handle, self.volspec, target_group, target_subvolname, SubvolumeOpType.CLONE_CREATE):
  File \"/usr/lib64/python3.9/contextlib.py\", line 119, in __enter__
    return next(self.gen)
  File \"/usr/share/ceph/mgr/volumes/fs/operations/subvolume.py\", line 76, in open_subvol
    subvolume = loaded_subvolumes.get_subvolume_object(mgr, fs, vol_spec, group, subvolname)
  File \"/usr/share/ceph/mgr/volumes/fs/operations/versions/__init__.py\", line 110, in get_subvolume_object
    subvolume.discover()
  File \"/usr/share/ceph/mgr/volumes/fs/operations/versions/subvolume_base.py\", line 413, in discover
    raise (VolumeException(-errno.ENOENT,
volumes.fs.exception.VolumeException: -2 (subvolume 'nfs-export-f080d6e1-b2b0-4e8a-8de7-97514a402a58' does not exist)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File \"/usr/share/ceph/mgr/mgr_module.py\", line 1995, in _handle_command
    return self.handle_command(inbuf, cmd)
  File \"/usr/share/ceph/mgr/volumes/module.py\", line 696, in handle_command
    return handler(inbuf, cmd)
  File \"/usr/share/ceph/mgr/volumes/module.py\", line 40, in wrap
    return f(self, inbuf, cmd)
  File \"/usr/share/ceph/mgr/volumes/module.py\", line 1073, in _cmd_fs_subvolume_snapshot_clone
    return self.vc.clone_subvolume_snapshot(
  File \"/usr/share/ceph/mgr/volumes/fs/volume.py\", line 992, in clone_subvolume_snapshot
    self._clone_subvolume_snapshot(fs_handle, volname, s_group, s_subvolume, **kwargs)
  File \"/usr/share/ceph/mgr/volumes/fs/volume.py\", line 973, in _clone_subvolume_snapshot
    self._prepare_clone_subvolume(fs_handle, volname, s_subvolume, s_snapname,
  File \"/usr/share/ceph/mgr/volumes/fs/volume.py\", line 949, in _prepare_clone_subvolume
    self.clone_progress_reporter.initiate_reporting()
  File \"/usr/share/ceph/mgr/volumes/fs/stats_util.py\", line 157, in initiate_reporting
    self.update_task.start()
  File \"/usr/lib64/python3.9/threading.py\", line 899, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread

logs

nixpanic · 2026-07-23T08:31:18Z

/test ci/centos/mini-e2e/k8s-1.36

mergify · 2026-07-23T08:32:43Z

Merge Queue Status

🟠 Waiting for queue conditions
⏳ Enter queue
⏳ Run checks
⏳ Merge

Waiting for

All conditions

The IP-address of the gateway is configured in the StorageClass. This is required as the short-hostname can not be resolved in DNS. The gateway and Ceph-CSI get deployed in different namespaces, which means the domain names differ. The same issue happens in sripts/k8s-storage/create-storageclasses.sh. Signed-off-by: Niels de Vos <ndevos@ibm.com>

nixpanic · 2026-07-23T15:15:59Z

/test ci/centos/mini-e2e/k8s-1.36

Pull request has been modified.

nixpanic · 2026-07-24T08:52:29Z

/test ci/centos/mini-e2e/k8s-1.35

mergify · 2026-07-24T08:53:16Z

Merge Queue Status

🟠 Waiting for queue conditions
⏳ Enter queue
⏳ Run checks
⏳ Merge

Waiting for

All conditions

nixpanic · 2026-07-24T12:35:30Z

/test ci/centos/mini-e2e/k8s-1.34

nixpanic added the ci/skip/multi-arch-build skip building on multiple architectures label Mar 31, 2026

mergify Bot added component/nvme-of Issues and PRs related to NVMe-oF. component/testing Additional test cases or CI work labels Mar 31, 2026

nixpanic force-pushed the nvmeof/e2e-external-storage branch from 44d2d02 to 7f22c0f Compare March 31, 2026 11:36

ceph-csi-bot force-pushed the nvmeof/e2e-external-storage branch from 7f22c0f to 0a51e55 Compare April 8, 2026 16:41

nixpanic force-pushed the nvmeof/e2e-external-storage branch from 0a51e55 to 9d2cfcf Compare April 9, 2026 08:04

nixpanic force-pushed the nvmeof/e2e-external-storage branch from 691d9e4 to ca6dc6e Compare April 9, 2026 08:50

nixpanic force-pushed the nvmeof/e2e-external-storage branch from ca6dc6e to 9e6d963 Compare April 9, 2026 09:07

nixpanic force-pushed the nvmeof/e2e-external-storage branch from 9e6d963 to 5a2e14b Compare April 9, 2026 09:24

nixpanic force-pushed the nvmeof/e2e-external-storage branch from 5a2e14b to 783a7fd Compare April 9, 2026 13:03

nixpanic force-pushed the nvmeof/e2e-external-storage branch from 8cc3df5 to 47c3972 Compare April 9, 2026 14:59

nixpanic force-pushed the nvmeof/e2e-external-storage branch from 47c3972 to 51d1351 Compare April 10, 2026 06:45

nixpanic force-pushed the nvmeof/e2e-external-storage branch from 51d1351 to f182e4c Compare April 10, 2026 08:51

iPraveenParihar added the ok-to-test Label to trigger E2E tests label Jul 20, 2026

ceph-csi-bot added ci/in-progress/e2e This label acts like a guard and prevents Mergify from adding the `ok-to-test` label again. and removed ok-to-test Label to trigger E2E tests labels Jul 20, 2026

mergify Bot removed the ci/in-progress/e2e This label acts like a guard and prevents Mergify from adding the `ok-to-test` label again. label Jul 20, 2026

gadididi previously approved these changes Jul 21, 2026

View reviewed changes

Uh oh!

Conversation

nixpanic commented Mar 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nixpanic commented Mar 31, 2026

Uh oh!

nixpanic commented Mar 31, 2026

Uh oh!

nixpanic commented Apr 8, 2026

Uh oh!

nixpanic commented Apr 8, 2026

Uh oh!

mergify Bot commented Apr 8, 2026

Uh oh!

mergify Bot commented Apr 8, 2026

✅ Branch has been successfully rebased

Uh oh!

nixpanic commented Apr 9, 2026

Uh oh!

nixpanic commented Apr 9, 2026

Uh oh!

nixpanic commented Apr 9, 2026

Uh oh!

nixpanic commented Apr 9, 2026

Uh oh!

nixpanic commented Apr 9, 2026

Uh oh!

nixpanic commented Apr 9, 2026

Uh oh!

nixpanic commented Apr 9, 2026

Uh oh!

nixpanic commented Apr 9, 2026

Uh oh!

nixpanic commented Apr 9, 2026

Uh oh!

nixpanic commented Apr 10, 2026

Uh oh!

nixpanic commented Apr 10, 2026

Uh oh!

ceph-csi-bot commented Jul 20, 2026

Uh oh!

ceph-csi-bot commented Jul 20, 2026

Uh oh!

ceph-csi-bot commented Jul 20, 2026

Uh oh!

ceph-csi-bot commented Jul 20, 2026

Uh oh!

ceph-csi-bot commented Jul 20, 2026

Uh oh!

ceph-csi-bot commented Jul 20, 2026

Uh oh!

ceph-csi-bot commented Jul 20, 2026

Uh oh!

ceph-csi-bot commented Jul 20, 2026

Uh oh!

ceph-csi-bot commented Jul 20, 2026

Uh oh!

ceph-csi-bot commented Jul 20, 2026

Uh oh!

ceph-csi-bot commented Jul 20, 2026

Uh oh!

mergify Bot commented Jul 21, 2026

Merge Queue Status

Uh oh!

nixpanic commented Jul 21, 2026

Uh oh!

nixpanic commented Jul 21, 2026

Uh oh!

nixpanic commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nixpanic commented Jul 23, 2026

Uh oh!

mergify Bot commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge Queue Status

Uh oh!

nixpanic commented Jul 23, 2026

Uh oh!

nixpanic commented Mar 31, 2026 •

edited

Loading

nixpanic commented Jul 22, 2026 •

edited

Loading

mergify Bot commented Jul 23, 2026 •

edited

Loading