Cannot import DV into a block device

[prezhdarov]

What happened:
Trying the usual import of a qcow2 image into a block device DV that worked in 1.64, now fails.

What you expected to happen:
Import the image and start the virtual machine.

How to reproduce it (as minimally and precisely as possible):
Create a virtual machine template or just DV with similar specs:

apiVersion: cdi.kubevirt.io/v1beta1
kind: DataVolume
metadata:
   name: somevm-root
spec:
   preallocation: false
   source:
      http:
         url: https://some.web.tld/somevm.qcow2
   storage:
      accessModes:
      - ReadWriteMany
      resources:
         requests:
            storage: 60Gi
      storageClassName: aff-a-nvme-flexvol  
      volumeMode: Block

where 'aff-a-nvme-flexvol' is NetApp Trident nvme provisioned storage class.

Additional context:
Had to change scratch storage configuration to:

      scratchSpaceStorageClass: aff-a-nfs-flexvol
      filesystemOverhead: 
        storageClass:
          aff-a-nfs-flexvol: "0.15"

or importer would fail with either not enough space or permission denied if block storage scratch was used. The 'aff-a-nfs-flexvol' storage class is similar NetApp Trident pnfs4.1 provisioned storage class.

Environment:

• CDI version (use kubectl get deployments cdi-deployment -o yaml): 1.65
• Kubernetes version (use kubectl version): 1.35.5
• DV specification: posted above
• Cloud provider or hardware configuration: Dell R6615 with AMD EPYC 9334
• OS (e.g. from /etc/os-release): Debian 13.5
• Kernel (e.g. uname -a): 7.0.10+deb13-amd64
• Install tools: ?
• Others: ?

importer-prime logs

I0617 12:15:08.550549       1 prometheus.go:78] 99.87
I0617 12:15:08.972676       1 file.go:195] Read 6710820864 bytes, wrote 6701330432 bytes to /scratch/tmpimage
I0617 12:15:08.972915       1 data-processor.go:260] New phase: Convert
I0617 12:15:08.972922       1 data-processor.go:266] Validating image
E0617 12:15:08.979122       1 prlimit.go:156] failed to kill the process; os: process already finished
I0617 12:15:08.979174       1 qemu.go:119] Running qemu-img with args: [convert -t writeback -p -O raw /scratch/tmpimage /dev/cdi-block-volume]
I0617 12:15:08.981911       1 qemu.go:283] 0.00
E0617 12:15:08.983110       1 prlimit.go:178] qemu-img failed output is:
E0617 12:15:08.983117       1 prlimit.go:179]     (0.00/100%)


E0617 12:15:08.983121       1 prlimit.go:180] qemu-img: /dev/cdi-block-volume: error while converting raw: Failed to clear the new image's first sector: Invalid argument

E0617 12:15:08.983136       1 prlimit.go:156] failed to kill the process; os: process already finished
E0617 12:15:08.983199       1 data-processor.go:256] exit status 1
qemu-img execution failed
kubevirt.io/containerized-data-importer/pkg/system.executeWithLimits
	pkg/system/prlimit.go:182
kubevirt.io/containerized-data-importer/pkg/system.ExecWithLimits
	pkg/system/prlimit.go:112
kubevirt.io/containerized-data-importer/pkg/image.convertToRaw
	pkg/image/qemu.go:120
kubevirt.io/containerized-data-importer/pkg/image.(*qemuOperations).ConvertToRawStream
	pkg/image/qemu.go:169
kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).convert
	pkg/importer/data-processor.go:285
kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).initDefaultPhases.func7
	pkg/importer/data-processor.go:218
kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessDataWithPause
	pkg/importer/data-processor.go:253
kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessData
	pkg/importer/data-processor.go:151
main.handleImport
	cmd/cdi-importer/importer.go:189
main.main
	cmd/cdi-importer/importer.go:149
runtime.main
	GOROOT/src/runtime/proc.go:283
runtime.goexit
	src/runtime/asm_amd64.s:1700
could not convert image to raw Log line from nbdkit: nbdkit: curl[1]: error: readahead: warning: underlying plugin does not support NBD_CMD_CACHE or PARALLEL thread model, so the filter won't do anythingLog line from nbdkit: nbdkit: curl[2]: error: readahead: warning: underlying plugin does not support NBD_CMD_CACHE or PARALLEL thread model, so the filter won't do anything
kubevirt.io/containerized-data-importer/pkg/image.convertToRaw
	pkg/image/qemu.go:128
kubevirt.io/containerized-data-importer/pkg/image.(*qemuOperations).ConvertToRawStream
	pkg/image/qemu.go:169
kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).convert
	pkg/importer/data-processor.go:285
kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).initDefaultPhases.func7
	pkg/importer/data-processor.go:218
kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessDataWithPause
	pkg/importer/data-processor.go:253
kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessData
	pkg/importer/data-processor.go:151
main.handleImport
	cmd/cdi-importer/importer.go:189
main.main
	cmd/cdi-importer/importer.go:149
runtime.main
	GOROOT/src/runtime/proc.go:283
runtime.goexit
	src/runtime/asm_amd64.s:1700
Conversion to Raw failed
kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).convert
	pkg/importer/data-processor.go:287
kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).initDefaultPhases.func7
	pkg/importer/data-processor.go:218
kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessDataWithPause
	pkg/importer/data-processor.go:253
kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessData
	pkg/importer/data-processor.go:151
main.handleImport
	cmd/cdi-importer/importer.go:189
main.main
	cmd/cdi-importer/importer.go:149
runtime.main
	GOROOT/src/runtime/proc.go:283
runtime.goexit
	src/runtime/asm_amd64.s:1700
Unable to convert source data to target format
kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).initDefaultPhases.func7
	pkg/importer/data-processor.go:220
kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessDataWithPause
	pkg/importer/data-processor.go:253
kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessData
	pkg/importer/data-processor.go:151
main.handleImport
	cmd/cdi-importer/importer.go:189
main.main
	cmd/cdi-importer/importer.go:149
runtime.main
	GOROOT/src/runtime/proc.go:283
runtime.goexit
	src/runtime/asm_amd64.s:1700
E0617 12:15:08.983274       1 importer.go:194] exit status 1
qemu-img execution failed
kubevirt.io/containerized-data-importer/pkg/system.executeWithLimits
	pkg/system/prlimit.go:182
kubevirt.io/containerized-data-importer/pkg/system.ExecWithLimits
	pkg/system/prlimit.go:112
kubevirt.io/containerized-data-importer/pkg/image.convertToRaw
	pkg/image/qemu.go:120
kubevirt.io/containerized-data-importer/pkg/image.(*qemuOperations).ConvertToRawStream
	pkg/image/qemu.go:169
kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).convert
	pkg/importer/data-processor.go:285
kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).initDefaultPhases.func7
	pkg/importer/data-processor.go:218
kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessDataWithPause
	pkg/importer/data-processor.go:253
kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessData
	pkg/importer/data-processor.go:151
main.handleImport
	cmd/cdi-importer/importer.go:189
main.main
	cmd/cdi-importer/importer.go:149
runtime.main
	GOROOT/src/runtime/proc.go:283
runtime.goexit
	src/runtime/asm_amd64.s:1700
could not convert image to raw Log line from nbdkit: nbdkit: curl[1]: error: readahead: warning: underlying plugin does not support NBD_CMD_CACHE or PARALLEL thread model, so the filter won't do anythingLog line from nbdkit: nbdkit: curl[2]: error: readahead: warning: underlying plugin does not support NBD_CMD_CACHE or PARALLEL thread model, so the filter won't do anything
kubevirt.io/containerized-data-importer/pkg/image.convertToRaw
	pkg/image/qemu.go:128
kubevirt.io/containerized-data-importer/pkg/image.(*qemuOperations).ConvertToRawStream
	pkg/image/qemu.go:169
kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).convert
	pkg/importer/data-processor.go:285
kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).initDefaultPhases.func7
	pkg/importer/data-processor.go:218
kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessDataWithPause
	pkg/importer/data-processor.go:253
kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessData
	pkg/importer/data-processor.go:151
main.handleImport
	cmd/cdi-importer/importer.go:189
main.main
	cmd/cdi-importer/importer.go:149
runtime.main
	GOROOT/src/runtime/proc.go:283
runtime.goexit
	src/runtime/asm_amd64.s:1700
Conversion to Raw failed
kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).convert
	pkg/importer/data-processor.go:287
kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).initDefaultPhases.func7
	pkg/importer/data-processor.go:218
kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessDataWithPause
	pkg/importer/data-processor.go:253
kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessData
	pkg/importer/data-processor.go:151
main.handleImport
	cmd/cdi-importer/importer.go:189
main.main
	cmd/cdi-importer/importer.go:149
runtime.main
	GOROOT/src/runtime/proc.go:283
runtime.goexit
	src/runtime/asm_amd64.s:1700
Unable to convert source data to target format
kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).initDefaultPhases.func7
	pkg/importer/data-processor.go:220
kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessDataWithPause
	pkg/importer/data-processor.go:253
kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessData
	pkg/importer/data-processor.go:151
main.handleImport
	cmd/cdi-importer/importer.go:189
main.main
	cmd/cdi-importer/importer.go:149
runtime.main
	GOROOT/src/runtime/proc.go:283
runtime.goexit
	src/runtime/asm_amd64.s:1700

[prezhdarov]

Tried this with importer 1.64.0 and worked fine:

I0618 12:17:56.678152       1 prometheus.go:78] 99.24
I0618 12:17:57.678956       1 prometheus.go:78] 99.47
I0618 12:17:58.679207       1 prometheus.go:78] 99.83
I0618 12:17:59.357888       1 file.go:195] Read 6710820864 bytes, wrote 6700920832 bytes to /scratch/tmpimage
I0618 12:17:59.358275       1 data-processor.go:260] New phase: Convert
I0618 12:17:59.358284       1 data-processor.go:266] Validating image
E0618 12:17:59.364564       1 prlimit.go:156] failed to kill the process; os: process already finished
I0618 12:17:59.364612       1 qemu.go:119] Running qemu-img with args: [convert -t writeback -p -O raw /scratch/tmpimage /dev/cdi-block-volume]
I0618 12:17:59.367503       1 qemu.go:283] 0.00
I0618 12:17:59.741810       1 prometheus.go:78] 100.00
I0618 12:17:59.974223       1 qemu.go:283] 1.00
I0618 12:18:00.370339       1 qemu.go:283] 2.00
I0618 12:18:00.857544       1 qemu.go:283] 3.00
I0618 12:18:01.254979       1 qemu.go:283] 4.00
...
I0618 12:18:46.547364       1 qemu.go:283] 98.35
I0618 12:18:46.798444       1 qemu.go:283] 99.36
E0618 12:19:05.167023       1 prlimit.go:156] failed to kill the process; os: process already finished
I0618 12:19:05.167043       1 data-processor.go:260] New phase: Resize
I0618 12:19:05.167819       1 data-processor.go:260] New phase: Complete
I0618 12:19:05.168034       1 importer.go:231] {"scratchSpaceRequired":false,"preallocationApplied":false,"message":"Import Complete"}

The only difference is setting this in the operator deployment:

        - name: IMPORTER_IMAGE
          value: quay.io/kubevirt/cdi-importer:v1.64.0

[akalenyu]

Hey, thanks for reporting.
I'm confused since the importer log you attached does not have errors about permission/size limits?

EDIT: seems to be related to this change - #4054
though I'm not sure why you'd have to bump the overhead % (1.64/65 makes no difference there)

Anyhow, the scratch space logic change is desired from our side. We put out a release note but TBH we hoped that the target SC could provision stuff just as well as the default SC. Does this not hold for you case with

error while converting raw: Failed to clear the new image's first sector: Invalid argument

[akalenyu]

/cc @dsanatar

[lexfrei]

I hit the same failure on a different storage backend — a Cozystack cluster with a LINSTOR/DRBD-backed Block storage class — so it doesn't look storage-specific.

I ran the real cdi-importer binary for both versions in identical pods (same node, same SC, same 5Gi Block target + scratch, identical IMPORTER_* env), changing only the image tag:

• cdi-importer:v1.64.0 → ✅ Import Complete
• cdi-importer:v1.65.0 → ❌ qemu-img: /dev/cdi-block-volume: error while converting raw: Failed to clear the new image's first sector: Invalid argument (exit 1, at 0%)

The telling part: pkg/image/qemu.go and pkg/importer/data-processor.go are byte-identical between the v1.64.0 and v1.65.0 tags, and both versions stage to /scratch/tmpimage and run the same qemu-img convert -t writeback -p -O raw /scratch/tmpimage /dev/cdi-block-volume. The only delta between the two images is the bundled qemu:

• v1.64.0: qemu-img 9.1.0 (qemu-kvm-9.1.0-19.el9)
• v1.65.0: qemu-img 10.1.0 (qemu-kvm-10.1.0-9.el9)

(nbdkit is 1.38.5 in both, both CentOS Stream 9.)

So this looks like a qemu-img 10.x change — clearing the first sector of the raw block target now returns EINVAL on at least two different block backends — rather than the scratch-space SC change in #4054 (that path is unchanged here, and both versions stage to scratch identically). Happy to grab an strace of the failing ioctl if that helps narrow it down.

[akalenyu]

I hit the same failure on a different storage backend — a Cozystack cluster with a LINSTOR/DRBD-backed Block storage class — so it doesn't look storage-specific.

I ran the real cdi-importer binary for both versions in identical pods (same node, same SC, same 5Gi Block target + scratch, identical IMPORTER_* env), changing only the image tag:

• cdi-importer:v1.64.0 → ✅ Import Complete
• cdi-importer:v1.65.0 → ❌ qemu-img: /dev/cdi-block-volume: error while converting raw: Failed to clear the new image's first sector: Invalid argument (exit 1, at 0%)

The telling part: pkg/image/qemu.go and pkg/importer/data-processor.go are byte-identical between the v1.64.0 and v1.65.0 tags, and both versions stage to /scratch/tmpimage and run the same qemu-img convert -t writeback -p -O raw /scratch/tmpimage /dev/cdi-block-volume. The only delta between the two images is the bundled qemu:

• v1.64.0: qemu-img 9.1.0 (qemu-kvm-9.1.0-19.el9)
• v1.65.0: qemu-img 10.1.0 (qemu-kvm-10.1.0-9.el9)

(nbdkit is 1.38.5 in both, both CentOS Stream 9.)

So this looks like a qemu-img 10.x change — clearing the first sector of the raw block target now returns EINVAL on at least two different block backends — rather than the scratch-space SC change in #4054 (that path is unchanged here, and both versions stage to scratch identically). Happy to grab an strace of the failing ioctl if that helps narrow it down.

thanks for the deep dive! are those 4k sector size block devices?

[lexfrei]

Yes — 4Kn. Reading the freshly-provisioned LINSTOR/DRBD block volume from inside the importer pod:

$ blockdev --getss   /dev/cdi-block-volume   →  4096    # logical
$ blockdev --getpbsz /dev/cdi-block-volume   →  16384   # physical
$ blockdev --getiomin /dev/cdi-block-volume  →  16384
$ blockdev --getioopt /dev/cdi-block-volume  →  33554432

And the same qemu-img: ... Failed to clear the new image's first sector: Invalid argument reproduces with an empty source file — no nbdkit, no real image data:

$ : >/scratch/empty
$ qemu-img convert -t writeback -p -O raw /scratch/empty /dev/cdi-block-volume
    (0.00/100%)qemu-img: /dev/cdi-block-volume: error while converting raw: Failed to clear the new image's first sector: Invalid argument

So it really is the "clear first sector" zero-out itself returning EINVAL on a 4Kn block device, independent of the source path. Fits the original reporter's NetApp Trident NVMe FlexVol too — those default to 4Kn as well. Likely a qemu 10 change in bdrv_make_zero / pwrite_zeroes that doesn't honour the device's logical block size; happy to grab the strace if useful.

[akalenyu]

Yes — 4Kn. Reading the freshly-provisioned LINSTOR/DRBD block volume from inside the importer pod:

$ blockdev --getss   /dev/cdi-block-volume   →  4096    # logical
$ blockdev --getpbsz /dev/cdi-block-volume   →  16384   # physical
$ blockdev --getiomin /dev/cdi-block-volume  →  16384
$ blockdev --getioopt /dev/cdi-block-volume  →  33554432

And the same qemu-img: ... Failed to clear the new image's first sector: Invalid argument reproduces with an empty source file — no nbdkit, no real image data:

$ : >/scratch/empty
$ qemu-img convert -t writeback -p -O raw /scratch/empty /dev/cdi-block-volume
    (0.00/100%)qemu-img: /dev/cdi-block-volume: error while converting raw: Failed to clear the new image's first sector: Invalid argument

So it really is the "clear first sector" zero-out itself returning EINVAL on a 4Kn block device, independent of the source path. Fits the original reporter's NetApp Trident NVMe FlexVol too — those default to 4Kn as well. Likely a qemu 10 change in bdrv_make_zero / pwrite_zeroes that doesn't honour the device's logical block size; happy to grab the strace if useful.

Hey, checked with virt folks, specifically Kevin Wolf, it could very well be an issue that got fixed in later 10.1:
#4179
Once that merges you should be able to try the latest quay image with the fix.

[akalenyu]

/reopen
chance to test..

[kubevirt-bot]

@akalenyu: Reopened this issue.

Details

In response to this:

/reopen
chance to test..

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.