Feature/support disk cache

[mowangdk]

What type of PR is this?

What this PR does / why we need it:

Support local disk cache for EBS

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mowangdk

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• deploy/OWNERS [mowangdk]
• pkg/disk/OWNERS [mowangdk]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[mowangdk]

Data Cache Test Report

Date: 2026-06-22
Cluster: ACK 1.36.1-aliyun.1, cn-hangzhou
CSI Plugin: v1.36.2-test

---

Environment

Item	Value
Nodes	3 (192.168.0.245/246/247)
OS	Alibaba Cloud Linux 4.0.3
Kernel	6.6.102-5.3.1.alnx4.x86_64
Local disk per node	/dev/nvme1n1, 894G, unformatted

---

Test Steps & Results

1. Node Local Disk Initialization

Action: Format /dev/nvme1n1 as ext4, mount to /var/alibaba-cloud-csi/data-cache, add to /etc/fstab.

Result: ✅ PASS — All 3 nodes initialized successfully.

Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme1n1    879G  2.1M  835G   1% /var/alibaba-cloud-csi/data-cache

2. StorageClass Creation

Action: Create alicloud-disk-essd-datacache with parameters:

• dataCacheSize: "10Gi"
• dataCacheMode: "writethrough"
• type: cloud_essd

Result: ✅ PASS

3. PVC Provisioning

Action: Create 100Gi PVC with the data-cache StorageClass.

Result: ✅ PASS — PV d-bp17dwhutfzhjsnzvpoj bound, cloud disk provisioned.

4. CSI Plugin Volume Mount Fix

Issue discovered: CSI DaemonSet did not mount /var/alibaba-cloud-csi/data-cache into the plugin container. Logs showed:

"data cache path not exist on node, proceed without cache"

Fix: Patched CSI DaemonSet to add a hostPath volume (/var/alibaba-cloud-csi/data-cache) with HostToContainer mount propagation. CSI pods restarted successfully.

Result: ✅ PASS — After fix, logs confirmed dm-cache setup.

5. Test Pod (data-cache-test)

Action: Deploy pod with PVC attached to /data.

Result: ✅ PASS — Pod Running on node 192.168.0.246.

6. dm-cache Verification

Confirmed on node via dmsetup status and losetup:

Check	Result
dm-cache device created	✅ /dev/mapper/d-bp17dwhutfzhjsnzvpoj
Cache mode	✅ writethrough
Data file (10G)	✅ d-bp17dwhutfzhjsnzvpoj.data
Meta file (16M)	✅ d-bp17dwhutfzhjsnzvpoj.meta
Loop devices	✅ loop0 → data, loop1 → meta
dmsetup status shows cache type	✅ cache 8 90/4096 512 19/40960 ... metadata2 writethrough

---

Issues Found

#	Severity	Description	Resolution
1	Critical	CSI DaemonSet missing /var/alibaba-cloud-csi/data-cache hostPath mount. Cache silently skipped.	Patched DaemonSet with hostPath volume + mount. CSI plugin code at data_cache.go:356 logs the skip but does not surface it as a warning or event.

Recommendation: The CSI Helm chart / operator should include this hostPath mount by default when data cache is supported. The silent fallback makes it easy to deploy the feature without realizing cache is not active.

---

Summary

Test Case	Status
Node disk init (format + mount + fstab)	✅ Pass
StorageClass with dataCacheSize/dataCacheMode	✅ Pass
PVC dynamic provisioning	✅ Pass
CSI plugin sees cache directory	✅ Pass (after DaemonSet fix)
dm-cache device created with correct params	✅ Pass
Test pod Running with cached volume	✅ Pass

Overall: PASS — Data cache feature works as documented, after the CSI DaemonSet mount fix.

[mowangdk]

Data Cache Test Report — Writeback Mode

Date: 2026-06-22
Cluster: ACK 1.36.1-aliyun.1, cn-hangzhou
CSI Plugin: v1.36.2-test

---

Environment

Same as previous writethrough test — 3 nodes, Alibaba Cloud Linux 4.0.3, kernel 6.6.102, local /dev/nvme1n1 (894G) formatted and mounted at /var/alibaba-cloud-csi/data-cache. CSI DaemonSet patched with the hostPath mount fix from the earlier test.

---

StorageClass Configuration

provisioner: diskplugin.csi.alibabacloud.com
parameters:
  type: cloud_essd
  dataCacheSize: "10Gi"
  dataCacheMode: "writeback"
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer

---

Test Steps & Results

1. PVC Provisioning

Result: ✅ PASS — PV d-bp12h49y8kd4wioexcuj bound, 100Gi cloud_essd provisioned.

2. Test Pod (data-cache-wb-test)

Result: ✅ PASS — Pod Running on node 192.168.0.246.

3. dm-cache Setup (from CSI logs)

"setup dm-cache" volumeID="d-bp12h49y8kd4wioexcuj"
args="/dev/loop1 /dev/loop0 /dev/disk/by-id/nvme-Alibaba_Cloud_Elastic_Block_Storage_bp12h49y8kd4wioexcuj 512 2 metadata2 writeback mq 2 migration_threshold 4096"

✅ Confirmed writeback mode passed to dmsetup.

4. dm-cache Verification (on node)

Check	Result
dm-cache device	✅ /dev/mapper/d-bp12h49y8kd4wioexcuj
Cache mode in dmsetup status	✅ metadata2 writeback
Data file (10G)	✅ d-bp12h49y8kd4wioexcuj.data
Meta file (16M)	✅ d-bp12h49y8kd4wioexcuj.meta
Loop devices	✅ loop0 → data, loop1 → meta

dmsetup status output:

0 209715200 cache 8 90/4096 512 8009/40960 107 43 915 1771 0 8009 0 2 metadata2 writeback 2 migration_threshold 4096 mq ...

Note the write-specific counters (dirty blocks: 8009, write hits: 915, write misses: 1771) are non-zero, confirming write operations are being cached locally rather than passed through to the origin — expected behavior for writeback mode.

---

Combined Summary (Both Modes)

Test Case	Writethrough	Writeback
PVC dynamic provisioning	✅	✅
Pod Running	✅	✅
dm-cache device created	✅	✅
Correct cache mode in dmsetup	✅ writethrough	✅ writeback
Cache data + meta files created	✅	✅
Loop devices attached	✅	✅

Overall: PASS — Both writethrough and writeback cache modes work as documented.

[kubernetes-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mowangdk

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• deploy/OWNERS [mowangdk]
• pkg/disk/OWNERS [mowangdk]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[mowangdk]

Complete Investigation: dm-cache Writeback Dirty Block Flush

Environment

• Kernel: 6.6.102-5.3.1.alnx4 (Alibaba Cloud Linux 4), 6.8.0-88-generic (Ubuntu 24.04)
• dm-cache: v2.2.0
• LVM: 2.03.21
• Devices: NVMe local SSD (cache), cloud disk (origin)

All Tested Flush Methods

#	Method	Device Type	Result	Notes
1	mq idle writeback	loop + dmsetup	FAIL	dirty unchanged after 30s idle
2	cleaner policy reload (our 2-step)	loop + dmsetup	FAIL	dirty unchanged; ftrace: queue_writeback never called
3	cleaner policy reload (LVM 3-step)	loop + dmsetup	FAIL	dirty unchanged with exact LVM flag sequence
4	DM_DEV_REMOVE	loop + dmsetup	FAIL	hangs or data lost
5	writethrough switch + full read	loop + dmsetup	FAIL	dirty unchanged after full device read
6	suspend without --noflush	loop + dmsetup	FAIL	metadata synced but dirty data not flushed
7	cache_writeback tool	loop + dmsetup	PASS	259/259 blocks copied
8	LVM lvconvert --uncache	dm-linear (LVM)	PASS	1592 blocks flushed, data verified
9	Manual cleaner reload on LVM dm-cache	dm-linear (LVM)	FAIL	dirty unchanged after 10min with exact LVM flags

Root Cause Analysis

Why cleaner policy doesn't work via table reload (Tests 1-3, 9):

Confirmed via ftrace on both kernels (6.6 and 6.8):

• do_waker → check_migrations → smq_get_background_work called every second ✓
• queue_writeback never called ✗
• mg_start never called ✗

This means clean_target_met() returns true (q_size(&mq->dirty) == 0). After table reload, the new smq policy object's dirty queue is empty despite metadata showing dirty blocks.

Why LVM lvconvert --uncache works (Test 8) but manual cleaner doesn't (Test 9):

LVM strace revealed lvconvert --uncache does NOT rely on cleaner policy to flush. The actual sequence:

Round 1: Load cleaner table → suspend(SKIP_LOCKFS) → resume → print "Flushing N blocks"
Round 2: Suspend all → resume all → Load LINEAR table (replaces cache) → suspend → resume
         → DM_DEV_REMOVE cache/meta/data devices

Key evidence: Before lvconvert --uncache, raw origin device cannot be mounted (data only in cache). After uncache, data exists on origin. The flush happens during Round 2's table swap from cache → linear, NOT from the cleaner policy.

When dm-cache table is swapped to linear table via dm_swap_table:

• cache_postsuspend is called on old (cache) table → syncs metadata + flushes dirty data
• This flush works when meta/data/origin are dm-linear devices (LVM's setup)
• This flush does NOT work when meta/data are loop devices (our CSI setup)

The difference is the underlying device type, not the ioctl sequence.

Verified Working Solutions

Solution A: cache_writeback tool (verified on loop devices)

1. umount filesystem
2. dmsetup suspend DM --noflush --nolockfs
3. cp meta_file meta.copy; cp data_file data.copy
4. fallocate meta.repaired; cache_repair --input meta.copy --output meta.repaired
5. dmsetup remove --deferred DM; losetup -D
6. cache_writeback --no-metadata-update \
     --metadata-device meta.repaired \
     --origin-device /dev/vdb \
     --fast-device data.copy
Result: 259/259 blocks copied, data verified PASS

Solution B: Use dm-linear instead of loop devices (verified via LVM)

LVM's approach works because it uses dm-linear devices (on NVMe partitions) instead of loop devices for cache meta/data. The kernel's cache_postsuspend correctly flushes dirty blocks when the underlying devices are dm-linear.

Trade-off: dm-linear requires manual block allocation on NVMe (partition/sector management), significantly more complex than file-based loop device approach. Essentially requires implementing a simplified LVM-like allocator.

Recommendation

Use Solution A (cache_writeback tool) for the CSI implementation:

• Works with existing loop device architecture
• No partition management complexity
• Already verified PASS
• cache_writeback is part of thin-provisioning-tools (standard package)
• User-configurable via dataCacheWritebackOnDetach StorageClass parameter

Code changes already implemented in pkg/disk/data_cache.go:

• teardownDataCache() checks writebackOnDetach config
• If enabled: suspend → copy files → cache_repair → deferred remove → cache_writeback
• If disabled: deferred remove + delete cache files (dirty data discarded)

Test Scripts

• test/disk/test-cache-writeback-flush.sh — dm-cache writeback flush verification
• test/disk/test-all-flush-methods.sh — all 5 flush methods comparison
• test/disk/lvm-test.sh — LVM lvmcache uncache verification
• test/disk/disk-data-cache-writeback.sh — E2E test for CSI integration

[mowangdk]

Evidence: LVM flush works via table swap, not cleaner policy

Test 1: Raw origin has NO data before lvconvert --uncache

# After writing 30MB data through writeback dm-cache:
dirty=1626

# Try mounting the raw origin device (bypassing cache):
mount -o ro /dev/mapper/testvg-origin_corig /mnt/raw
=> CANNOT_MOUNT (data only in cache, not on origin)

Proves: In writeback mode, data exists ONLY in cache device. Origin has no data.

Test 2: After lvconvert --uncache, data appears on origin

lvconvert --uncache testvg/origin
=> Flushing 1592 blocks for cache testvg/origin.
=> Logical volume "cachedata_cpool" successfully removed.

mount -o ro /dev/testvg/origin /mnt/v
=> data=ORIGIN_CHECK bulk=31457280
=> PASS

Proves: lvconvert --uncache successfully flushed dirty blocks to origin.

Test 3: Manual cleaner reload on same LVM dm-cache does NOT flush

# Same LVM setup, same kernel, same devices
# Using exact LVM ioctl flags: DM_TABLE_LOAD → DM_DEV_SUSPEND(SUSPEND|SKIP_LOCKFS) → resume

load=0 suspend=0 resume=0
table=cleaner (confirmed switched)
dirty after resume=1626

# Poll 10 minutes:
[1s] dirty=1626 ... [300s] dirty=1626
final=1626

Proves: Cleaner policy does NOT flush dirty blocks, even with LVM's exact flag combination.

Test 4: LVM strace shows flush happens during table swap, not cleaner

LVM does TWO rounds of suspend/load/resume:

• Round 1: Switch to cleaner table (dirty stays at 1626 — cleaner doesn't work)
• Round 2: Replace cache table with LINEAR table → kernel flushes during cache_postsuspend of table swap

Key strace evidence:

# Round 1: Load cleaner + suspend + resume (no actual flush)
DM_TABLE_LOAD (cleaner table)
DM_DEV_SUSPEND (SUSPEND_FLAG|SKIP_LOCKFS_FLAG)
DM_DEV_SUSPEND (resume - no SUSPEND_FLAG)
write(1, "Flushing 1591 blocks") ← just prints dirty count, not actual flush

# Round 2: Replace cache with linear (actual flush happens here)
DM_TABLE_LOAD (linear table - replaces cache)
DM_DEV_SUSPEND (SUSPEND_FLAG|SKIP_LOCKFS_FLAG)
DM_DEV_SUSPEND (resume)
DM_DEV_REMOVE (cache devices)

Proves: LVM's "Flushing N blocks" message is just logging the dirty count. The actual data flush occurs when the kernel swaps the cache table for a linear table during dm_swap_table → cache_postsuspend.

Test 5: This kernel-level flush only works with dm-linear, not loop devices

• dm-linear (LVM): cache_postsuspend during table swap flushes dirty blocks ✓
• loop devices (our CSI): cache_postsuspend does NOT flush dirty blocks ✗
• Confirmed by: dmsetup remove on loop-based dm-cache hangs or loses data, while LVM's DM_DEV_REMOVE on dm-linear-based dm-cache succeeds with data intact

Conclusion

The dm-cache writeback flush mechanism in kernel 6.6 only works correctly when cache meta/data devices are dm-linear (backed by real block devices). When using loop devices (backed by files), the kernel's cache_postsuspend flush path does not function. This is why cache_writeback (userspace tool) is needed for our CSI implementation which uses loop devices for cache storage.

[mowangdk]

We decided to exclude cache writeback from this PR after testing. @huww98 ptal