The below comes from a Copilot suggestion, it applies to CephFS and RBD as well.
Race Condition During Concurrent Clone and Delete:
Scenario:
- PVC-A (vol-123) exists and is mounted
- User creates PVC-B as clone of PVC-A
- User deletes PVC-A while clone is in progress
Without Source Volume Lock:
Thread A: CreateVolume(clone from vol-123)
- Locks: "pvc-b-name" (destination)
- Calls RBD CreateVolume (starts cloning)
Thread B: DeleteVolume(vol-123) - runs concurrently
- Locks: "vol-123" (different lock - both threads proceed!)
- Deletes NVMe-oF namespace → success
- Tries to delete RBD image → FAILS (has clones, RBD blocks it)
- Returns error
Result: vol-123 has RBD image but NO namespace - PVC-A cannot be mounted until retry succeeds.
Option 1: Add Source Volume Lock (prevents race)
// In CreateVolume, after extracting clone source:
if vol := contentSource.GetVolume(); vol != nil {
sourceVolumeID := vol.GetVolumeId()
if acquired := cs.volumeLocks.TryAcquire(sourceVolumeID); !acquired {
return nil, status.Errorf(codes.Aborted, "source volume operation in progress")
}
defer cs.volumeLocks.Release(sourceVolumeID)
}
- Pros: No broken state window, cleaner user experience
- Cons: Additional lock complexity, serializes operations
Option 2: Leave As-Is (rely on idempotency)
- DeleteNamespace is idempotent (returns nil on ENODEV)
- DeleteVolume will be retried by Kubernetes
- Eventually succeeds after clones are deleted
- Pros: Simpler code, no additional locks
- Cons: Temporary unpublishable state, more error logs during retries
Both approaches are valid - lock prevents the issue, idempotency recovers from it.
Originally posted by @gadididi in #6277 (comment)
The below comes from a Copilot suggestion, it applies to CephFS and RBD as well.
Race Condition During Concurrent Clone and Delete:
Scenario:
Without Source Volume Lock:
Thread A: CreateVolume(clone from vol-123)
Thread B: DeleteVolume(vol-123) - runs concurrently
Result: vol-123 has RBD image but NO namespace - PVC-A cannot be mounted until retry succeeds.
Option 1: Add Source Volume Lock (prevents race)
Option 2: Leave As-Is (rely on idempotency)
Both approaches are valid - lock prevents the issue, idempotency recovers from it.
Originally posted by @gadididi in #6277 (comment)