Skip to content

fix: NFS error 2370 retry and mutex to prevent race condition on concurrent PVC provisioning#140

Open
paowuh wants to merge 2 commits into
SynologyOpenSource:mainfrom
paowuh:fix/nfs-error-2370-retry
Open

fix: NFS error 2370 retry and mutex to prevent race condition on concurrent PVC provisioning#140
paowuh wants to merge 2 commits into
SynologyOpenSource:mainfrom
paowuh:fix/nfs-error-2370-retry

Conversation

@paowuh

@paowuh paowuh commented May 11, 2026

Copy link
Copy Markdown

Problem

Under concurrent PVC provisioning (e.g. during Kasten K10 restores), DSM's NFS subsystem returns error 2370 ("NFS system busy") when ShareNfsPrivilegeSave is called multiple times in rapid succession. The original code ignores this error, causing NFS exports to be silently misconfigured.

The result is:

  • PVC goes Bound (share exists on DSM) but the pod fails with: mount.nfs: failed, reason given by server: No such file or directory
  • The share exists as a folder on DSM but is not exported via NFS
  • The issue is amplified during parallel PVC creation (5+ PVCs simultaneously)

Root Cause

ShareNfsPrivilegeSave is called during NodeStageVolume with no retry logic.
When multiple nodes call it concurrently against the same DSM, the NFS subsystem becomes temporarily busy and returns error 2370. The error was silently ignored, leaving the NFS export unconfigured.

Additionally, there was no serialization between concurrent calls from different goroutines targeting the same DSM instance.

Changes

pkg/utils/error.go

  • Added NfsSystemBusyError typed error for DSM error code 2370

pkg/dsm/webapi/share.go

  • Added exponential backoff retry in ShareNfsPrivilegeSave (11 attempts, backoff from 500ms up to 10s cap, ~60s total) when error 2370 is returned

pkg/driver/nodeserver.go + pkg/driver/utils.go

  • Added per-DSM mutex map (nfsPrivMus) to nodeServer
  • Wrapped ShareNfsPrivilegeSave calls with a per-DSM mutex to serialize concurrent NFS privilege saves on the same DSM, reducing thundering herd

Dockerfile

  • Added COPY synocli ./synocli so the Docker build context includes the synocli source required by the Go module during compilation

Testing

Verified in a production on-premise Kubernetes cluster (6 nodes, Synology NAS on Btrfs) during Kasten K10 restore of 10+ PVCs in parallel. NFS mount failures no longer occur after these changes

paowuh added 2 commits May 11, 2026 14:18
Add NfsSystemBusyError type and exponential backoff retry (11 attempts, ~60s total) when ShareNfsPrivilegeSave returns error 2370 (NFS subsystem temporarily busy), which occurs under concurrent DSM API calls.
Add per-DSM mutex map in nodeServer to prevent concurrent calls to ShareNfsPrivilegeSave on the same DSM, reducing the thundering herd effect during parallel PVC provisioning (e.g. Kasten K10 restores).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant