fix: NFS error 2370 retry and mutex to prevent race condition on concurrent PVC provisioning#140
Open
paowuh wants to merge 2 commits into
Open
fix: NFS error 2370 retry and mutex to prevent race condition on concurrent PVC provisioning#140paowuh wants to merge 2 commits into
paowuh wants to merge 2 commits into
Conversation
Add NfsSystemBusyError type and exponential backoff retry (11 attempts, ~60s total) when ShareNfsPrivilegeSave returns error 2370 (NFS subsystem temporarily busy), which occurs under concurrent DSM API calls.
Add per-DSM mutex map in nodeServer to prevent concurrent calls to ShareNfsPrivilegeSave on the same DSM, reducing the thundering herd effect during parallel PVC provisioning (e.g. Kasten K10 restores).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Under concurrent PVC provisioning (e.g. during Kasten K10 restores), DSM's NFS subsystem returns error 2370 ("NFS system busy") when
ShareNfsPrivilegeSaveis called multiple times in rapid succession. The original code ignores this error, causing NFS exports to be silently misconfigured.The result is:
Bound(share exists on DSM) but the pod fails with:mount.nfs: failed, reason given by server: No such file or directoryRoot Cause
ShareNfsPrivilegeSaveis called duringNodeStageVolumewith no retry logic.When multiple nodes call it concurrently against the same DSM, the NFS subsystem becomes temporarily busy and returns error 2370. The error was silently ignored, leaving the NFS export unconfigured.
Additionally, there was no serialization between concurrent calls from different goroutines targeting the same DSM instance.
Changes
pkg/utils/error.goNfsSystemBusyErrortyped error for DSM error code 2370pkg/dsm/webapi/share.goShareNfsPrivilegeSave(11 attempts, backoff from 500ms up to 10s cap, ~60s total) when error 2370 is returnedpkg/driver/nodeserver.go+pkg/driver/utils.gonfsPrivMus) tonodeServerShareNfsPrivilegeSavecalls with a per-DSM mutex to serialize concurrent NFS privilege saves on the same DSM, reducing thundering herdDockerfileCOPY synocli ./synocliso the Docker build context includes the synocli source required by the Go module during compilationTesting
Verified in a production on-premise Kubernetes cluster (6 nodes, Synology NAS on Btrfs) during Kasten K10 restore of 10+ PVCs in parallel. NFS mount failures no longer occur after these changes