Define and align symlink semantics across local, SFTP, and SFTP2

[bbtfr]

Context

While reviewing the SFTP/SFTP2 stat-reuse PR, we rechecked symlink semantics across local, SFTP, and SFTP2. Some small bugs are being handled in the stat-reuse PR, but broader symlink behavior is inconsistent and should be discussed/fixed separately.

Expected direction from the discussion:

• A user-provided root path that is a symlink to a directory should generally be resolved as the operation root for listing/traversal APIs.
• followlinks should primarily control whether traversal follows child symlink directories encountered inside a tree.
• is_file(followlinks=False) should treat symlinks as file-like.
• open(link_to_file) should read the target content; open(link_to_dir) should raise IsADirectoryError.
• unlink is file-only: directories should raise IsADirectoryError, while symlinks themselves can be unlinked.
• Copy/sync/upload/download/rename symlink behavior needs a documented policy before changing defaults.

Remaining Issues

Local root symlink traversal

FSPath.scan() / smart_scan() and FSPath.walk() / smart_walk() do not match the expected root symlink-to-directory behavior:

• scan(link_to_dir) currently treats the root symlink as a file-like path and yields the link itself.
• walk(link_to_dir) currently returns empty when followlinks=False.
• scandir(link_to_dir) and glob(link_to_dir/*) already expand the root symlink.

This makes local inconsistent with SFTP/SFTP2 and with the intended root-path semantics.

SFTP/SFTP2 copy(followlinks=False) on symlink

SFTP/SFTP2 currently materialize the symlink target content instead of preserving the symlink object. This differs from local FSPath.copy(..., followlinks=False), which delegates to shutil.copy2(..., follow_symlinks=False).

This is a behavior change if fixed, so it needs an explicit compatibility decision.

SFTP/SFTP2 download(followlinks=False) on symlink

SFTP download currently reads through backend file operations and materializes target content instead of creating a local symlink. SFTP2 likely has the same class of behavior, but coverage should be checked.

This differs from a strict “preserve link object when followlinks=False” interpretation.

SFTP/SFTP2 upload(followlinks=False) from local symlink

Upload from a local symlink generally materializes the target content. If followlinks=False is meant to preserve symlink objects, upload needs a different behavior or explicit documentation.

SFTP/SFTP2 sync(followlinks=True) traversal

sync(followlinks=True) passes followlinks to leaf copy(), but _sftp_scan_pairs() / _sftp2_scan_pairs() do not appear to use the same traversal flag. That means followlinks=True does not fully control the recursive copy set.

Fixing this may change which files are copied, so it should be handled carefully.

Cross-backend rename() on symlink

Cross-backend SFTP/SFTP2 rename on a symlink currently goes through generic copy/open behavior, materializes the target content at the destination, and removes the source link. This is surprising compared with same-backend/native rename, which moves the symlink object.

Changing this would alter default behavior. Possible policies include:

• preserve current materialization behavior and document it,
• reject cross-backend symlink rename,
• preserve symlink objects when the destination backend can represent them.

md5(followlinks=...)

md5(followlinks=...) does not consistently honor the followlinks flag for symlink-vs-directory decisions across local/SFTP/SFTP2. The expected behavior probably should be:

• symlink to file: hash target content when following links,
• symlink to directory: either follow and apply directory hash semantics, or reject/document explicitly,
• followlinks=False: define whether the link object itself is hashable or should raise.

Suggested Next Step

Handle this as a dedicated symlink semantics topic instead of mixing it into stat optimization PRs:

1. Write the target behavior matrix for local, SFTP, and SFTP2.
2. Add tests around root symlink traversal, child symlink traversal, copy/upload/download/sync, rename, unlink/remove, and md5.
3. Split fixes into default-preserving bug fixes vs compatibility-sensitive behavior changes.

[bbtfr]

Proposed unified symlink semantics for local / SFTP / SFTP2.

Core split: input path resolution and recursive traversal/copy behavior are separate concepts. Each operation family is described as one unit below; followlinks=True/False, follow_symlinks=True/False, and file-only vs recursive variants are options of the same operation family.

1. stat(path, follow_symlinks=True) / lstat(path)

   • stat(..., follow_symlinks=True) follows symlinks and returns the target object's stat.
   • stat(..., follow_symlinks=False) returns the link object's stat.
   • lstat(path) is equivalent to stat(path, follow_symlinks=False).

2. exists(path, followlinks=False)

   • followlinks=False: lstat-style existence. A symlink object exists even if its target is missing.
   • followlinks=True: target-style existence. A broken symlink does not exist.

3. is_file(path, followlinks=False) / is_dir(path, followlinks=False)

   • followlinks=False: classify symlinks as file-like objects. So is_file(link) is True, including link_to_dir; is_dir(link_to_dir) is False.
   • followlinks=True: classify by the target object. So is_dir(link_to_dir) is True, and is_file(link_to_file) is True.

4. open(path)

   • Always follows the input symlink.
   • open(file) opens the file.
   • open(directory) raises IsADirectoryError.
   • open(symlink_to_file) opens the target file.
   • open(symlink_to_dir) raises IsADirectoryError.
   • Do not add a separate followlinks semantic to open.

5. scan(path, followlinks=False) / scan_stat(path, followlinks=False) / scandir(path) / glob(path) / walk(path, followlinks=False)

   • Root path semantics: if the user-provided root path is a symlink to a directory, resolve it as the operation root. Returned paths should stay under the user-provided root shape, e.g. scan(link_to_dir) yields link_to_dir/file.
   • Child traversal semantics with followlinks=False: child symlink-to-directory entries are not recursively entered and are treated as file-like entries.
   • Child traversal semantics with followlinks=True: child symlink-to-directory entries are recursively entered.
   • This child traversal behavior is the main meaning of followlinks for traversal APIs.

6. Delete family: unlink(path, missing_ok=False) / remove(path, missing_ok=False)

   • Model: unlink is the file-only form; remove is the recursive/file-or-directory form.
   • Regular file: both unlink and remove delete the file.
   • Regular directory: unlink raises IsADirectoryError; remove recursively deletes the directory tree.
   • Symlink-to-file: both delete the symlink object itself; target file remains.
   • Symlink-to-directory: both delete the symlink object itself; target directory remains and is not traversed.
   • Symlink inside a directory being removed recursively: delete the symlink object as a child entry; do not recurse into the symlink target.
   • Missing path behavior follows missing_ok.

7. Rename family: rename(path, dst, recursive=True)

   • Model: recursive=False is the file-only form; recursive=True is the recursive/file-or-directory form.
   • Regular file: move the file to dst.
   • Regular directory with recursive=False: raise IsADirectoryError.
   • Regular directory with recursive=True: move the directory tree.
   • Symlink-to-file: move the symlink object itself; target file remains at its original location.
   • Symlink-to-directory: move the symlink object itself; target directory remains at its original location and is not traversed.
   • Symlink inside a directory being recursively renamed: preserve/move the symlink object as an entry in the moved directory tree; do not materialize or traverse its target just because the parent directory is being moved.
   • Same-backend/native rename should follow the above object-moving semantics.
   • Cross-backend symlink rename should not silently materialize target content. Preferred policy: preserve the link object if the destination backend supports symlinks; otherwise raise an unsupported-operation error. This is compatibility-sensitive and should be handled separately.

8. Copy family: copy(path, dst, followlinks=False) / sync(path, dst, followlinks=False) / upload(path, dst, followlinks=False) / download(path, dst, followlinks=False)

   • Model: copy is the file-only form; sync is the recursive/file-or-directory form. This mirrors rename(recursive=False/True) and unlink/remove.
   • upload / download are cross-backend local-remote specializations of the same copy family. File-like upload/download should behave like copy; recursive upload/download should behave like sync if such recursive mode exists.
   • Regular file: copy/upload/download/sync copy file content to dst.
   • Regular directory: copy and file-like upload/download raise IsADirectoryError; sync and recursive upload/download recursively copy/sync the directory tree.
   • Symlink-to-file with followlinks=False: preserve/copy the symlink object as dst; target file is not materialized. If the destination backend cannot represent symlinks, raise an unsupported-operation error.
   • Symlink-to-file with followlinks=True: copy target file content to dst.
   • Symlink-to-directory with followlinks=False: preserve/copy the symlink object as dst; target directory is not traversed. If the destination backend cannot represent symlinks, raise an unsupported-operation error.
   • Symlink-to-directory with followlinks=True: file-only forms (copy and file-like upload/download) raise IsADirectoryError; recursive forms (sync and recursive upload/download) recursively copy/sync the target directory content under dst.
   • Symlink-to-file inside a directory being synced/copied recursively with followlinks=False: preserve/copy the symlink object in the destination tree.
   • Symlink-to-file inside a directory being synced/copied recursively with followlinks=True: materialize target file content in the destination tree.
   • Symlink-to-directory inside a directory being synced/copied recursively with followlinks=False: preserve/copy the symlink object and do not recurse into target directory.
   • Symlink-to-directory inside a directory being synced/copied recursively with followlinks=True: recurse into the target directory and materialize its tree in the destination tree.
   • If destination backend cannot represent symlinks while followlinks=False, fail explicitly rather than creating broken links or silently materializing targets.

9. md5(path, followlinks=False)

   • followlinks=True: symlink-to-file hashes target file content. Symlink-to-directory follows directory md5 semantics if directory md5 is supported; otherwise raise the same directory error as the backend normally would.
   • followlinks=False: prefer raising an unsupported-operation error for symlink objects, because link objects do not have a stable cross-backend content hash.

[bbtfr]

Current behavior matrix against the proposed definition.

Legend:

• ✅ matches the proposed definition.
• ❌ does not match; the cell explains the current behavior.
• N/A means the reference API does not provide that operation family.

Assumptions:

• SFTP/SFTP2 behavior is evaluated as if PR perf: reduce SFTP stat calls #659 is merged.
• S3 is intentionally out of scope for this issue.
• WebDAV/HDFS/HTTP are out of scope because they do not expose real symlink semantics.
• os / shutil / pathlib.Path are reference behaviors, not APIs with the same complete operation set as megfile.

Operation family / case	Local FSPath current	SFTP/SFTP2 current after #659	os / shutil reference	pathlib.Path reference
stat(..., follow_symlinks=True) on symlink	✅	✅	✅	✅
lstat() / stat(..., follow_symlinks=False) on symlink	✅	✅	✅	✅
exists(link, followlinks=False), including broken symlink	✅	✅	✅ via os.path.lexists; os.path.exists follows target	❌ Path.exists() follows target and returns false for broken symlink
exists(link, followlinks=True)	✅	✅	✅	✅
is_file(link, followlinks=False)	✅	✅	❌ os.path.isfile() follows target; link-to-dir is false	❌ Path.is_file() follows target; link-to-dir is false
is_dir(link_to_dir, followlinks=False)	✅	✅	❌ os.path.isdir() follows target and returns true	❌ Path.is_dir() follows target and returns true
is_file/is_dir(..., followlinks=True)	✅	✅	✅	✅
open(link_to_file)	✅	✅	✅	✅
open(link_to_dir)	✅	✅	✅	✅
Traversal root symlink-to-dir: scandir / glob / closest equivalents	✅	✅	✅	✅
Traversal root symlink-to-dir: scan / scan_stat / walk / closest recursive equivalents	❌ FSPath.scan(linkdir) yields the link itself; FSPath.walk(linkdir) returns empty with default followlinks=False	✅	✅ os.walk(linkdir, followlinks=False) expands the root symlink	✅ closest Path(linkdir).rglob('*') expands root
Traversal child symlink-to-dir with followlinks=False	✅	✅	Mostly ✅: os.walk does not recurse, but lists child symlink dirs in dirs rather than file-like entries	Mostly ✅: Path.rglob('*') lists child symlink dir but does not recurse
Traversal child symlink-to-dir with followlinks=True	✅	✅	✅	N/A: no equivalent follow flag for Path.rglob()
Delete family: regular file via unlink / remove	✅	✅	✅	✅ for Path.unlink; no recursive Path.remove
Delete family: regular directory via unlink / remove	✅	✅	✅	✅ for Path.unlink / rmdir; no recursive Path.remove
Delete family: symlink-to-file via unlink / remove	✅	✅	✅	✅
Delete family: symlink-to-dir via unlink / remove	✅	✅	✅ os.unlink/remove delete link object; shutil.rmtree(linkdir) refuses symlink-to-dir	✅ Path.unlink deletes link object
Delete family: symlink inside recursively removed directory	✅	✅	✅ shutil.rmtree removes link entry and does not recurse into target	N/A
Rename family: regular file	✅	✅	✅	✅
Rename family: regular directory with recursive/file-or-dir form	✅ current default supports directory rename/move	✅ current default supports directory rename/move	✅	✅
Rename family: regular directory with file-only form	N/A until recursive flag PR #656	N/A until recursive flag PR #656	N/A	N/A
Rename family: symlink-to-file same-backend/native	✅	✅	✅	✅
Rename family: symlink-to-dir same-backend/native	❌ FSPath.rename(link_to_dir) currently treats it like a directory copy/move and can leave the source link behind	✅ after #659 same-backend native rename moves the link object	✅	✅
Rename family: symlink inside recursively renamed directory	Needs explicit tests	Needs explicit tests	✅ native directory rename preserves child symlink object	✅ native directory rename preserves child symlink object
Rename family: cross-backend symlink	N/A for pure local same-backend	❌ currently materializes target content via generic copy/open and removes source link	N/A	N/A
Copy family file-only: regular file via copy / file-like upload/download	✅	✅	✅ shutil.copy2	N/A
Copy family recursive: regular directory via sync / recursive upload/download	✅ for local sync generally; detailed symlink cases below	✅ for ordinary directories; detailed symlink cases below	✅ via shutil.copytree	N/A
Copy family file-only: regular directory via copy / file-like upload/download	✅ raises directory error	✅ raises directory error	✅ shutil.copy2(dir) raises directory error	N/A
Copy family: symlink-to-file root with followlinks=False	✅ copy preserves symlink object	❌ currently materializes target content instead of preserving link object	✅ shutil.copy2(..., follow_symlinks=False) preserves link object	N/A
Copy family: symlink-to-file root with followlinks=True	✅	✅	✅	N/A
Copy family: symlink-to-dir root with followlinks=False	Mostly ✅ for copy; needs explicit sync tests	❌ currently does not reliably preserve link object; may raise directory error or materialize depending path/API	✅ for shutil.copy2(..., follow_symlinks=False); copytree policy depends on symlinks option	N/A
Copy family file-only: symlink-to-dir root with followlinks=True	✅ raises directory error	✅ raises directory error	✅ shutil.copy2(..., follow_symlinks=True) raises directory error	N/A
Copy family recursive: symlink-to-dir root with followlinks=True	Needs explicit tests	❌ traversal/copy set is not fully wired through followlinks; see sync row below	✅ copytree(..., symlinks=False) can materialize, but exact policy differs	N/A
Copy family recursive: child symlink with followlinks=False	❌ likely not aligned: local sync uses shutil.copytree without explicit symlink preservation policy	❌ child symlinks are copied through leaf copy, which generally materializes target content instead of preserving link objects	✅ only with shutil.copytree(..., symlinks=True); default materializes	N/A
Copy family recursive: child symlink-to-file with followlinks=True	Needs explicit tests	Likely ✅ for leaf copy materialization	✅ default copytree(..., symlinks=False) materializes	N/A
Copy family recursive: child symlink-to-dir with followlinks=True	Needs explicit tests	❌ sync(followlinks=True) passes followlinks to leaf copy, but scan-pair traversal does not fully use the same follow flag	✅ default copytree(..., symlinks=False) follows/materializes directory symlink content depending platform/options	N/A
md5(link_to_file, followlinks=True)	✅	✅	✅ common md5sum link hashes target content	N/A
md5(link_to_dir, followlinks=True)	❌ currently raises directory-open error instead of applying directory md5 semantics	❌ open(link_to_dir) raises IsADirectoryError, so directory md5 semantics are not applied	N/A	N/A
md5(link, followlinks=False)	❌ currently hashes target content for link-to-file instead of rejecting link-object hash	❌ currently hashes/materializes target content for link-to-file instead of rejecting link-object hash	N/A	N/A

Main current gaps to fix or decide:

1. Local scan() / walk() root symlink-to-directory behavior is inconsistent with the proposed root-path semantics and with os.walk().
2. Local rename(link_to_dir) is surprising and does not behave like native os.rename() / Path.rename().
3. SFTP/SFTP2 cross-backend rename(link) still materializes target content.
4. SFTP/SFTP2 copy-family operations with followlinks=False do not preserve symlink objects.
5. Recursive copy-family operations need traversal-level followlinks support, not only leaf-copy support.
6. md5(followlinks=...) needs a clear symlink policy and tests.

[LoveEatCandy]

状态与存在性 (stat/exists/is_file/is_dir)

• followlinks=False 时应将软链接视为文件本身（即使指向目录），软链接对应的文件不存在也应视为存在
• followlinks=True 时则指向目标对象
• 默认 AUTO，除 s3 之外都是 followlinks=True

文件打开 (open)

• 始终跟随软链接。如果指向目录，则抛出 IsADirectoryError
• s3 不跟随软连接

路径遍历 (scan/walk)

• 如果根路径是软链接目录，应直接解析为源目录
• followlinks 专门用于控制是否递归进入子目录中的软链接
• 没有 followlinks 时，软链接目录应被视为文件而不递归进入
• 默认 AUTO，除 s3 之外都是 followlinks=True

删除与重命名 (unlink/remove/rename)

• 对软链接本身进行操作（删除/移动链接对象），而不应影响或递归遍历其指向的目标
• 跨后端重命名若不支持链接保留，则应报错而非隐式实体化（Materialize）内容

复制家族 (copy/sync/upload/download)

• copy/sync 默认 AUTO
  • src 和 dst 同 endpoint 时，followlinks=False
  • src 和 dst 不同 endpoint 时，followlinks=True
• upload/download 不需要 followlinks 参数，始终 followlinks=True

哈希 (md5)

• 不做变化，还是默认 followlinks=False，s3 默认 recalculate = False 的时候用 etag 计算

[bbtfr]

一些细节需要澄清并确认一下：

1. 路径遍历：

   1. 对于 local/sftp，对于根路径是软链接目录，总是解析为源目录，followlinks 只控制目录里的软连接是否解析：行为和 os.walk 的 followlinks 保持一致，followlinks 默认是 True
   2. 对于 s3，对于根路径是软链接目录，followlinks 同时控制软连接根目录是否解析，以及目录里的软连接是否解析：s3 软连接是客户端实现，这样做虽然行为和 local/sftp 不一致但有利于性能，followlinks 默认是 False
   3. 这一点目前开发者内部已经达成一致

2. 复制家族：followlinks 实际控制对 src 的遍历行为，参考路径遍历；需要澄清并确认一下

   1. 拷贝 link 具体是什么行为，可能的行为有：
      1. 方案一：不改变 link 指向的内容直接拷贝：源 link 可能是相对路径或绝对路径，不做修改的拷贝在某些场景下都可能会有问题，但正常 link 拷贝也可能出问题，所以认为符合预期
      2. 方案二：对 link 内容做一些修改，使它尽量可用？但修改会变得比较复杂，涉及到原链接是相对路径或绝对路径，以及目标位置是否跨 endpoint（跨 endpoint 绝对路径很可能会变得不可能，比如本地的绝对路径变成 s3 上的什么东西）
      3. 我支持方案一，但这样做已知会产生一些不可用的链接
   2. 拷贝链接很可能出现失败：1. 上面说的链接不可用的时候是否报错；2. 目标 endpoint 不支持链接（例如 webdav 等），这时是否报错，但报错都发生在拷贝进行了一段时间之后，这时报错对用户也会比较棘手，一般 cli 似乎都是 warning 而不是中途报错？
      1. 我支持 SDK 给参数控制拷贝过程中出现问题是否报错，默认 True，但 CLI 是否应该默认只打 warning ？主要问题是报错在 SDK 听起来比较合理，但用户总需要一个知道问题但也需要把能拷贝的拷过去的开关，而且仿照其他 CLI 的做法，在 CLI 上最好默认 warning？现在的 force 似乎可以承担这个职责？

3. 重命名：

   1. move 是否要给 followlinks 参数，现在没给，但这是否意味着想要 followlinks 的 move 一定要 cp + rm
      1. 我支持保持现状，不给 followlinks 参数，这里已经挺复杂了，mv 一半出问题的断点恢复比 cp 麻烦很多，不在这里加戏了
   2. move 如何处理目标 endpoint 不支持链接时，源路径是个链接：报错？warning？但一个目录拷一半出现一个链接然后报错，怎么断点恢复呢？也要给他加开关控制是否忽略 mv 过程中的报错吗？
      1. 我支持跟拷贝一样用开关控制是否报错

4. 哈希：

   1. 对于 followlinks=True，当前 megfile 的行为没有问题
   2. 对于 followlinks=False，如果输入路径是个链接，或者目录里有链接，如何处理？readlink 然后参与哈希？这里目前实现好像没有明确定义；我猜可以定义成 readlink 参与哈希