Skip to content

Percent-decode username and password parsed from URLs (#1871) - #2114

Open
mokashang wants to merge 1 commit into
fsspec:masterfrom
mokashang:fix/unquote-url-userinfo
Open

Percent-decode username and password parsed from URLs (#1871)#2114
mokashang wants to merge 1 commit into
fsspec:masterfrom
mokashang:fix/unquote-url-userinfo

Conversation

@mokashang

Copy link
Copy Markdown
Contributor

Fixes #1871

Problem

infer_storage_options returned parsed_path.username and
parsed_path.password verbatim from urllib.parse.urlsplit, which does not
decode percent-encoded characters in the userinfo component. When a user is
forced to percent-encode a reserved character in the URL — the classic case
being # in a password, which would otherwise be parsed as the fragment
delimiter — the backend receives the literal encoded string:

>>> fsspec.open_files("sftp://user:pass%23with%23hash@host/path")
# password reaching paramiko: "pass%23with%23hash"  (auth fails)

The two reports on #1871 both hit this from the FTP/SFTP side, and every
infer_storage_options consumer that maps username/password straight into
its backend client — FTPFileSystem, SFTPFileSystem, SMBFileSystem,
WebHDFS, arrow — has the same failure mode.

Change

Percent-decode both fields with urllib.parse.unquote before returning them:

-        if parsed_path.username:
-            options["username"] = parsed_path.username
-        if parsed_path.password:
-            options["password"] = parsed_path.password
+        if parsed_path.username:
+            options["username"] = unquote(parsed_path.username)
+        if parsed_path.password:
+            options["password"] = unquote(parsed_path.password)

unquote returns its input unchanged when there is nothing to decode, so
URLs with plain-ASCII credentials (which every existing test uses) are
unaffected. The HTTP/HTTPS branch short-circuits at line 87 and never reaches
this code, so requests continues to handle its own URL parsing as noted in
the comment.

Regarding @martindurant's caution on the issue ("SSH would not expect to
have encoded strings"): SSH/SFTP/FTP servers do not implement URL decoding
themselves — they compare raw credentials — so decoding on our side is what
lets an encoded URL work at all. The @Jeansidharta reproducer is exactly this
case: paramiko is handed pass%23with%23hash%23char and rejects it.

Test plan

  • Added test_infer_options_percent_encoded_userinfo covering both the
    encoded case (user%40corp / p%23ass%2Fword%20!) and the unencoded
    passthrough case.
  • pytest fsspec/tests/test_utils.py — 99 passed.
  • pytest fsspec/tests/ fsspec/implementations/tests/ with backend suites
    that need network/cloud services excluded — 1553 passed, 176 skipped,
    2 xfailed.
  • ruff check / ruff format --check clean on the touched files.
  • Changelog entry added under the Dev section.

`infer_storage_options` was returning `parsed_path.username` and
`parsed_path.password` verbatim from `urllib.parse.urlsplit`, which does
not decode percent-encoded characters in the userinfo component.  When a
user was forced to percent-encode a reserved character in the URL —
e.g. `sftp://user:pass%23with%23hash@host/path` to keep `#` out of the
fragment — the FTP/SFTP/SMB backends then received the literal
`pass%23with%23hash` and authentication failed.

Decode both fields with `urllib.parse.unquote` before returning them so
that backends receive the real credentials.  URLs without percent-
encoded userinfo are unaffected (`unquote` returns the input unchanged
when there is nothing to decode), and the HTTP/HTTPS branch short-
circuits before this code path, so its behavior does not change.

Adds a regression test covering both the encoded and plain forms.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Unquote Username and Password

1 participant