Lowercase ASCII labels in the stdlib IDNA host fallback#1780
Draft
gaoflow wants to merge 1 commit into
Draft
Conversation
The stdlib IDNA-2003 codec leaves already-ASCII labels untouched, so a host that mixes a non-ASCII label with an ASCII label the strict idna package rejects kept the ASCII label in its original case. That made raw_host and the rendered URL differ when the URL was parsed a second time. Lowercasing the host before the fallback encode matches the ASCII fast path and the idna.encode(uts46=True) result.
psf-chronographer
Bot
added
the
bot:chronographer:provided
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #1780 +/- ##
==========================================
+ Coverage 99.86% 99.90% +0.04%
==========================================
Files 21 21
Lines 4310 4315 +5
Branches 252 252
==========================================
+ Hits 4304 4311 +7
+ Misses 4 2 -2
Partials 2 2
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What do these changes do?
_idna_encode()falls back to the stdlib"idna"codec when the strictidnapackage rejects a host (for example a host with an underscore DNS label such as_dmarc). The stdlib codec leaves already-ASCII labels untouched, so those labels kept whatever case they were given while the non-ASCII labels were still normalised. The host is now lowercased before the fallback encode, matching both the ASCII fast path and theidna.encode(..., uts46=True)result used on the success path.Are there changes in behavior for the user?
Yes. A host that mixes a non-ASCII label with an ASCII label the strict
idnapackage rejects now has its ASCII labels lowercased, soraw_hostis canonical and the URL round-trips. For exampleURL("http://_DMARC.café/").raw_hostis now_dmarc.xn--caf-dmainstead of_DMARC.xn--caf-dma; the old value changed again on a second parse, so the URL was not idempotent.Is it a substantial burden for the maintainers to support this?
No. The change is a single
.lower()on the fallback path, plus a regression test.Related issue number
No linked issue; found while checking host normalisation idempotence.
Checklist
CONTRIBUTORS.txt(N/A: no such file in the repo)CHANGES/folderAgent run details (optional, for reviewers)
Repro (on
master):Tests: the new
test_idna_fallback_lowercases_ascii_labelsfails onmasterand passes with the fix. Full suite: 1448 passed, 102 skipped, 4 xfailed.Lint:
ruff check yarl/_url.py tests/test_url.pyclean.Spelling: the
CHANGES/fragment words were checked with enchant againsten_USplusdocs/spelling_wordlist.txt; a fullmake doc-spellingdocs build was not run in this environment.Drafted with Claude Code (Claude Opus 4.8); pending review by
@gaoflowbefore this is taken out of draft.