Add --exclude-newer-last-modified to support non-PyPI indexes via Last-Modified HTTP header#18823
Conversation
…allback When `--exclude-newer` is set and a package file lacks the PEP 691 `upload-time` field (common with non-PyPI indexes like Artifactory), this opt-in flag makes a HEAD request to the file URL and uses the `Last-Modified` header as a fallback timestamp. HEAD requests are only issued for files the resolver actually selects as candidates (typically 1-2 per package), not for all files on the index page. Closes astral-sh#12449 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
Interesting idea. I'll need to look closer to know if it makes sense. On a high-level, if we support something like this, we might want to do it via an annotation on the index? e.g., [[tool.uv.index]]
...
exclude-newer-source = "last-modified | upload-time" |
- Fix `unparseable` → `unparsable` typo in doc comment - Add `exclude-newer-last-modified` to unknown-field error snapshot - Add `exclude_newer_last_modified` field to all show_settings snapshots Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
yes, there was another idea floating around to add per-index support of
as for the the configuration - flag-enabled fallback or explicit Just let me know your preference here so I can rework (if needed). |
3 participants
Summary
A follow-up on the discussion in #12449
Because of the recent surge of the supply chain attack, my company decided to enable guardrails against adding recently released dependencies - so basically leverage the
exclude-newerfeature ofuv.During our testing of the feature this worked like charm with PyPI, however JFrog's Artifactory - the intra-company index that we are using - seemingly lacks support of PEP 691 and PEP 700 entirely, which essentially prevents us from using
exclude-newer. It just crunches through every version of the first dependency it grabs, fails to understand what was the upload date and fails:So far my research shows that PyPI is pretty much the only index that currently supports PEP 691 / 700.
On the other hand, on example with Artifactory at least, it seems that we can rely on
Last-ModifiedHTTP header value to substitute absentupload-time.The testing in my specific case on example of
setuptools:This made me think of #12449 (comment) - basically implement an optional fallback for the situation when the index is not PEP 691/700 compliant, and basically rely on
Last-ModifiedHTTP header value as the next best thing.Unfortunate disadvantage here is that you need to do HEAD request for every version candidate during resolution phase.
And so I've tried to minimize the overhead by doing these requests only when resolver actually selects the candidates, and only when the
exclude-newerset andexclude-newer-last-modifiedswitched on.The idea is that the code path is simply not active for the PyPI users, but kicks in for the poor souls that need to use something else, and I think the overhead of additional requests is a fine trade-off between having security this feature at all or not having it, and thus acceptable.
DISCLAIMER: the code below was written entirely by Claude Opus 4.6. I have steered it conceptually throughout code creation, but I haven't worked with Rust before, and am not familiar with
uvcodebase in general. I did my best to review the code below using my overall engineering experience, but IDK if I am doing here anything Rust-specific or uv-specific dumb.Configuration:
--exclude-newer-last-modified/--no-exclude-newer-last-modified(wasn't sure about--no-flag, but rest ofuvseem to follow this pattern)UV_EXCLUDE_NEWER_LAST_MODIFIEDexclude-newer-last-modified = true(in[tool.uv],[tool.uv.pip])Test Plan
Tested with the dev build in my specific case.
(Unfortunately couldn't find any public Artifactory that mirrors PyPI, so did the testing with our privately deployed one)
No flag:
With flag, but non-compliant dependencies (notice it mentions the exclude criteria
Last-Modifiedheader):With flag, compliant deps, happy path:
Integration tests
Added