Add TOML v1.1 -> v1.0 backwards compatibility for source distributions#18741
Add TOML v1.1 -> v1.0 backwards compatibility for source distributions#18741
Conversation
konstin
added
the
enhancement
konstin
force-pushed
the
konsti/toml-backwards-compatibility
branch
2 times, most recently
from
f0d7ebe to
d580ff9
Compare
[TOML 1.1](https://github.com/toml-lang/toml/releases/tag/1.1.0) introduces support for new syntax that older tools with TOML 1.0 don't understand. Usually, the user is either in control of which tools need to read the TOML files or the TOML gets converted before publishing (`pyproject.toml` -> `METADATA` for wheels). The specific case where this doesn't work is when a user builds the source distribution of the package with a tool that only support TOML 1.0. Build tools need to parse `pyproject.toml` in source distributions to extract the `[build-system]` table, and if any other part of the file contains TOML 1.1 syntax, they fail to build. This generally doesn't trigger backtracking, so the user is left if a failure when any (transitive) dependency in their dependency tree has started using a single instance of TOML 1.1. Most package managers, including pip, are implemented in Python and use stdlib's tomllib, which only support TOML 1.0 up to including Python 3.14. To work around this, we do a best-effort rewrite of `pyproject.toml` to TOML 1.0 during source distribution builds. This approach is inspired by Cargo, which is successfully rewriting published `Cargo.toml`s for many versions. While the `toml` crate doesn't guarantee this downgrade is always done (toml-rs/toml#1088), this crate is also used by Cargo, and this best effort rewrite is sufficient currently. Similarly following Cargo, we also add a `pyproject.toml.orig` to the source distribution. https://discuss.python.org/t/adopting-toml-1-1/105624 went nowhere, but a best-in-class tool should do this transformation, so we're adding it.
konstin
force-pushed
the
konsti/toml-backwards-compatibility
branch
from
d580ff9 to
9edc16e
Compare
Gankra
left a comment
Gankra
left a comment
There was a problem hiding this comment.
Are there any cases where the prettyification will "arbitrarily" modify a pyproject.toml that isn't using 1.1 features?
| writer.write_file( | ||
| &Path::new(&top_level) | ||
| .join("pyproject.toml.orig") | ||
| .portable_display() | ||
| .to_string(), | ||
| &pyproject_path, | ||
| )?; |
There was a problem hiding this comment.
I'm not totally opposed to just always doing this, but I can't help but wonder if we can detect when this rewrite is necessary and only write a .orig when it is?
| // `pyproject.toml` is handled separately. | ||
| if relative == "pyproject.toml" { | ||
| continue; | ||
| } | ||
| if relative == "pyproject.toml.orig" { | ||
| debug!("Ignoring existing `pyproject.toml.orig`"); | ||
| continue; | ||
| } |
There was a problem hiding this comment.
Slightly interesting that one of these merits a debug while the other doesn't
| assert!( | ||
| contents | ||
| .iter() | ||
| .any(|f| f.contains("nested_pyproject/pyproject.toml")), | ||
| ); |
There was a problem hiding this comment.
For clarity I would appreciate this test asserting that .orig's do or don't exist.
|
For additional context on this approach in Cargo, the .orig files actually primarily exist to handle the fact that Cargo workspaces allow a Cargo.toml to be defined such that things in the parent directory are required to resolve it properly -- things which should not be shipped in the equivalent of an sdist. As such, Cargo has a step where it "resolves" these references in a Cargo.toml to actual concrete values. i.e. This is the kind of thing we might also find desirable to be able to do, so establishing this precedent/idiom now is nice in and of itself. |
|
Also I guess I should raise the question "do we want to ship this under --preview for a version or two, or with some kind of opt-out out of an abundance of caution"? |
2 participants
TOML 1.1 introduces support for new syntax that older tools with only TOML 1.0 support don't understand.
Usually, the user is either in control of which tools need to read the TOML files or the TOML gets converted before publishing (for wheels:
pyproject.toml->METADATA). The specific case where this doesn't work is when a package manager that only support TOML 1.0 tries to build the source distribution of a dependency. Build tools need to parsepyproject.tomlin source distributions to extract the[build-system]table, and if any other part of the file contains TOML 1.1 syntax, they fail to build. This generally doesn't trigger backtracking, so the user is left with a failure when any (transitive) dependency in their dependency tree has started using a single instance of TOML 1.1. Most package managers, including pip, are implemented in Python and use stdlib's tomllib, which only support TOML 1.0 up to including Python 3.14.To work around this, we do a best-effort rewrite of
pyproject.tomlto TOML 1.0 during source distribution builds.This approach is inspired by Cargo, which has been successfully rewriting published
Cargo.tomls for many versions. While thetomlcrate doesn't guarantee this downgrade always works (toml-rs/toml#1088), this crate is also used by Cargo, and this best effort rewrite handles the biggest failure case: Newlines and trailing commas in inline tables. Similarly following Cargo, we also add apyproject.toml.origto the source distribution.https://discuss.python.org/t/adopting-toml-1-1/105624 was inconclusive, but a best-in-class tool should do this transformation.