Support including workspace-root files in subproject sdist without duplication

[trim21]

Summary

In a monorepo/workspace layout, when building sdist from a Python binding subdirectory, tool.maturin.include cannot reference files in parent directories because .. is rejected.

This makes it hard to include required root-level files like LICENSE and rust-toolchain.toml without duplicating them into each binding package directory.

Current behavior

• include entries containing .. are rejected (include/exclude pattern must not contain '..').
• For a project like:

repo/
  LICENSE
  rust-toolchain.toml
  crates/
    ast/
      pyproject.toml
    lsp-py/
      pyproject.toml

we cannot express:

[tool.maturin]
include = [
  { path = "../../LICENSE", format = "sdist" },
  { path = "../../rust-toolchain.toml", format = "sdist" },
]

So users have to:

1. duplicate these files in each binding crate, or
2. add custom copy/sync scripts in CI/build.

Expected behavior

Please provide a supported way for subproject pyproject.toml to include selected workspace-root files into sdist, without file duplication.

Why this matters

• Common Rust+Python monorepo pattern (multiple bindings in crates/*-py).
• Duplicated legal/toolchain files are easy to drift.
• Symlink-based workarounds are fragile, especially on Windows developer environments.

Related issues

• maturin sdist doesn't include LICENSE on root when the root is not current directory #2531 (root LICENSE handling / PEP 639 context)

[trim21]

I'm not sure if this even possible, hope there is a solution

[messense]

I've been looking into this. Allowing .. in include patterns is feasible, but there are several problems that need to be solved:

1. Target path mapping in sdist

When a pattern like ../../LICENSE is used, we need to decide where the file lands inside the sdist archive. The natural answer is to strip the leading .. components and use the remainder: ../../LICENSE → LICENSE, ../../config/foo.toml → config/foo.toml.

2. Building a wheel from the unpacked sdist

This is the trickiest part. When pip install builds a wheel from an sdist, the project root is the unpacked sdist directory. The pattern ../../LICENSE would try to navigate above the temp directory and fail.

A possible solution is fallback resolution: when a .. pattern doesn't resolve (because we're inside an unpacked sdist), automatically fall back to trying the stripped path at the project root. So ../../LICENSE → not found → try LICENSE → found (because the sdist placed it there). This makes a single include entry work transparently in both contexts.

3. Security / path escaping

Currently .. is rejected as a safety measure to prevent accidentally including files far outside the project. If we allow it, we should probably bound it — e.g., refuse to resolve above the Cargo workspace root (which is already known from cargo metadata). This prevents ../../../../etc/passwd scenarios.

4. Interaction with exclude patterns

Exclude patterns are resolved relative to the project directory via OverrideBuilder. If .. is allowed in include but not in exclude, there's no way to selectively exclude files matched by a parent-traversing glob like ../../data/**/*.csv. We'd need to decide whether to also allow .. in exclude, or accept that limitation.

5. Glob expansion with ..

The current glob_files() does root.join(pattern) which handles .. natively via OS path joining. However, this joins to an absolute path first, and glob::Pattern::escape(root) + joining ../../foo could produce unexpected results on some platforms. Needs careful testing on Windows.

6. Duplicate file conflicts

If the sdist places ../../LICENSE as LICENSE at the root, it could collide with an existing LICENSE that's already in the project directory or added by license-files (PEP 639). We need to handle this gracefully — either skip duplicates or error clearly.

---

In summary, the core problems are (2) the sdist→wheel roundtrip and (3) bounding the traversal. The fallback + workspace-root boundary approach seems workable. Thoughts?

[trim21]

In my understanding, that matruin is also building sdist, right?

So there are 2 ways to build wheels:

1. source -> wheel
2. source -> sdist -> wheel

when we are building sdist or building wheels from source directly, we can consider that the parent directive is trusted, and we handle the includings, with something like this:

include = [
  # the dist will collaborate with project.license.file
  { path = "../../LICENSE", dist = "./LICENSE", format = "sdist" },
  { path = "../../rust-toolchain.toml", format = "sdist" },
]

and when building sdist, we strip the include from pyproject.toml., so when builidng wheel from sdist, there is no include involved.

[messense]

Good point about stripping includes from pyproject.toml. Thinking about this more carefully though, I think the real question is whether include is even the right mechanism for this. Like what should we do when:

include = [
  { path = "../../LICENSE", dist = "./LICENSE", format = "sdist" },
  # what should this path be?, ../../LICENSE or ./LICENSE
  { path = "../../LICENSE", format = "wheel" },
]

Cargo already solves this exact problem for package.license-file and package.readme: you can point to ../../LICENSE in Cargo.toml, and cargo package resolves it, copies the file into the package, and rewrites the path to point to the local copy. The sdist is self-contained without any special fallback logic.

The files mentioned in the issue (LICENSE, rust-toolchain.toml) are really build/metadata files that need to be copied into the sdist so it's self-contained, not project files that happen to live elsewhere. include patterns are meant for project-relative content that should appear in the build artifact.

For LICENSE specifically, this is already handled: project.license-files (PEP 639) and package.license-file in Cargo.toml both support paths outside the project and are already processed correctly.

For other files like rust-toolchain.toml, what specific files do you need? It might be better to understand the concrete use cases before deciding on the mechanism — some might already have solutions.

[messense]

To be clear, I'm in favor of lifting this limit as long as we can come up with a proper design that is intuitive and requires less workarounds in maturin code, the sdist creation code is already pretty complex and every time we add new features/enhancements we risk breaking existing functionalities.

I would encourage you to explore this repository with help from AI or not, and try to implement it to explore maybe different solutions.

[trim21]

For other files like rust-toolchain.toml, what specific files do you need? It might be better to understand the concrete use cases before deciding on the mechanism — some might already have solutions.

Thank your explanation about

For now, I want to include rust-toolchain.toml and Cargo.lock and all crates in this workspace into the sdist, so later when I build wheels from sdist I got locked toolchain and packages. In my CI I want to use the path source -> sdist -> wheel to make sure the sdist are buildable and get reproducible builds from sdist.

[messense]

Cargo.lock should have been handled already, for rust-toolchain.toml IMHO you should not include it in sdist, downstream consumers of the sdist should have the freedom to choose which rust version to use, rust-toolchain.toml can cause trouble for distros that build from sdist with managed system global rustc versions.

[trim21]

Cargo.lock should have been handled already, for rust-toolchain.toml IMHO you should not include it in sdist, downstream consumers of the sdist should have the freedom to choose which rust version to use, rust-toolchain.toml can cause trouble for distros that build from sdist with managed system global rustc versions.

IMO a dll/so (the python wheel) is a "application" for a rust project, not "library", the toolchain should be locked in this case, the same reason why we include Cargo.lock in sidst.

[messense]

There's a key difference between the two:

• Cargo.lock locks dependency versions — this ensures reproducible builds without changing behavior. A downstream builder using the lockfile gets the same crate versions but can still compile with whatever rustc they have.
• rust-toolchain.toml locks the compiler itself — this actively overrides the user's rustc. When a distro packager (Fedora, Debian, conda-forge, etc.) builds from sdist, rust-toolchain.toml would force rustup to download and switch to a specific toolchain, potentially conflicting with their system-managed rustc. Some build environments don't even have rustup at all, only a system rustc — rust-toolchain.toml would cause the build to fail.

The analogy would be: Cargo.lock is like package-lock.json (lock deps, use whatever Node version), while rust-toolchain.toml is like forcing a specific Node version via .nvmrc — fine for development, but shouldn't be imposed on downstream builders.