Skip to content

Commit 5593228

Browse files
DavisVaughanDavisVaughan
authored
Add PyPI distribution workflows (#467)
* Add PyPI distribution workflows * More docs * `cargo dist generate` * Split apart `uv tool install` vs `uv tool run` advice Now that I understand them more * Add wheel documentation * One liner * CHANGELOG
1 parent fc49102 commit 5593228

15 files changed

Lines changed: 358 additions & 10 deletions

File tree

‎.claude/.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
settings.local.json

‎.claude/settings.json‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
{
2+
"$schema": "https://json.schemastore.org/claude-code-settings.json",
3+
"permissions": {
4+
"defaultMode": "acceptEdits",
5+
"allow": [
6+
"Bash(cat:*)",
7+
"Bash(find:*)",
8+
"Bash(gh issue list:*)",
9+
"Bash(gh issue view:*)",
10+
"Bash(gh pr diff:*)",
11+
"Bash(gh pr view:*)",
12+
"Bash(git checkout:*)",
13+
"Bash(git grep:*)",
14+
"Bash(grep:*)",
15+
"Bash(ls:*)",
16+
"Bash(rm:*)",
17+
"Bash(sed:*)",
18+
"WebFetch(domain:github.com)",
19+
"WebFetch(domain:raw.githubusercontent.com)"
20+
]
21+
}
22+
}

‎.github/workflows/build-wheels.yml‎

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
# Build Python wheels for PyPI
2+
#
3+
# Assumed to run as a subworkflow of `.github/workflows/release.yml`.
4+
# Specifically, as a `local-artifacts-jobs` step within `cargo-dist`.
5+
name: "Build Wheels"
6+
7+
on:
8+
workflow_call:
9+
10+
jobs:
11+
build:
12+
name: "Build Wheels (${{ matrix.this.target }})"
13+
runs-on: ${{ matrix.this.runner }}
14+
timeout-minutes: 30
15+
16+
defaults:
17+
run:
18+
shell: bash
19+
20+
strategy:
21+
matrix:
22+
this:
23+
# Platform tags are based on https://pypi.org/project/ruff/#files
24+
# - macosx_10_12 matches Rust's default support
25+
# - macosx_11_0 is the minimum for ARM Mac
26+
# - manylinux_2_28 is what we build the binaries in for Linux
27+
- target: x86_64-apple-darwin
28+
platform_tag: macosx_10_12_x86_64
29+
runner: macos-15-intel
30+
- target: aarch64-apple-darwin
31+
platform_tag: macosx_11_0_arm64
32+
runner: macos-14
33+
- target: x86_64-unknown-linux-gnu
34+
platform_tag: manylinux_2_28_x86_64
35+
runner: ubuntu-24.04
36+
- target: aarch64-unknown-linux-gnu
37+
platform_tag: manylinux_2_28_aarch64
38+
runner: ubuntu-24.04-arm
39+
- target: x86_64-pc-windows-msvc
40+
platform_tag: win_amd64
41+
runner: windows-2022
42+
- target: aarch64-pc-windows-msvc
43+
platform_tag: win_arm64
44+
runner: windows-11-arm
45+
46+
steps:
47+
- uses: actions/checkout@v4
48+
49+
- uses: astral-sh/setup-uv@v7
50+
51+
- name: "Download artifact"
52+
uses: actions/download-artifact@v4
53+
with:
54+
name: artifacts-${{ matrix.this.target }}
55+
path: artifacts
56+
57+
- name: "Extract binary"
58+
run: |
59+
TARGET=${{ matrix.this.target }}
60+
61+
# uv knows to copy `scripts/` to `.data/scripts/` in the wheel
62+
mkdir -p python/scripts
63+
64+
if [[ "$TARGET" == *windows* ]]; then
65+
unzip artifacts/air-$TARGET.zip
66+
cp air-$TARGET/air.exe python/scripts/air.exe
67+
else
68+
tar xzf artifacts/air-$TARGET.tar.gz
69+
cp air-$TARGET/air python/scripts/air
70+
chmod +x python/scripts/air
71+
fi
72+
73+
- name: "Build wheel"
74+
# Builds the wheel as generic `air_formatter-{air-version}-py3-none-any.whl`
75+
run: uv build --wheel --out-dir wheel/ python/
76+
77+
- name: "Retag wheel"
78+
# Retag the wheel with known platform-tag, becoming `air_formatter-{air-version}-py3-none-{platform-tag}.whl`
79+
run: uvx wheel tags --platform-tag ${{ matrix.this.platform_tag }} --remove wheel/*.whl
80+
81+
- name: "Test wheel"
82+
run: uv tool run --from wheel/*.whl air --help
83+
84+
- name: "Upload wheel"
85+
uses: actions/upload-artifact@v4
86+
with:
87+
name: wheels-${{ matrix.this.target }}
88+
path: wheel/*.whl

‎.github/workflows/build.yml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,3 +23,9 @@ jobs:
2323
name: Build linux
2424
uses: ./.github/workflows/build-linux.yml
2525
secrets: inherit
26+
27+
build_wheels:
28+
name: Build wheels
29+
uses: ./.github/workflows/build-wheels.yml
30+
secrets: inherit
31+
needs: [build_macos, build_windows, build_linux]

‎.github/workflows/publish-pypi.yml‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
# Publish wheels to PyPI
2+
#
3+
# Assumed to run as a subworkflow of `.github/workflows/release.yml`.
4+
# Specifically, as a `publish` step within `cargo-dist`.
5+
#
6+
# Pulls the wheels generated by `build-wheels.yml` during the
7+
# `local-artifacts-jobs` step.
8+
#
9+
# See these docs for more about trusted publishing
10+
# https://docs.pypi.org/trusted-publishers/
11+
name: "Publish to PyPI"
12+
13+
on:
14+
workflow_call:
15+
# Passed via dist, but not used
16+
inputs:
17+
plan:
18+
required: true
19+
type: string
20+
21+
jobs:
22+
publish:
23+
name: "Publish to PyPI"
24+
runs-on: ubuntu-latest
25+
26+
# For the restricted GitHub Environment
27+
environment: pypi
28+
29+
permissions:
30+
# For PyPI's trusted publishing
31+
id-token: write
32+
33+
steps:
34+
- uses: astral-sh/setup-uv@v7
35+
36+
- name: "Download wheels"
37+
uses: actions/download-artifact@v4
38+
with:
39+
pattern: wheels-*
40+
path: wheels
41+
merge-multiple: true
42+
43+
- name: "Publish to PyPI"
44+
run: uv publish wheels/*

‎.github/workflows/release.yml‎

Lines changed: 16 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -209,14 +209,29 @@ jobs:
209209
210210
gh release create "${{ needs.plan.outputs.tag }}" --target "$RELEASE_COMMIT" $PRERELEASE_FLAG --title "$ANNOUNCEMENT_TITLE" --notes-file "$RUNNER_TEMP/notes.txt" artifacts/*
211211
212+
custom-publish-pypi:
213+
needs:
214+
- plan
215+
- host
216+
if: ${{ !fromJson(needs.plan.outputs.val).announcement_is_prerelease || fromJson(needs.plan.outputs.val).publish_prereleases }}
217+
uses: ./.github/workflows/publish-pypi.yml
218+
with:
219+
plan: ${{ needs.plan.outputs.val }}
220+
secrets: inherit
221+
# publish jobs get escalated permissions
222+
permissions:
223+
"id-token": "write"
224+
"packages": "write"
225+
212226
announce:
213227
needs:
214228
- plan
215229
- host
230+
- custom-publish-pypi
216231
# use "always() && ..." to allow us to wait for all publish jobs while
217232
# still allowing individual publish jobs to skip themselves (for prereleases).
218233
# "host" however must run to completion, no skipping allowed!
219-
if: ${{ always() && needs.host.result == 'success' }}
234+
if: ${{ always() && needs.host.result == 'success' && (needs.custom-publish-pypi.result == 'skipped' || needs.custom-publish-pypi.result == 'success') }}
220235
runs-on: "ubuntu-latest"
221236
env:
222237
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}

‎.vscode/settings.json‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,10 @@
88
"editor.formatOnSave": true,
99
"editor.defaultFormatter": "esbenp.prettier-vscode"
1010
},
11+
"[python]": {
12+
"editor.formatOnSave": true,
13+
"editor.defaultFormatter": "charliermarsh.ruff"
14+
},
1115
"rust-analyzer.check.command": "clippy",
1216
"rust-analyzer.imports.prefix": "crate",
1317
"rust-analyzer.imports.granularity.group": "item",

‎CHANGELOG.md‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,19 @@
22

33
# Development version
44

5+
- Air is now distributed on PyPI as `air-formatter` (#467).
6+
7+
This allows air to be invoked via uv:
8+
9+
```bash
10+
# Global install of `air`
11+
uv tool install air-formatter
12+
air format .
13+
14+
# One-off run
15+
uvx --from air-formatter air format .
16+
```
17+
518
- Air is now code-signed on Windows (#461).
619

720

‎CONTRIBUTING.md‎

Lines changed: 45 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -4,21 +4,39 @@ Welcome! We really appreciate that you'd like to contribute to Air, thanks in ad
44

55
# Release process
66

7-
The release process of Air has some manual steps. One complication is that for each release of the CLI binary, we create a new release of the VS Code / OpenVSX extension as this is our primary way of distributing Air. The version numbers between the CLI binary and the VS Code / OpenVSX extension will end up being different.
7+
The release process of Air has some manual steps.
88

9-
When you want to cut a release of the Air binary and Air VS Code / OpenVSX extension:
9+
For each release of the CLI binary, we also create a release of:
10+
11+
- The air-formatter PyPI package (with the same version number)
12+
13+
- The VS Code and OpenVSX extension (with a different version number)
14+
15+
When you want to cut a release of Air:
1016

1117
- Create a release branch
1218

13-
- Polish `CHANGELOG.md`, bump the version and add a new `Development version` header (yep, right away - `cargo dist` is smart enough to ignore this header).
19+
- Polish `CHANGELOG.md`
20+
21+
- Clean up any bullets that need reorganization
22+
23+
- Bump the `CHANGELOG.md` version
1424

15-
- Polish `editors/code/CHANGELOG.md`, bump the version and add a new `Development version` header.
25+
- Add a new `Development version` header (yep, right away - `cargo dist` is smart enough to ignore this header)
1626

17-
- Mention that the new version of the binary is shipped with the extension.
27+
- Polish `editors/code/CHANGELOG.md`
28+
29+
- Mention that the new version of the binary is shipped with the extension
30+
31+
- Bump the `CHANGELOG.md` version
32+
33+
- Add a new `Development version` header
1834

1935
- In `crates/air/Cargo.toml`, bump the version.
2036

21-
- Run `cargo check` to sync `Cargo.lock`, in case your LSP didn't do it already.
37+
- Run `cargo check` to sync `Cargo.lock`, in case your LSP didn't do it already.
38+
39+
- In `python/pyproject.toml`, bump the version.
2240

2341
- In `editors/code/package.json`, bump the minor version to the next even number for standard releases, or to the next odd number for preview releases.
2442

@@ -30,19 +48,23 @@ When you want to cut a release of the Air binary and Air VS Code / OpenVSX exten
3048

3149
- The release workflow will:
3250

33-
- Build the binaries and installer scripts.
51+
- Build the Air binaries and installer scripts.
52+
53+
- Build the Python wheels from the Air binaries.
54+
55+
- Push the Python wheels to PyPI.
3456

3557
- Create and push a git tag for the version.
3658

3759
- Create a GitHub Release attached to that git tag.
3860

39-
- Attach the binaries and scripts to that GitHub Release as artifacts.
61+
- Attach the binaries and installer scripts to that GitHub Release as artifacts.
4062

4163
- Manually run the [extension release workflow](https://github.com/posit-dev/air/actions/workflows/release-vscode.yml)
4264

4365
- It runs on `workflow_dispatch`, and automatically pulls in the latest release binary of Air from the binary release workflow above. It will release to both the VS Code marketplace and the OpenVSX marketplace.
4466

45-
- Bump the version of Air recorded in Positron's [`product.json`](https://github.com/posit-dev/positron/blob/main/product.json).
67+
- Bump the version of Air recorded in Positron's [`product.json`](https://github.com/posit-dev/positron/blob/main/product.json) and do a PR to Positron.
4668

4769
- Merge the release branch
4870

@@ -87,6 +109,20 @@ For a new release:
87109

88110
If you have any questions about the process, refer to [Zed's update guide](https://zed.dev/docs/extensions/developing-extensions#updating-an-extension).
89111
112+
# Python wheels
113+
114+
Python wheel creation and publishing is handled automatically at release time through `release.yml`. Here we document parts of that automated process.
115+
116+
The Python wheels we distribute have the sole purpose of shipping the Air binary. There is no Python code in the wheel, and we don't support `python -m air` (meaning there is no `__main__.py` entry point). We expect it is more likely used as `uvx --from air-formatter air format .` (for a one off run) or as `uv tool install air-formatter` (for a global install of `air` which is symlinked into `~/.local/bin`), neither of which go through the thin Python shim that `python -m air` would do. Instead, these just call the shipped air binary directly.
117+
118+
The scaffolding for the Python package is in `python/`. We use `uv_build` as the build system, since it has nice support for the `scripts/` directory, which is where we put the Air binary for distribution.
119+
120+
In CI, `build-wheels.yml` runs as part of `build.yml`, which itself is called via cargo-dist's `release.yml`. `build-wheels.yml` collects the binaries from the other build steps and builds a per-platform wheel that puts the platform specific binary into `scripts/`. In `pyproject.toml`, we've set `[tool.uv.build-backend.data]` so that `uv_build` knows to copy over `scripts/` into the resulting wheel at build time. We then run `uv build` to build a generic "any" wheel without a specific platform, however, because there is a platform specific binary in there we really need it to be tagged with a specific platform. So we have to retag it with the known platform tag as a follow up. These platform tags tell PyPI how to deliver the right wheel when the user requests `air-formatter`.
121+
122+
Later on in `release.yml`, the `publish-pypi.yml` job runs at publish time. It collects the wheels and uses `uv publish` to send them off to PyPI. This is a specially named job! It uses PyPI's Trusted Publishing so that we don't need any tokens. Instead, on Davis's PyPI account we have told PyPI to expect that `posit-dev/air` has a `publish-pypi.yml` workflow with a `environment: pypi` GitHub Environment set up, and when binaries are pushed from that source, PyPI will accept them without any additional tokens.
123+
124+
If you're testing the Python wheel generation locally, use `just build-wheel` to build the wheel, and `just run-wheel <air args>` to run it. This will build release Air, copy it into `scripts/`, build the "any" wheel (which is correct for you, since you just built Air), and then `run-wheel` will run it with `uv tool run`.
125+
90126
# VS Code Extension development installation
91127

92128
- Build the development version of the Air CLI with:

‎dist-workspace.toml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,8 @@ targets = [
3131
build-local-artifacts = false
3232
# Local artifacts jobs to run in CI
3333
local-artifacts-jobs = ["./build"]
34+
# Publish jobs to run in CI
35+
publish-jobs = ["./publish-pypi"]
3436
# Whether to auto-include files like READMEs, LICENSEs, and CHANGELOGs (default true)
3537
auto-includes = false
3638
# Which actions to run on pull requests (use "upload" to force a build in CI for testing)

0 commit comments

Comments
 (0)