|
| 1 | +#ai-slop |
| 2 | +# Python Runner Startup Benchmarks |
| 3 | + |
| 4 | +Measures the warm-invocation overhead of Python version managers and script runners using [hyperfine](https://github.com/sharkdp/hyperfine). Goal: concrete "rules of thumb" for how much each runner layer adds. |
| 5 | + |
| 6 | +## Quick start |
| 7 | + |
| 8 | +```bash |
| 9 | +# One-time setup (creates venvs, installs deps, primes caches) |
| 10 | +./setup-python-projects.sh |
| 11 | + |
| 12 | +# Run all benchmarks |
| 13 | +./bench-python.sh |
| 14 | + |
| 15 | +# Just one category |
| 16 | +./bench-python.sh --category 1 # bare Python interpreters |
| 17 | +./bench-python.sh --category 2 # with-deps runners |
| 18 | + |
| 19 | +# More runs for tighter confidence intervals |
| 20 | +./bench-python.sh --min-runs 50 |
| 21 | + |
| 22 | +# Correctness check (1 iteration each, no warmup — verifies commands work) |
| 23 | +./bench-python.sh --run1 |
| 24 | +``` |
| 25 | + |
| 26 | +Results are written to `results/*.md` and printed at the end. |
| 27 | + |
| 28 | +## What's being measured |
| 29 | + |
| 30 | +### Category 1 — Bare Python interpreter |
| 31 | + |
| 32 | +Workload: `import sys; print(dir(sys))` |
| 33 | + |
| 34 | +Tests each Python binary on PATH directly — no runner wrapper. Compares system, brew, pyenv shim, uv-managed, and mise-managed Pythons. Version labels are queried at runtime so they stay accurate as Python versions change. |
| 35 | + |
| 36 | +### Category 2 — With dependencies (requests) |
| 37 | + |
| 38 | +Workload: `import requests; print(dir(requests))` for project runners |
| 39 | + |
| 40 | +Tests the runner tax on a warm cache: uv run (project + script + offline), uvx, pipenv, poetry, pdm, hatch, pipx. |
| 41 | + |
| 42 | +### Category 3 -- Ephemeral Tools |
| 43 | +- [ ] `cowsay --text hello` for ephemeral tool runners (uvx, pipx). |
| 44 | + |
| 45 | +## Stability / reading the results |
| 46 | + |
| 47 | +hyperfine has no interleave/shuffle mode — it runs all N runs of command 1, then all N of command 2, etc. ([open issue #21](https://github.com/sharkdp/hyperfine/issues/21)). A background burst (Docker, Spotlight) inflates whichever command was running at that moment. |
| 48 | + |
| 49 | +Mitigations in this script: |
| 50 | +- `--min-runs 30` — lets hyperfine extend runs for high-variance commands automatically |
| 51 | +- `--warmup 5` — stabilizes caches before timing begins |
| 52 | + |
| 53 | +**The Min column is more reliable than Mean** for "what can this do uncontested." Treat Mean ± σ as a range. The *ranking* is stable across runs even when absolute ms values drift ±5–10ms. |
| 54 | + |
| 55 | +## Diagnosing slow startup with `-X importtime` and `-S` |
| 56 | + |
| 57 | +When a specific tool or script is slow, use Python's built-in profilers before assuming it's the runner: |
| 58 | + |
| 59 | +### `-X importtime` — trace which imports are slow |
| 60 | + |
| 61 | +```bash |
| 62 | +python -X importtime -c 'import requests' 2>&1 | sort -k2 -n | tail -20 |
| 63 | +``` |
| 64 | + |
| 65 | +Can use [python-importtime-graph](https://github.com/kmichel/python-importtime-graph) for a browser treemap: |
| 66 | + |
| 67 | +```bash |
| 68 | +python -X importtime -c 'import requests' 2>&1 | python-importtime-graph > out.html |
| 69 | +open out.html |
| 70 | +``` |
| 71 | + |
| 72 | +[Simon Willison used this](https://simonwillison.net/2025/Jun/20/python-importtime-graph/) to diagnose a tool taking >1s to start — turned out a single transitive import was the culprit. |
| 73 | + |
| 74 | +## `-S` is not significant |
| 75 | + |
| 76 | +`-S` (CPython internals flag) disables the `site` module, which sets up `sys.path` and scans for installed `site-packages`. [Victor Stinner's analysis](https://pythondev.readthedocs.io/startup_time.html) shows it accounted for roughly half of Python's startup time. |
| 77 | + |
| 78 | +Benchmarked: mise's `python -S` runs ~1ms faster than the bare mise binary (~21ms), within noise at this scale. Where `-S` helps more is on older/slower Pythons or when `site-packages` is large (many installed packages slow the scan). |
| 79 | + |
| 80 | +```bash |
| 81 | +time python -c 'print("hello")' # normal |
| 82 | +time python -S -c 'print("hello")' # skip site module |
| 83 | +``` |
| 84 | + |
| 85 | +To use it: add `-S` to the shebang line: `#!/usr/bin/env -S python3 -S` |
| 86 | + |
| 87 | +## Prior art |
| 88 | + |
| 89 | +No existing benchmark covers the **same scope:** various standalone python, and projects. The closest sources: |
| 90 | + |
| 91 | +- **[pdm-project/pdm #1527](https://github.com/pdm-project/pdm/issues/1527)** and **[python-poetry/poetry #3502](https://github.com/python-poetry/poetry/issues/3502)** — issue threads with raw `time`/hyperfine numbers showing pdm: 390–1028ms, poetry: 526–752ms, bare Python: 21ms. Not controlled for warmup, but order of magnitude matches. |
| 92 | +- **[DEV.to — Python's Hidden Bottleneck (Werner Smit, 2025)](https://dev.to/werner_smit_355bfa500f8c3/pythons-startup-tax-when-script-startup-time-becomes-the-bottleneck-2np6)** — bare Python startup on Linux (not runner overhead). Uses hyperfine with warmup. Key: `import requests` adds ~100ms over bare Python. |
| 93 | +- **[Victor Stinner — Python Startup Time](https://pythondev.readthedocs.io/startup_time.html)** — CPython core dev analysis. Covers `-S`, `-X importtime`, and historical benchmarks. Foundational. |
| 94 | +- **[CPython issue #118761](https://github.com/python/cpython/issues/118761)** — active effort to reduce stdlib import times. Context for why Python 3.14 may benchmark faster than 3.11. |
| 95 | +- **[bdrung/startup-time](https://github.com/bdrung/startup-time)** — referenced in `ShellScripting.md`. 1000 hello-world runs across many languages; solid discipline but only bare interpreter, no runner/version manager comparison. |
| 96 | + |
| 97 | +## Diagnosing uv cache misses after long breaks |
| 98 | + |
| 99 | +When `uv run --script` is slow after coming back from a holiday or long gap, run with `-v` to see what's actually happening: |
| 100 | + |
| 101 | +```bash |
| 102 | +uv run -v your_script.py 2>&1 | grep -E 'Creating|Resolving|Downloading|Fetching|Updating' |
| 103 | +``` |
| 104 | + |
| 105 | +Key lines to look for: |
| 106 | +- `Creating virtual environment` — new ephemeral env being built (cache miss or version bump) |
| 107 | +- `Resolving` + `Downloading` — unpinned dep resolved to a newer version |
| 108 | +- `Fetching` — index metadata revalidated from network |
| 109 | + |
| 110 | +**Most likely causes after a long break:** |
| 111 | +1. **Unpinned dep got a new release** — `requests` (no version pin) resolved to a newer version; uv builds a fresh environment. Partial fix: add a major-version cap (`requests<3`) to reduce churn without blocking security updates. |
| 112 | +2. **uv version bump changed cache bucket format** — caches are versioned; upgrading uv invalidates old cached environments. Verify: `du -sh "$(uv cache dir)"` before and after a slow run. |
| 113 | +3. **macOS Storage Management purged `~/.cache/uv`** — the default cache location can be targeted by "Optimize Storage." Check: `uv cache dir`. |
| 114 | + |
| 115 | +## Follow-ups |
| 116 | +- [ ] Split out a category 3 for cowsay for ephemeral tool |
| 117 | +- [ ] Research options to improve startup time of other runners (pdm, hatch, pipenv, poetry), like `--offline` in uv |
| 118 | +- [ ] Try same tests inside docker |
| 119 | +- [ ] Try similar tests inside windows |
| 120 | +- [ ] [interpreter-startup-times](https://github.com/MaxGyver83/interpreter-startup-times) has gnuplot examples that would be useful to visualize these numbers |
| 121 | + |
| 122 | +### Projects using different python executables |
| 123 | + |
| 124 | +- [ ] [[setup-python-projects.sh]] should start each tool using uv python, not a mix of pyenv and brew |
| 125 | + |
| 126 | +| Project | Source | |
| 127 | +| ------------------ | --------------------------------------------------------------------------- | |
| 128 | +| **venv-baseline** | `~/.local/share/uv/python/cpython-3.13.3-macos-aarch64-none/bin/python3.13` | |
| 129 | +| **uv-project** | `~/.local/share/uv/python/cpython-3.13.3-macos-aarch64-none/bin/python3.13` | |
| 130 | +| **pipenv-project** | `~/.pyenv/versions/3.14.3/bin/python3.14` | |
| 131 | +| **poetry-project** | `~/.local/share/uv/python/cpython-3.13.3-macos-aarch64-none/bin/python3.13` | |
| 132 | +| **pdm-project** | `~/.pyenv/versions/3.11.6/bin/python3.11` | |
| 133 | +| **hatch-project** | `/opt/homebrew/Cellar/python@3.14/3.14.3_1/.../python3.14` | |
| 134 | +### JS runner benchmark (`bench-js.sh`) |
| 135 | + |
| 136 | +- [ ] Add a `bench-js.sh` comparing `node`, `bun`, `deno` bare startup + `npx`/`bunx`/`pnpm dlx` wrapper tax on the same trivial workload |
| 137 | +- [ ] Compare `uvx` vs `bunx` vs `npx` for the same ephemeral tool (`cowsay` exists on both PyPI and npm — same workload across ecosystems) |
| 138 | +- [ ] also try [deno compile](https://deno.com/blog/v2.7#deno-install---compile) as a "pre-compiled native binary" baseline |
| 139 | + |
| 140 | +### Network condition matrix |
| 141 | + |
| 142 | +Run each runner under three network states (prime cache first, then measure): |
| 143 | + |
| 144 | +| Condition | What it tests | |
| 145 | +|---|---| |
| 146 | +| Regular network | warm-cache baseline | |
| 147 | +| VPN | DNS/proxy latency; whether internal mirrors are used | |
| 148 | +| No network | does the runner phone home on warm cache? timeout penalty | |
| 149 | + |
| 150 | +For VPN runs, swap in `.npmrc` pointing to internal npm mirror and `UV_INDEX` for internal PyPI mirror — remove both off VPN. |
| 151 | + |
| 152 | +### Spying on runner network calls |
| 153 | + |
| 154 | +To see which hostnames a runner contacts during a run (without full MITM): |
| 155 | + |
| 156 | +```bash |
| 157 | +# Capture DNS + TLS SNI for one command |
| 158 | +sudo tcpdump -i any -n -s 0 -w /tmp/runner.pcap "tcp port 443 or port 53" & |
| 159 | +uvx cowsay --text hello |
| 160 | +sudo kill %1 |
| 161 | +tshark -r /tmp/runner.pcap -Y "dns.qry.name" -T fields -e frame.time -e dns.qry.name |
| 162 | +tshark -r /tmp/runner.pcap -Y "tls.handshake.extensions_server_name" -T fields -e frame.time -e tls.handshake.extensions_server_name |
| 163 | +``` |
| 164 | + |
| 165 | +For response sizes and exact URLs, use **Proxyman** (GUI, easy CA trust) or **mitmproxy** — both work via `HTTPS_PROXY` which most CLIs honor. |
| 166 | + |
0 commit comments