chore(python): update pyproject, docs, and cleanup makefile (#6809)

While working in the python bindings directory I kept hitting some
issues while developing, so I cleaned up a few things that I was able to
figure out.

The main problem was uv sync kept re-resolving the lockfile since our
optional deps had unpinned packages, even if I was just running the
tests. Fixed that by pinning the dev dependencies but ultimately can
revert if im lacking context as to why they weren't pinned before. But
now `uv sync` is stable out the box.

While I was in there I also, did an overhaul of the `MakeFile` so that
install was just one command, leveraged a bit more uv there, and added
help to document everything. Also, updated the docs that referenced the
old logic to use new changes. Although there are a few more places that
need this. i.e. direct `maturin` references.

After this PR:

```
cd python && make install # once
uv run make test  # every day
make build  # after Rust changes
uv sync # after pulling commits with dep changes
```

---------

Co-authored-by: Xuanwo <github@xuanwo.io>

Author: geruh

.github/workflows/ci-benchmarks.yml

@@ -47,8 +47,8 @@ jobs:
         run: |
           python -m venv venv
           source venv/bin/activate
-          pip install maturin duckdb requests pytest pytest-benchmark datasets
-          maturin develop --locked --release --features datagen
+          pip install maturin==1.13.3 uv duckdb requests pytest pytest-benchmark datasets
+          maturin develop --uv --locked --release --features datagen
       - name: Build memtest
         run: |
           source venv/bin/activate

.github/workflows/codex-backport-pr.yml

@@ -142,7 +142,7 @@ jobs:
           10. Run ONLY the tests related to the changes in these PRs:
               - Use "git diff --name-only ${RELEASE_BRANCH}...HEAD" to see all files changed across all cherry-picked commits.
               - For Rust changes: Run tests for the affected crates only (e.g., "cargo test -p lance-core" if lance-core files changed).
-              - For Python changes (python/** files): Build with "cd python && maturin develop" then run "pytest" on the specific test files that were modified, or related test files.
+              - For Python changes (python/** files): Build with "cd python && make build" then run "uv run pytest" on the specific test files that were modified, or related test files.
               - For Java changes (java/** files): Run "cd java && mvn test" for the affected modules.
               - If test files themselves were modified, run those specific tests.
               - Do NOT run the full test suite - only run tests related to the changed files.

.github/workflows/codex-fix-ci.yml

@@ -144,7 +144,7 @@ jobs:
              - Run "cargo clippy --workspace --tests --benches -- -D warnings" to check for issues
              - Run ONLY the specific failing tests to confirm they pass now:
                - For Rust test failures: Run the specific test with "cargo test -p <crate> <test_name>"
-               - For Python test failures: Build with "cd python && maturin develop" then run "pytest <specific_test_file>::<test_name>"
+               - For Python test failures: Build with "cd python && make build" then run "uv run pytest <specific_test_file>::<test_name>"
                - For Java test failures: Run "cd java && mvn test -Dtest=<TestClass>#<testMethod>"
                - Do NOT run the full test suite - only run the tests that were failing
 

.github/workflows/file_verification.yml

@@ -51,8 +51,8 @@ jobs:
         run: |
           python -m venv venv
           source venv/bin/activate
-          pip install pyarrow maturin
-          maturin develop --release
+          pip install pyarrow maturin==1.13.3 uv
+          maturin develop --uv --release
 
       - name: Test Lance File Write Read Round Trip
         run: |

.github/workflows/python.yml

@@ -87,8 +87,8 @@ jobs:
           python -m venv venv
           source venv/bin/activate
           pip install torch tqdm --index-url https://download.pytorch.org/whl/cpu
-          pip install maturin
-          maturin develop --profile ci --locked --extras tests,ray
+          pip install maturin==1.13.3 uv
+          maturin develop --uv --profile ci --locked --extras tests,ray
       - name: Run doctest
         run: |
           source venv/bin/activate

python/AGENTS.md

@@ -5,13 +5,15 @@ Also see [root AGENTS.md](../AGENTS.md) for cross-language standards.
 ## Commands
 
 * Environment: use `uv` for all local Python environment setup in this repository.
-* First step in every new worktree or fresh checkout: run `uv sync --extra tests --extra dev` from `python/` before any Python command. Add other extras such as `benchmarks`, `torch`, or `geo` only when needed.
+* First step in every new worktree or fresh checkout: run `make install` from `python/` before any Python command. This runs `uv sync` (dev and test dependencies are included by default via `[tool.uv] default-groups`) and sets up pre-commit hooks. Add `--group benchmarks`, `--extra torch`, or `--extra geo` only when needed.
 * `uv sync` builds the local `pylance` Rust extension as part of environment setup. This can take a long time. Start it early, let it finish, and do not interrupt it or switch to a different setup path just because the build is slow.
+* After the initial `make install`, use `uv run make test` to run tests.
+* Only run `uv sync` again when dependencies change (e.g., after pulling new commits that update `pyproject.toml` or `uv.lock`).
 * Command execution: always use `uv run ...` for Python-related repository commands. Do not rely on a globally activated environment.
 * Never invoke bare `python`, `pytest`, `pip`, `maturin`, `make test`, `make doctest`, `make lint`, or `make format` for repository work.
 * If a Python command fails outside `uv run`, that does not count as a dependency or test failure. Fix the environment usage first and rerun correctly.
-* Build time expectations: `uv sync` and `uv run maturin develop` build the local `pylance` Rust extension as part of the environment workflow. This can be slow, especially on the first run or after Rust dependency changes; treat that as expected and do not switch to a different environment manager or shortcut around the build just because it takes time.
-* Build: `uv run maturin develop` (required after Rust changes)
+* Build time expectations: `make install` and `make build` build the local `pylance` Rust extension as part of the environment workflow. This can be slow, especially on the first run or after Rust dependency changes; treat that as expected and do not switch to a different environment manager or shortcut around the build just because it takes time.
+* Build: `make build` (required after Rust changes)
 * Test: `uv run make test`
 * Run single test: `uv run pytest python/tests/<test_file>.py::<test_name>`
 * Doctest: `uv run make doctest`
@@ -34,4 +36,4 @@ Also see [root AGENTS.md](../AGENTS.md) for cross-language standards.
 ## Common Failure Mode
 
 - A missing module or missing command error from bare `python`, `pytest`, `pip`, `maturin`, or `make` is usually an environment usage mistake, not a repository issue.
-- Before reporting a Python dependency as unavailable, verify that `uv sync --extra tests --extra dev` has been run in the current worktree and that the failing command was executed with `uv run ...`.
+- Before reporting a Python dependency as unavailable, verify that `uv sync` has been run in the current worktree and that the failing command was executed with `uv run ...`.

python/CONTRIBUTING.md

@@ -10,14 +10,14 @@ The python integration is done via pyo3 + custom python code:
 To build the Python bindings, first install requirements:
 
 ```bash
-pip install maturin
+cd python
+make install
 ```
 
 To make a dev install:
 
 ```bash
-cd python
-maturin develop
+make build
 ```
 
 After installing, you can run `import lance` in a Python shell within the virtual environment.

python/DEVELOPMENT.md

@@ -3,10 +3,10 @@
 For local development, prefer [uv](https://docs.astral.sh/uv/) to create and manage the Python environment:
 
 ```shell
-uv sync --extra tests --extra dev
+uv sync
 ```
 
-Add extras such as `benchmarks`, `torch`, or `geo` only when you need them. After the environment is initialized, either activate it or use `uv run ...` for commands.
+Add `--group benchmarks` for benchmarks, or `--extra torch` / `--extra geo` for those optional features. After the environment is initialized, either activate it or use `uv run ...` for commands.
 
 `uv sync` is not just downloading Python packages here. It also builds the local `pylance` Rust extension as part of the editable environment, so the first run, cache misses, or Rust dependency changes can make it noticeably slow. This is expected; let the build finish instead of interrupting it and switching to a different environment setup.
 
@@ -17,7 +17,7 @@ This project is built with [maturin](https://github.com/PyO3/maturin).
 It can be built in development mode with:
 
 ```shell
-uv run maturin develop
+uv run maturin develop --uv
 ```
 
 This builds the Rust native module in place. You will need to re-run this
@@ -104,8 +104,8 @@ benchmarks added there should run in less than 5 seconds.
 Before running benchmarks, you should build pylance in release mode:
 
 ```shell
-uv sync --extra tests --extra dev --extra benchmarks
-uv run maturin develop --profile release-with-debug --extras benchmarks --features datagen
+uv sync --group benchmarks
+uv run maturin develop --uv --profile release-with-debug --extras benchmarks --features datagen
 ```
 
 (You can also use `--release` or `--profile release`, but `--profile release-with-debug`
@@ -172,13 +172,13 @@ the benchmarks again with `--benchmark-compare`.
 ```shell
 CURRENT_BRANCH=$(git branch --show-current)
 git checkout main
-uv sync --extra tests --extra dev --extra benchmarks
-uv run maturin develop --profile release-with-debug --features datagen
+uv sync --group benchmarks
+uv run maturin develop --uv --profile release-with-debug --features datagen
 uv run pytest --benchmark-save=baseline python/benchmarks -m "not slow"
 COMPARE_ID=$(ls .benchmarks/*/ | tail -1 | cut -c1-4)
 git checkout $CURRENT_BRANCH
-uv sync --extra tests --extra dev --extra benchmarks
-uv run maturin develop --profile release-with-debug --features datagen
+uv sync --group benchmarks
+uv run maturin develop --uv --profile release-with-debug --features datagen
 uv run pytest --benchmark-compare=$COMPARE_ID python/benchmarks -m "not slow"
 ```
 

python/Makefile

@@ -1,53 +1,74 @@
+.DEFAULT_GOAL := help
+.PHONY: help install build test integtest doctest compattest format format-python lint lint-python lint-rust clean
+PYTHON ?=
+PYTEST_ARGS ?= -vvv -s -m "not recurring"
+KEEP_COMPOSE ?= 0
+COMPOSE_FILE ?= ../docker-compose.yml
+UV_SYNC = uv sync
+UV_RUN = uv run --frozen
+
+# Set PYTHON=3.12 to use a specific Python version with uv.
+ifneq ($(strip $(PYTHON)),)
+  UV_PYTHON_ARG = --python $(PYTHON)
+  UV_SYNC += $(UV_PYTHON_ARG)
+  UV_RUN += $(UV_PYTHON_ARG)
+endif
+
 ifeq ($(CI), true)
-    PYTEST_ARGS = -vvv -s --durations=30 -m "not recurring"
-else
-    PYTEST_ARGS = -vvv -s -m "not recurring"
+  PYTEST_ARGS += --durations=30
 endif
 
-test:
+help: ## Show this help
+	@awk 'BEGIN {FS = ":.*##"; printf "\nUsage:\n  make \033[36m<target>\033[0m\n"} /^[a-zA-Z_-]+:.*?##/ { printf "  \033[36m%-20s\033[0m %s\n", $$1, $$2 }' $(MAKEFILE_LIST)
+
+install: ## Sync dependencies and set up local development tools
+	@if ! command -v uv > /dev/null 2>&1; then \
+		echo "uv not found. Install it from https://docs.astral.sh/uv/getting-started/installation/"; \
+		exit 1; \
+	fi
+	$(UV_SYNC)
+	uv tool run pre-commit install
+
+build: ## Build the local Rust extension with maturin
+	$(UV_RUN) maturin develop --uv
+
+test: ## Run Python tests except recurring tests
 	pytest $(PYTEST_ARGS) python/tests
-.PHONY: test
 
-integtest:
-	pytest --run-integration $(PYTEST_ARGS) python/tests/test_s3_ddb.py python/tests/test_namespace_integration.py
-.PHONY: integtest
+integtest: ## Start LocalStack and run integration tests
+	@if [ "$(KEEP_COMPOSE)" = "1" ]; then \
+		docker compose -f $(COMPOSE_FILE) up -d --wait && \
+		pytest --run-integration $(PYTEST_ARGS) python/tests/test_s3_ddb.py python/tests/test_namespace_integration.py; \
+	else \
+		trap 'docker compose -f $(COMPOSE_FILE) down -v --remove-orphans --timeout 0 >/dev/null 2>&1 || true' EXIT; \
+		docker compose -f $(COMPOSE_FILE) up -d --wait && \
+		pytest --run-integration $(PYTEST_ARGS) python/tests/test_s3_ddb.py python/tests/test_namespace_integration.py; \
+	fi
 
-doctest:
+doctest: ## Run Python doctests
 	pytest --doctest-modules $(PYTEST_ARGS) python/lance
-.PHONY: doctest
 
-compattest:
-	pytest --run-compat $(PYTEST_ARGS)  python/tests/compat
-.PHONY: compattest
+compattest: ## Run upgrade/downgrade compatibility tests
+	pytest --run-compat $(PYTEST_ARGS) python/tests/compat
 
-format: format-python
+format: format-python ## Auto-format Python and Rust
 	cargo fmt
-.PHONY: format
-
-build:
-	maturin develop
-.PHONY: build
 
-format-python:
+format-python: ## Auto-format Python (ruff)
 	ruff format python
 	ruff check --fix python
-.PHONY: format-python
 
-lint: lint-python lint-rust
-.PHONY: lint
+lint: lint-python lint-rust ## Check Python and Rust formatting/lints
 
-lint-python:
+lint-python: ## Check Python formatting, linting, and types
 	ruff format --check --diff python
 	ruff check python
 	pyright
-.PHONY: lint-python
 
-lint-rust:
+lint-rust: ## Check Rust formatting and clippy lints
 	cargo fmt -- --check
 	cargo clippy -- -D warnings
-.PHONY: lint-rust
 
-clean:
+clean: ## Remove build artifacts
 	cargo clean
 	rm -rf target
-.PHONY: clean

python/pyproject.toml

@@ -71,6 +71,33 @@ geo = [
     "geoarrow-rust-io",
 ]
 
+[dependency-groups]
+tests = [
+    "boto3==1.40.43",
+    "datasets==4.1.1; python_version >= '3.10'",
+    "duckdb==1.4.0",
+    "ml_dtypes==0.5.3",
+    "pillow==11.3.0",
+    "pandas==2.3.3",
+    "polars[pyarrow,pandas]==1.34.0",
+    "psutil==7.1.0",
+    "pytest==8.4.2",
+    "tensorflow==2.20.0; sys_platform == 'linux' and python_version >= '3.10'",
+    "tqdm==4.67.1",
+    "datafusion==53.0.0; python_version >= '3.10'",
+]
+dev = [
+    "maturin==1.13.3",
+    "pyright==1.1.406",
+    "ruff==0.11.2",
+]
+benchmarks = [
+    "pytest-benchmark==5.1.0",
+]
+
+[tool.uv]
+default-groups = ["dev", "tests"]
+
 [tool.ruff]
 lint.select = ["F", "E", "W", "I", "G", "TCH", "PERF", "B019"]