Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
204 changes: 204 additions & 0 deletions .github/deploy.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,204 @@
# Deployment Setup

This document describes the one-time GitHub and external-service configuration
required for the `Release` workflow (`.github/workflows/release.yml`) to publish
to crates.io, PyPI, and GitHub Releases.

## Overview

The release workflow is **manually dispatched** via `workflow_dispatch` and runs
only on branches named `release/*` or `release-*`. It has three boolean inputs:

| Input | Default | Effect |
|---|---|---|
| `publish_rust` | `true` | Publish the crate to crates.io |
| `publish_python` | `true` | Build wheels + publish to PyPI |
| `dry_run` | `false` | Validate only (no publishing, no tag, no GitHub Release) |

On a successful non-dry-run, the workflow:

1. Publishes `prollytree` to **crates.io** (`cargo publish --all-features`)
2. Builds Python wheels on Linux (x86_64, aarch64), Windows (x64), and macOS (arm64)
3. Publishes wheels + sdist to **PyPI** via OIDC trusted publishing
4. Creates a GitHub tag `v<version>` and a GitHub Release with auto-generated notes

## One-time setup

### 1. crates.io API token (repository secret)

The `publish-rust` job reads `CARGO_REGISTRY_TOKEN` from repository secrets.

1. Sign in at https://crates.io/me → **API Tokens** → **New Token**.
2. Configure the token:
- **Name**: e.g. `prollytree-gha`
- **Scopes**: `publish-update` only (do not enable `publish-new` or `yank` unless needed)
- **Crates**: scope to `prollytree`
- **Expiration**: 30–90 days (set a calendar reminder to rotate)
3. Click **Create** and copy the token immediately — it is shown only once. A
valid modern token is ~40+ characters with no whitespace.
4. In the GitHub repo → **Settings** → **Secrets and variables** → **Actions**
→ **New repository secret**:
- **Name**: `CARGO_REGISTRY_TOKEN` (exact case)
- **Secret**: paste the token (no leading/trailing whitespace)

Must be a **repository** secret, not an environment secret — `publish-rust`
does not declare an `environment:` and therefore cannot see environment secrets.

To verify the token format is well-formed before saving to GitHub, `cargo login
<token>` locally will reject a malformed token.

### 2. PyPI trusted publishing (OIDC, no secret)

PyPI does not use a long-lived API token. Instead, the `publish-python` job
mints a short-lived GitHub OIDC token (enabled by `permissions: id-token:
write` and `environment: pypi`), which PyPI exchanges for a short-lived upload
token.

1. Sign in at https://pypi.org/.
2. If `prollytree` already exists on PyPI: go to
https://pypi.org/manage/project/prollytree/settings/publishing/ and click
**Add a new publisher**.
If not: go to https://pypi.org/manage/account/publishing/ and add a
**pending publisher**.
3. In the **GitHub** tab, enter **exactly**:

| Field | Value |
|---|---|
| Owner | `zhangfengcdt` |
| Repository name | `prollytree` |
| Workflow name | `release.yml` (filename only — no `.github/workflows/` prefix) |
| Environment name | `pypi` |

4. Click **Add**.

If any field mismatches by even one character, PyPI returns `invalid-publisher`
at publish time. The error page includes the exact OIDC claims that were
presented — compare them side-by-side with the PyPI configuration.

### 3. GitHub `pypi` environment

The `publish-python` job declares `environment: pypi`. The environment must
exist in the repo or the job will fail.

1. Repo → **Settings** → **Environments** (left sidebar) → **New environment**.
2. Name: `pypi` (exact match, lowercase).
3. Recommended protection:
- **Deployment branches and tags** → **Selected branches and tags** → add
pattern `release/*` so only release branches can deploy to PyPI.
- Optionally add **Required reviewers** to gate each publish on manual
approval.
4. Click **Save protection rules**.

No environment **secrets** are needed — OIDC replaces the token.

## Releasing

### Pre-release checklist

Before dispatching, make sure these are in sync on the release branch:

- `Cargo.toml` `version`
- `pyproject.toml` `version`
- `python/prollytree/__init__.py` `__version__`
- `python/docs/conf.py` `release` and `version`
- `README.md` install examples
- `src/lib.rs` doc example
- `src/bin/prolly-ui.rs` `#[command(version = ...)]`
- `Cargo.lock` (`cargo check` to regenerate)

Local validation:

```bash
cargo fmt --all -- --check
cargo clippy --all --all-features -- -D warnings
cargo check --all-features
cargo test --all-features # runs in release workflow; Linux CI covers it
cargo publish --dry-run --features "git,sql" --allow-dirty # macOS cannot run --all-features locally due to pyo3 libpython linkage
```

### Dispatching

1. Create a branch named `release/v<version>` (workflow rejects anything else).
2. Commit the version bump.
3. Push the branch.
4. Either via GitHub UI (**Actions** → **Release** → **Run workflow**) or CLI:

```bash
# Dry run first (publishes wheels to TestPyPI, does not create tag or GitHub Release)
gh workflow run release.yml --ref release/v<version> \
-f publish_rust=true -f publish_python=true -f dry_run=true
gh run watch

# Production release
gh workflow run release.yml --ref release/v<version> \
-f publish_rust=true -f publish_python=true -f dry_run=false
gh run watch
```

5. After success, merge the release branch back to `main`.

### Post-release verification

```bash
curl -s https://crates.io/api/v1/crates/prollytree | jq '.crate.max_version'
curl -s https://pypi.org/pypi/prollytree/json | jq '.info.version'
gh release view v<version>
```

## Troubleshooting

### `please provide a non-empty token` (publish-rust)

`CARGO_REGISTRY_TOKEN` secret is missing or empty. See [§1](#1-cratesio-api-token-repository-secret).
Must be a repository secret, not an environment secret.

### `The given API token does not match the format used by crates.io`

The token was pasted with corruption (trailing newline, partial copy) or is a
pre-2020-07-14 token. Regenerate on crates.io, verify with `cargo login`, and
replace the secret.

### `invalid-publisher: valid token, but no corresponding publisher` (publish-python)

OIDC succeeded but PyPI's trusted publisher config doesn't match the claims.
Compare the claims shown in the error message to the PyPI form field-by-field.
Most common cause: entering the workflow **path** (`.github/workflows/release.yml`)
instead of the **filename** (`release.yml`).

### macOS wheel build stuck at `Waiting for a runner to pick up this job...`

GitHub retired the free `macos-13` (Intel) runner image in early 2026. Either
drop the x86_64 macOS entry (arm64 wheels run on Intel Macs under Rosetta),
switch to the paid `macos-13-large` runner, or build a `universal2-apple-darwin`
wheel on `macos-14`.

### `error: the package 'prollytree' does not contain this feature: <name>`

A feature referenced in the workflow `--features` list was removed from
`Cargo.toml`. Update `release.yml` to match the current feature set in
`Cargo.toml`.

### maturin fails with `Couldn't find any python interpreters`

Pass `-i python3.11` (or another interpreter present on the runner) to the
maturin args. Combined with `pyo3` `abi3-py38` feature, one wheel per platform
covers all Python 3.8+.

### Tests fail with missing imports under `cargo test --all-features`

A test references a type gated on a feature (e.g. `rocksdb_storage`) without
importing it. Add a `#[cfg(feature = "<name>")] use ...;` in the test module.
Consider adding a CI matrix row that enables the feature so the failure is
caught on PRs rather than during release.

## Rollback

| Target | Command / action | Limitations |
|---|---|---|
| crates.io | `cargo yank --version <v>` | Cannot delete; yanked versions are permanently unusable for new `cargo publish` of the same version |
| PyPI | Delete release via web UI | Permanent — version number can never be reused |
| GitHub Release | `gh release delete v<v> --cleanup-tag` | Safe to recreate |

If a half-published release must be recovered, generally the right move is to
bump to the next patch version (`0.X.Y+1`) rather than attempting to re-publish
the same number.
8 changes: 2 additions & 6 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -124,11 +124,7 @@ jobs:
- os: windows-latest
target: x64
manylinux: false
# macOS x86_64
- os: macos-13
target: x86_64
manylinux: false
# macOS ARM64
# macOS ARM64 (Apple Silicon) — arm64 wheel works on Intel Macs via Rosetta
- os: macos-14
target: aarch64
manylinux: false
Expand All @@ -151,7 +147,7 @@ jobs:
uses: PyO3/maturin-action@v1
with:
target: ${{ matrix.target }}
args: --release --out dist --features python,git,sql,rig
args: --release --out dist --features python,git,sql -i python3.11
manylinux: ${{ matrix.manylinux }}
before-script-linux: |
# Install any system dependencies if needed
Expand Down
2 changes: 1 addition & 1 deletion Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
name = "prollytree"
description = "A prolly (probabilistic) tree for efficient storage, retrieval, and modification of ordered data."
authors = ["Feng Zhang <f.feng.zhang@gmail.com>"]
version = "0.3.2-beta"
version = "0.3.2"
edition = "2021"

license = "Apache-2.0"
Expand Down Expand Up @@ -40,7 +40,7 @@ async-trait = { version = "0.1", optional = true }
uuid = { version = "1.0", features = ["v4"], optional = true }
futures = { version = "0.3", optional = true }
tokio = { version = "1.0", features = ["rt-multi-thread", "macros", "sync"], optional = true }
pyo3 = { version = "0.22", features = ["extension-module"], optional = true }
pyo3 = { version = "0.22", features = ["extension-module", "abi3-py38"], optional = true }
rocksdb = { version = "0.22", optional = true }

[dev-dependencies]
Expand Down
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,10 +22,10 @@ Add to your `Cargo.toml`:

```toml
[dependencies]
prollytree = "0.3.2-beta"
prollytree = "0.3.2"

# Optional features
prollytree = { version = "0.3.2-beta", features = ["git", "sql"] }
prollytree = { version = "0.3.2", features = ["git", "sql"] }
```

## Examples
Expand Down Expand Up @@ -118,7 +118,7 @@ if let Some(node) = mem_tree.find(b"session:abc123") {

```toml
[dependencies.prollytree]
version = "0.3.2-beta"
version = "0.3.2"
features = ["git", "sql", "rocksdb_storage"]
```

Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ build-backend = "maturin"

[project]
name = "prollytree"
version = "0.3.2-beta"
version = "0.3.2"
description = "Python bindings for ProllyTree - a probabilistic tree for efficient storage and retrieval"
readme = "python/README.md"
requires-python = ">=3.8"
Expand Down
4 changes: 2 additions & 2 deletions python/docs/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,8 @@
author = 'ProllyTree Contributors'

# The full version, including alpha/beta/rc tags
release = '0.3.2-beta'
version = '0.3.2-beta'
release = '0.3.2'
version = '0.3.2'

# -- General configuration ---------------------------------------------------
# https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration
Expand Down
2 changes: 1 addition & 1 deletion python/prollytree/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -58,4 +58,4 @@
if git_available:
__all__.extend(["WorktreeManager", "WorktreeVersionedKvStore"])

__version__ = "0.3.2-beta"
__version__ = "0.3.2"
2 changes: 1 addition & 1 deletion src/bin/prolly-ui.rs
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ use std::path::{Path, PathBuf};
#[derive(Parser)]
#[command(name = "prolly-ui")]
#[command(about = "Generate static HTML visualization for git-prolly repositories")]
#[command(version = "0.3.2-beta")]
#[command(version = "0.3.2")]
struct Cli {
/// Path to the main git repository containing datasets as subdirectories
#[arg(help = "Repository path (defaults to current directory)")]
Expand Down
2 changes: 2 additions & 0 deletions src/git/versioned_store/tests.rs
Original file line number Diff line number Diff line change
Expand Up @@ -220,6 +220,8 @@ mod proof_tests {
#[cfg(test)]
mod tests {
use crate::git::types::DiffOperation;
#[cfg(feature = "rocksdb_storage")]
use crate::git::versioned_store::RocksDBVersionedKvStore;
use crate::git::versioned_store::{
FileVersionedKvStore, GitVersionedKvStore, HistoricalAccess, HistoricalCommitAccess,
InMemoryVersionedKvStore, ThreadSafeGitVersionedKvStore,
Expand Down
2 changes: 1 addition & 1 deletion src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ limitations under the License.
//!
//! ```toml
//! [dependencies]
//! prollytree = "0.3.2-beta"
//! prollytree = "0.3.2"
//! ```
//!
//! Follow examples in the github repository to get started.
Expand Down
Loading