Skip to content

CLAUDE.md

Latest commit

 

History

History
119 lines (86 loc) · 7.52 KB

CLAUDE.md

File metadata and controls

119 lines (86 loc) · 7.52 KB

Zerobus SDK — Monorepo Guide

Architecture

This is a polyglot SDK monorepo. A single Rust core (rust/sdk/) implements gRPC streaming, OAuth, recovery, and ingestion logic. Four wrapper SDKs expose it to other languages:

SDK Directory Binding mechanism Build system
Rust rust/ Native Cargo workspace
Python python/ PyO3 / maturin maturin + pip
TypeScript typescript/ NAPI-RS npm + napi-rs
Java java/ JNI (rust/jni/) Maven
Go go/ cgo + static FFI lib Go modules

The data flow is: User code → Wrapper API → FFI boundary → Rust core → gRPC → Zerobus service.

Go uses the C FFI layer (rust/ffi/, header auto-generated by cbindgen). Java has its own JNI crate (rust/jni/). Python and TypeScript bind Rust directly via PyO3 and NAPI-RS respectively.

Versioning and Breaking Changes

Strict semver. Breaking changes are only allowed in major version bumps.

  • Removing or renaming a public API element is a breaking change. Mark it #[deprecated] (or language equivalent) in a minor release first, then remove in the next major.
  • Changing the FFI C header (rust/ffi/zerobus.h) in a non-additive way breaks Go and Java. Treat FFI signature changes with the same rigor as public API changes.
  • Each SDK is versioned independently. A Rust core bump does not automatically require wrapper version bumps unless the wrapper's public API changes.
  • Release tags follow <sdk>/v<semver> (e.g., rust/v1.1.0, python/v1.0.0).

Cross-FFI Concerns

Because every non-Rust SDK crosses a foreign-function boundary, keep these in mind:

Performance

  • Every call across FFI serializes/deserializes data. Prefer batch APIs (ingest_records) over single-record APIs in hot paths.
  • Proto descriptors and Arrow IPC schemas are serialized as byte arrays across the boundary. These are typically one-time costs at stream creation.
  • Arrow Flight uses IPC format across FFI for near-zero-copy transfer; avoid unnecessary re-encoding.

Memory safety

  • Rust owns the memory for SDK and stream handles. Wrappers hold opaque pointers and must call the corresponding free/close/destroy function.
  • C-allocated error strings (CResult.error_message) must be freed by the caller via zerobus_free_error_message().
  • Go uses runtime.Pinner to prevent GC from relocating memory while Rust holds a pointer. Never remove pinner calls.
  • Go's cgo.Handle registry keeps HeadersProvider callbacks alive; leaking these handles leaks Go objects.
  • Java has no finalizers on streams — users must call close() or use try-with-resources. Forgetting this leaks JNI resources.
  • Python and TypeScript rely on GC-triggered cleanup via PyO3/NAPI-RS reference counting.

Thread safety

  • Rust core is async (tokio). The FFI layer manages its own tokio runtime.
  • Go: safe for concurrent goroutines.
  • Python: async streams are safe; sync streams are single-threaded per instance.
  • TypeScript: async-safe via Node.js event loop.
  • Java: not thread-safe — external synchronization required for concurrent access.

Build and Test Commands

Each SDK has its own build/test commands. Run from the SDK directory:

SDK Build Test Lint Format
Rust make build make test make lint make fmt
Python make build-rust make test make lint make fmt
TypeScript npm run build npm test cargo clippy cargo fmt
Java mvn compile mvn test mvn spotless:check mvn spotless:apply
Go make build make test make lint make fmt

CI/CD

  • push.yml is the main gate — uses path filtering to run only affected SDK CI jobs.
  • Cross-SDK CI (non-blocking): when rust/** changes, wrapper SDK tests also run against local Rust source to catch FFI breakage early.
  • Release workflows are tag-triggered (see Release Process below).

Changelog Process

Every SDK has two changelog files:

  • NEXT_CHANGELOG.md — Accumulates entries for the upcoming release. Every PR that changes user-facing behavior must add an entry here under the appropriate section (New Features, Bug Fixes, Breaking Changes, Deprecations, API Changes, Documentation, Internal Changes).
  • CHANGELOG.md — Finalized release history. When a version-bump PR is merged, the contents of NEXT_CHANGELOG.md are prepended to CHANGELOG.md and NEXT_CHANGELOG.md is reset to an empty template with the next version header.

Rules:

  • If a PR adds a feature, fixes a bug, deprecates an API, or changes behavior — it must update NEXT_CHANGELOG.md. No exceptions.
  • Breaking changes must be called out under ### Breaking Changes with migration instructions.
  • Deprecations must include what to use instead and the planned removal timeline.

Documentation Requirements

Every PR that changes user-facing behavior must also update:

  • README — If the change affects usage, setup, or API surface, update the SDK's README.md.
  • Examples — If the change adds a new API or modifies an existing one, add or update examples in the SDK's examples/ directory.
  • API docs — Rust: doc comments. Python: docstrings + type stubs. TypeScript: JSDoc. Java: Javadoc. Go: godoc comments.

Release Process

Releases are per-SDK and tag-triggered:

  1. Version bump PR: Update the version in the SDK's config file (Cargo.toml, package.json, pom.xml, etc.). Move NEXT_CHANGELOG.md contents into CHANGELOG.md. Reset NEXT_CHANGELOG.md with the next planned version header.
  2. Merge to main.
  3. Tag: Push a tag matching <sdk>/v<semver> (e.g., rust/v1.2.0, python/v1.1.0). This triggers the release workflow.
  4. Release workflow:
    • Builds platform-specific artifacts (5 platforms: Linux x86_64/aarch64, macOS x86_64/arm64, Windows x86_64).
    • Publishes to the SDK's package registry (crates.io, PyPI, npm, Maven Central, or git tag for Go).
    • Creates a GitHub Release with release notes extracted from CHANGELOG.md.

SDK-specific release details:

SDK Version source Registry Tag pattern
Rust rust/sdk/Cargo.toml crates.io rust/v*
Python Derived from Cargo.toml via maturin PyPI python/v*
TypeScript typescript/package.json npm typescript/v*
Java java/pom.xml Maven Central java/v*
Go Git tag (no version file) pkg.go.dev go/v*
FFI rust/ffi/Cargo.toml GitHub Release ffi/v*

The FFI library has its own release cycle because Go and Java depend on pre-built static/dynamic libraries. An FFI release must happen before Go/Java can pick up Rust core changes.

Commit Conventions

  • Prefix commit messages with [SDK] when changes are scoped to one SDK (e.g., [Rust], [Python], [All]).
  • Signed commits required (DCO). Use git commit -s.
  • Present tense, imperative mood. Keep subject line under 50 chars.