Skip to content

AGENTS.md

Latest commit

 

History

History
428 lines (366 loc) · 26.9 KB

AGENTS.md

File metadata and controls

428 lines (366 loc) · 26.9 KB
Raw

AGENTS.md

Project Overview

Delta-kernel-rs is a Rust library for building Delta Lake connectors. It encapsulates the Delta protocol so connectors can read and write Delta tables without understanding protocol internals. Kernel never does I/O directly: it defines what to do via its APIs (Snapshot, Scan, Transaction) and delegates how to the Engine trait.

Current capabilities include table reads with predicates, data skipping, deletion vectors, change data feed, incremental scans (incremental_scan_builder) and commit ranges, checkpoints (V1 & V2), version checksums, blind appends, file removals, table creation (including clustered tables), limited schema alteration, and catalog-managed tables. Log compaction remains disabled (#2337).

Build & Test Commands

datafusion-executor and integration-tests are separate workspaces. Root --workspace commands do not include them. For datafusion_executor commands, see datafusion-executor/CLAUDE.md. integration-tests/test-all-arrow-versions.sh tests each supported Arrow version.

# Build
cargo build --workspace --all-features

# Run all tests (prefer nextest over cargo test)
cargo nextest run --workspace --all-features

# Run tests for a specific crate
cargo nextest run -p delta_kernel --all-features

# Run a single test in a specific crate (fastest: only compiles that crate)
cargo nextest run -p delta_kernel --lib --all-features test_name_here

# Run a test by name, searching all crates (slow: compiles everything)
cargo nextest run --workspace --all-features test_name_here

# Format, lint, and doc check (always run after code changes)
cargo +nightly fmt \
  && cargo clippy --workspace --benches --tests --all-features -- -D warnings \
  && cargo doc --workspace --all-features --no-deps

# Split no-default-features CI checks (cargo aliases from .cargo/config.toml)
cargo clippy-no-default-kernel-dependents
cargo check-no-default-kernel
cargo check-no-default-engine
cargo clippy-no-default-kernel-leaves

# Quick pre-push check (mimics CI)
cargo +nightly fmt \
  && cargo clippy --workspace --benches --tests --all-features -- -D warnings \
  && cargo doc --workspace --all-features --no-deps \
  && cargo nextest run --workspace --all-features

Crate Names for -p Flag

Crate Directory Description
delta_kernel kernel/ Core library
delta_kernel_default_engine default-engine/ Default Arrow/Tokio Engine implementation
delta_kernel_default_engine_test_utils default-engine/test-utils/ Default-engine test utilities
delta_kernel_ffi ffi/ C/C++ FFI bindings
delta_kernel_ffi_macros ffi-proc-macros/ FFI proc macros
delta_kernel_derive derive-macros/ Proc macros
acceptance acceptance/ Acceptance tests (DAT)
test_utils test-utils/ Shared test utilities
delta_kernel_workloads workloads/ Shared workload spec types + SQL predicate parser
delta_kernel_benchmarks benchmarks/ Workload benchmarks
feature_tests feature-tests/ Feature flag tests
mem-test mem-test/ Memory-usage test executable
delta-kernel-unity-catalog delta-kernel-unity-catalog/ Unity Catalog integration (UCCommitter, snapshot + create-table helpers)
unity-catalog-delta-client-api unity-catalog-delta-client-api/ Transport-agnostic UC client traits + wire models
unity-catalog-delta-rest-client unity-catalog-delta-rest-client/ REST/HTTP client for the Unity Catalog Delta Tables API

Packages under kernel/examples/ are also workspace members. Use the package name from the example's Cargo.toml with -p.

Feature Flags

Some noteworthy ones (see [features] in kernel/Cargo.toml for the full list):

  • TLS backend selection (rustls / native-tls) lives on the delta_kernel_default_engine crate, not on kernel itself.
  • arrow, arrow-XX, arrow-YY: Arrow version selection (kernel tracks the latest two major Arrow releases; arrow defaults to latest). Kernel's core APIs are Arrow-independent; the Arrow dependencies are optional, while the default engine requires one version.
  • arrow-conversion, arrow-expression: Arrow interop (auto-enabled by default-engine-base)
  • prettyprint: enables Arrow pretty-print helpers (primarily test/example oriented)
  • schema-diff: experimental schema diffing
  • check-constraints-in-dev: enables the internal SQL tokenizer and single-comparison parser for check-constraint development
  • adaptive-metadata-in-dev: adaptiveMetadata (Iceberg V4 adaptive metadata tree) support (experimental, in development). Gates KernelSupport::Supported for the adaptiveMetadata-preview reader+writer feature. Without it, reads and writes to tables listing the feature are blocked.
  • geo-type-in-dev: geospatial type support (geometry and geography columns) (experimental, in development). Gates KernelSupport for the geospatial reader+writer feature: with the cargo feature off, any table listing it is rejected; with it on, scans and CDF are supported but writes are still blocked.
  • internal-api: unstable APIs like parallel_scan_metadata. Items are marked with the #[internal_api] proc macro attribute.
  • declarative-plans: experimental declarative-plan IR (kernel/src/plans/) and the prost proto wire format mirroring it (kernel/proto/). Auto-enables internal-api, but not Arrow.
  • vendored-protoc: supplies protoc for declarative-plans; without it, set PROTOC to a system or hermetic protobuf compiler.
  • test-utils, integration-test: development only (test-utils enables prettyprint)

Architecture at a Glance

Snapshot is the primary entry point for existing-table operations: an immutable view of a table at a specific version. From it you build a Scan (reads) or Transaction (writes).

Read path: Snapshot -> ScanBuilder -> Scan -> data. Execution paths: execute() (simple), scan_metadata() (advanced/distributed), parallel_scan_metadata() (two-phase distributed log replay).

Write path: Snapshot -> Transaction -> commit(). Writers call Transaction::write_state, then bind partition values through the returned WriteState to get a BoundWriteContext. Distributed writers can encode and transport the state before binding it. Kernel assembles commit actions, enforces protocol compliance, and delegates the atomic commit to a Committer.

Engine trait: exposes StorageHandler, JsonHandler, ParquetHandler, and EvaluationHandler, plus an optional PlanExecutor under declarative-plans. Metrics use tracing layers rather than an engine handler. DefaultEngine lives in default-engine/src/.

EngineData: opaque columnar data interface. NEVER access EngineData columns directly: ALWAYS use the visitor pattern (visit_rows with typed GetData accessors).

Testing

  • Unit tests test internal APIs and module internals. It is fine to use public APIs like create_table in a unit test as setup (e.g. to create a table for testing reads, writes, or state loading).
  • Integration tests exercise only public APIs end-to-end. See kernel/tests/README.md for a catalog of available test tables (schema, protocol, features, and which tests use them). Consult it before creating new test data to avoid duplication.
  • Consider TestTableBuilder (test_utils::table_builder) to build the table under test. Unlike create_table, which builds a single create transaction with a given set of features and data layout, the builder builds up a multi-version table: data files written across many commits, plus checkpoints, CRC files, a stale/missing _last_checkpoint hint, or post-cleanup logs. So when a test needs a populated table history in a specific state, the builder is often a good fit, composing LogState, FeatureSet, DataLayoutConfig, and TableConfig through the real kernel write path so the table is protocol-correct by construction. Load a snapshot at any VersionTarget with the build_snapshot! macro. For coverage across many table states, the default_sweep cross-product template (LogState x FeatureSet x (DataLayoutConfig, TableConfig) x VersionTarget: data layout and table config are bundled into one axis to avoid a cartesian explosion), or a per-axis template with your own #[values], can help; see kernel/tests/integration/cross_product/mod.rs. Drop to lower-level setup like test_table_setup (or hand-rolled add_commit / LocalMockTable) only when necessary: e.g. for states the builder cannot express, such as corrupt or malformed logs.
  • Consider how the feature interacts with Delta table features (see Protocol TLDR below).
  • Consider write paths: normal commits, checkpointing, CRC files, log compaction files.
  • When adding cloud-storage functionality to an engine, such as writing JSON files, make sure to test it against S3, Azure, and GCS.
  • Consider read paths: loading a snapshot from scratch at latest version, at a specific version (time travel), and updating from an existing snapshot.
  • Consider table state: only versioned JSON commits, after a checkpoint, after a version checksum (.crc) file, after log compaction, etc.
  • Prefer descriptive test names over doc comments. Encode the scenario and expected behavior in the test name. Only add a test doc comment when the intent is too verbose or complex to express succinctly in the name.
  • Use rstest to parameterize tests that share the same logic but differ in setup or inputs. Prefer #[case] over duplicating test functions. When parameters are independent and form a cartesian product, prefer #[values] over enumerating every combination with #[case].
  • Actively look for rstest consolidation opportunities: when writing multiple tests that share the same setup/flow and differ only in configuration and expected outcome, write one parameterized rstest instead of separate functions. Also check whether a new test duplicates the flow of an existing nearby test and should be merged into it as a new #[case]. A common pattern is toggling a feature (e.g. column mapping on/off) and asserting success vs. error.
  • Reuse helpers from test_utils and the integration-test fixtures instead of writing custom ones when possible. See Common test helpers below for a curated starter list.
  • Committing in tests: Use txn.commit(engine)?.unwrap_committed() to assert a successful commit and get the CommittedTransaction. When you only need the resulting snapshot, use txn.commit(engine)?.unwrap_post_commit_snapshot() to get the SnapshotRef directly. Do NOT use match + panic! for either: both helpers provide a clear error message on failure. Available under #[cfg(test)] and the test-utils feature.
  • Prefer snapshot/public API assertions over reading raw commit JSON. Only read raw commit JSON when the data is inaccessible via public API (e.g., system domain metadata is blocked by get_domain_metadata). For commit JSON reads, use read_actions_from_commit from test_utils: do NOT write local helpers that duplicate this.
  • add_commit and table setup in tests: add_commit takes a table_root string and resolves it to an absolute object-store path. The table_root must be a proper URL string with a trailing slash (e.g. "memory:///", "file:///tmp/my_table/"). Avoid using the Url type directly: most test helpers and kernel APIs accept impl AsRef<str>, so pass URL strings instead. When using local storage, use an un-prefixed store (LocalFileSystem::new()) with a file:/// URL string. Do NOT use LocalFileSystem::new_with_prefix() with add_commit: add_commit already resolves the full path from the URL, so the prefix causes double-nesting. For in-memory tests, use InMemory::new() with "memory:///". ALWAYS use the same table_root URL string for both add_commit (writing log files) and Snapshot::builder_for (reading the table). ALWAYS include a trailing slash in directory URLs to ensure correct path joining.

Common Test Helpers

Before writing a custom helper, check this curated list and the locations below. This list is non-exhaustive: when in doubt, browse the source files directly (test-utils/src/lib.rs, kernel/tests/integration/common/, kernel/tests/integration/<topic>/mod.rs).

Arrow construction (from delta_kernel::arrow)

  • arrow::array::new_null_array(&arrow_type, n): Arrow array of n nulls of any Arrow type. Prefer this over per-type Int32Array::from(vec![None as Option<i32>]) builders.
  • engine::arrow_conversion::TryIntoArrow: (&kernel_data_type).try_into_arrow() for DataType, (&kernel_struct_type).try_into_arrow() for StructType -> Arrow Schema.

Engine + table setup (from test_utils)

  • test_table_setup() / test_table_setup_mt(): engine + temp table path. Use the _mt variant under #[tokio::test(flavor = "multi_thread")]. Required whenever a test calls snapshot.checkpoint(): it issues nested block_on calls that deadlock on a single-threaded runtime / TokioBackgroundExecutor.
  • engine_store_setup(table_name, local_directory): returns (store, engine, table_location) when a test needs direct object-store access.
  • setup_test_tables(...): multiple pre-built tables for read/scan tests.

Table creation in tests

  • test_utils::table_builder::TestTableBuilder (and the test_table(...) shorthand): worth considering for a table in a specific state: composes LogState, FeatureSet, DataLayoutConfig, and TableConfig through the real write path. Pair with the build_snapshot! macro and, for broad coverage, the default_sweep template. See the Testing section above.
  • Prefer the kernel create_table builder (delta_kernel::transaction::create_table::create_table) when you need a single bespoke table rather than the builder's composed states. It exercises the same path connectors use and auto-derives the protocol from the schema and feature flags.
  • test_utils::create_table (a JSON helper that hand-rolls protocol + metadata) is older but still needed when the kernel builder cannot enable a particular feature combination.

Schema fixtures

  • test_utils: nested_schema, schema_with_type, nested_schema_with_type, multi_schema_with_type, top_level_ntz_schema / nested_ntz_schema / multiple_ntz_schema, top_level_variant_schema / nested_variant_schema / multiple_variant_schema.
  • kernel/tests/integration/create_table/mod.rs: simple_schema, partition_test_schema.

Commit + read helpers (from test_utils)

  • add_commit, add_staged_commit: write a JSON commit at a given version.
  • read_actions_from_commit: read raw JSON actions from a specific local-file commit. Use this instead of hand-rolled serde_json parsing.
  • test_read: full-scan read of a table; use for round-trip assertions.
  • into_record_batch: convert Box<dyn EngineData> to Arrow RecordBatch.

Assertion helpers (from test_utils)

  • assert_schema_has_field(schema, &["a".into(), "b".into()]): assert a (possibly nested) field path.
  • assert_result_error_with_message(result, "needle"): assert an error contains a substring.

If a name here doesn't match what's in code: the list may have drifted from a rename. Run rg '^pub (fn|async fn)' test-utils/src/lib.rs to discover the current public surface, and update this section in your PR. The same pattern works for kernel/tests/integration/common/write_utils.rs.

Protocol TLDR

The Delta protocol spec is the source of truth. Key concepts:

  • Actions: records in commits and checkpoints: Metadata, Add File, Remove File, Add CDC File, Protocol, CommitInfo, SetTransaction, Domain Metadata, Sidecar, Checkpoint Metadata, and the feature-gated adaptive metadata checkpoint action
  • Log structure: JSON commit files, checkpoints (V1 parquet, V2 multi-part), log compaction files, version checksum (CRC) files, _last_checkpoint
  • Protocol versioning: (readerVersion, writerVersion) pair. Reader version 3 requires readerFeatures; writer version 7 requires writerFeatures. Follow each feature's rules for activation, dependencies, and any permitted removal.
  • Data skipping: per-file column statistics (min, max, null count, row count) with tight/wide bounds
  • Schemas: JSON serialization format for StructType/StructField/DataType
  • Stats and partition values: per-file statistics are stored as a JSON-encoded string in the Add action's stats field. partitionValues is a JSON map from column names to serialized string or null values. The stats structure mirrors the table schema. See the protocol spec sections on "Per-file Statistics" and "Partition Value Serialization" for the exact formats.

Table features:

  • Writer: allowColumnDefaults, appendOnly, changeDataFeed, checkConstraints, clustering, domainMetadata, generatedColumns, icebergCompatV1, icebergCompatV2, icebergCompatV3, identityColumns, inCommitTimestamp, invariants, materializePartitionColumns, rowTracking
  • Reader + writer: adaptiveMetadata-preview, catalogManaged, catalogOwned-preview, columnMapping, deletionVectors, geospatial, timestampNtz, typeWidening, typeWidening-preview, v2Checkpoint, vacuumProtocolCheck, variantShredding, variantShredding-preview, variantType, variantType-preview

Keep this list updated when new protocol features are added to kernel.

Common Gotchas

  • EngineData is opaque: NEVER downcast to ArrowEngineData or any concrete type in production code (ok in tests). NEVER assume one batch per file: ALWAYS iterate.
  • Column mapping: Physical column names can differ from logical names. ALWAYS use the schema from Snapshot::schema() for user data columns. Metadata/system schema column names (defined by the protocol) are not subject to column mapping.
  • Transforms: Generic recursive schema and expression transform traits and helpers are in kernel/src/transforms/.
  • Tracing layer callbacks must not emit tracing events directly: Calling warn!() or any tracing macro inside a tracing_subscriber::Layer callback (on_event, on_record, on_close) while holding a span's extensions_mut() write lock will re-enter the layer and deadlock on the same lock. In on_new_span, no extension lock is held during attrs.record(), so direct warn!() is safe there. In on_record, store warnings in a pending_warnings: Vec<String> field on the visitor, take them out after the extensions block closes, and emit via warn!() only then. (on_event's visitor does no warning-eligible work, so it may run under the lock directly.) See kernel/src/metrics/reporter.rs for the canonical pattern.

Code Style

  • Line width is 100 characters. Wrap comments and string literals at 100, not 80.
  • Place use imports at the top of the file (for non-test code) or at the top of the mod tests block (for test code): never inside function bodies.
  • Prefer == over matches! for simple single-variant enum comparisons. matches! is for patterns with bindings or guards. For example: self == Variant not matches!(self, Variant).
  • Prefer #[repr(C)] enums for closed FFI choice sets instead of integer aliases and constants. Add cbindgen:prefix-with-name=true so generated variants remain unambiguous. Use an integer discriminator only when unknown values are intentionally recoverable, and validate them at the boundary. Invalid enum tags are undefined behavior, so unsafe FFI APIs must require valid tags.
  • Prefer StructField::nullable / StructField::not_null over StructField::new(name, type, bool) when nullability is known at compile time. Reserve StructField::new for cases where nullability is a runtime value.
  • Leverage impl Into<DataType> to avoid DataType::Struct/Array/Map(Box::new(...)) boilerplate. StructType, ArrayType, and MapType all implement Into<DataType>, and constructors like StructField::new/nullable/not_null, ArrayType::new, and MapType::new accept impl Into<DataType>. So:
    • When passing to a parameter that accepts impl Into<DataType>, pass the container type directly: StructField::nullable("a", ArrayType::new(DataType::INTEGER, true)); do NOT wrap in DataType::from(...) or .into() (redundant at best, and an ambiguous-type compile error at worst).
    • When a concrete DataType value is actually required (e.g. a DataType-typed binding/field, a [DataType]/Vec<DataType> element, or a &DataType argument), prefer DataType::from(ArrayType::new(...)) over DataType::Array(Box::new(ArrayType::new(...))).
  • Prefer the DeltaResultIterator<'a, T> / DeltaResultIteratorStatic<T> aliases over hand-rolled Box<dyn Iterator<Item = DeltaResult<T>> + Send (+ 'a)>.
  • Prefer the lit / null_lit constructors over Expression::literal(...) / lit(Scalar::Null(...)) when building expressions inline. They take impl Into<Scalar> and impl Into<DataType>, respectively. Prefer Predicate::TRUE / FALSE / NULL for predicates whose value is statically known, reserving Predicate::literal(b) for runtime bool values.
  • Prefer the col! macro and lit(value) constructor over Expression::column(...) / Expression::literal(...) when building expressions inline. col! uses the same compile-time segment rules as column_name! (string literals split on .; constants are single simple segments). Use Expression::column([...]) for runtime or non-simple names. (column_expr! is a doc-hidden compatibility alias of col!.)
  • Prefer the schema! / schema_ref! macros for inline declarative schema literals, lazy_schema_ref! for LazyLock<SchemaRef> statics, and try_schema! when names of interpolated fields might collide. For Delta log action schemas, reuse the canonical *_FIELD and LOG_*_SCHEMA statics from actions instead of re-declaring StructField::nullable(ACTION_NAME, Action::to_schema()) or projecting from get_commit_schema(). Prefer StructType::try_new or schema builder/patch APIs for complex data-dependent schema manipulation.
  • NEVER panic in production code: use errors instead. Panicking (including unwrap(), expect(), panic!(), unreachable!(), etc) is acceptable in test code only.
  • Order a file so the most important APIs and impls come first; put private helper functions toward the bottom. Within that, order by visibility: pub first, then pub(crate), then private. A reader scanning top to bottom should hit the public surface before the private plumbing. (Order-sensitive items like macro_rules! used within the file are exempt: they must precede their use.)

Comment & Doc Style

  • MUST include doc comments for all public functions, structs, enums, and methods.
  • MUST document function parameters, return values, and errors.
  • Doc comments focus on "what" (contract with caller) more than "how" (implementation), unless the "how" meaningfully impacts the "what".
  • Code comments state intent and explain "why": don't restate what the code self-documents.
  • Be succinct. No verbose AI-slop comments. With well-written and well-named code, verbose comments are worse than none.
  • Say each thing once, in the right place: don't repeat the same idea across doc comment and inline comment.
  • Comments earn their place only for hidden invariants, real-bug workarounds, or constraints the reader can't see from the code itself.
  • Don't enumerate what grep can answer. Lists like // Used by a, b, c rot the moment d lands. Describe the shape; let the reader grep.
  • No stale-prone anchors in durable docs or source comments: counts ("the 10 variants", "5-arm match"), line numbers, or enumeration lists. Describe the shape; let the reader grep.
  • Comments MUST NOT include temporal references: only refer to current code and design, not past iterations.
  • Keep comments up-to-date with code changes.
  • Include examples in doc comments for complex functions only.
  • Use == as a visual section divider in comments (e.g. // === Helpers === or // ============).
  • NEVER use emoji or unicode in comments that emulates emoji (e.g. special arrows, checkmarks). Use ASCII equivalents (->, =>, etc.) instead.

Pull Requests

Title: use conventional commit format, lowercase after prefix, no period at the end. Allowed types: feat, fix, refactor, chore, docs, perf, test, ci. If the pull request contains a breaking change, the type must have a ! suffix. Examples: feat: add checkpoint stream support, fix: handle empty log segment, refactor: extract common log replay logic Breaking change examples: feat!: make_physical takes column mapping and sets parquet field ids, chore!: remove the arrow-55 feature

Description: follow the template in .github/PULL_REQUEST_TEMPLATE.md. Err on the side of simplicity: don't list every change. Focus on key API changes, functionality, and data flow. Keep it concise.

Deep Context

Read these when relevant to the task at hand:

  • CLAUDE/architecture.md: kernel architecture: snapshot loading, read/write paths, engine trait system, EngineData, key modules, catalog-managed tables
  • docs/user-guide/CLAUDE.md: writing standards for the mdBook user guide
  • Always cross-check protocol behavior against the Delta protocol spec

Keeping docs current: If you notice renamed structs, traits, functions, modules, crates, APIs, stale data flows, or wrong file paths in these docs, inform the user so they can be updated. After major changes, update this file, CLAUDE/architecture.md, ffi/CLAUDE.md, .github/CLAUDE.md, and any relevant <crate>/CLAUDE.md files.