Skip to content

Commit 776d4ed

Browse files
rliao147rliao147
authored
feat!: migrate UC integration to Delta Tables LoadTable and UpdateTable APIs (delta-io#2855)
## What changes are proposed in this pull request? This change deprecates the legacy UC Delta-Commits client with the new Delta-Tables API. This PR adds reads and DML support. CREATE TABLE and UC server E2E testing will be completed in follow up changes. Key pieces: 1. `unity-catalog-delta-client-api`: - deleted the Delta-Commits API old wire types / old client traits - modified the UpdateTableRequest 2. Added a new UpdateTableRestClient 3. `delta-kernel-unity-catalog`: - `UCCommitter` changed to use the new `UpdateTableClient` - `snapshot_builder_from_load_table` helper function 4. Other: - migrated the committer FFI bindings to the new UpdateTableClient trait - new doc updates ## How was this change tested? - new tests for UpdateTableClient, InMemoryUpdateTableClient, UCCommitter - existing UC tests modified to use the new Delta-Tables API - tests against the UC server above the stack pass. - existing UC tests pass
1 parent 5b75854 commit 776d4ed

48 files changed

Lines changed: 1299 additions & 1862 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/scripts/seed_uc_table.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
#!/usr/bin/env bash
2-
# Seed a catalog-managed Delta table in the running UC OSS server via its `bin/uc` CLI,
2+
# Seed a catalog-managed Delta table in the running UC server via its `bin/uc` CLI,
33
# so the gated `load_table` live test has a table to read.
44
#
55
# Usage: seed_uc_table.sh <catalog> <schema> <table>

‎.github/scripts/setup_unitycatalog.sh‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
#!/usr/bin/env bash
2-
# Clone and build the Unity Catalog OSS server at a pinned commit so the gated
3-
# unity-catalog-delta-rest-client live integration tests can run against a real server in CI.
2+
# Clone and build the Unity Catalog server at a pinned commit so the gated
3+
# unity-catalog-delta-client-default live integration tests can run against a real server in CI.
44
#
55
# Pinned to a commit that serves the Delta-Tables API the kernel UC crate targets. The published
66
# `:latest` artifacts/images predate these endpoints, so building from this commit is required.

.github/scripts/uc_oss_server.sh renamed to .github/scripts/uc_server.sh

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
#!/usr/bin/env bash
2-
# Start or stop the Unity Catalog OSS server built by setup_unitycatalog.sh, for the gated
3-
# unity-catalog-delta-rest-client live integration tests.
2+
# Start or stop the Unity Catalog server built by setup_unitycatalog.sh, for the gated
3+
# unity-catalog-delta-client-default live integration tests.
44
#
55
# Usage: uc_oss_server.sh {start|stop}
66
# Overridable via env: UC_DIR, UC_HOST, UC_PID_FILE, UC_LOG_FILE.
@@ -10,7 +10,7 @@ UC_DIR="${UC_DIR:-$HOME/unitycatalog}"
1010
UC_HOST="${UC_HOST:-http://localhost:8080}"
1111
PID_FILE="${UC_PID_FILE:-$UC_DIR/uc-server.pid}"
1212
LOG_FILE="${UC_LOG_FILE:-$UC_DIR/uc-server.log}"
13-
# Used to poll the OSS UC server to see when it is ready to accept requests.
13+
# Used to poll the UC server to see when it is ready to accept requests.
1414
HEALTH_URL="$UC_HOST/api/2.1/unity-catalog/catalogs"
1515

1616
start() {

.github/workflows/unitycatalog_oss_test.yml renamed to .github/workflows/unitycatalog_test.yml

Lines changed: 16 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -1,26 +1,26 @@
1-
name: unity-catalog-oss-test
1+
name: unity-catalog-test
22

3-
# Runs the gated `unity-catalog-delta-rest-client` live integration tests against a real Unity
4-
# Catalog OSS server (built from a pinned commit). Exercises the read-only `get_config` handshake
3+
# Runs the gated `unity-catalog-delta-client-default` live integration tests against a real Unity
4+
# Catalog server (built from a pinned commit). Exercises the read-only `get_config` handshake
55
# (needs no table) and `load_table` against a table seeded via the Delta-Commits (legacy) API
66
# before the test step, plus the mutating create tests (UC_CREATE) against the ephemeral server.
77
on:
88
push:
99
branches: [main, "release/**"]
1010
paths:
1111
- "unity-catalog-delta-client-api/**"
12-
- "unity-catalog-delta-rest-client/**"
13-
- ".github/workflows/unitycatalog_oss_test.yml"
12+
- "unity-catalog-delta-client-default/**"
13+
- ".github/workflows/unitycatalog_test.yml"
1414
- ".github/scripts/setup_unitycatalog.sh"
15-
- ".github/scripts/uc_oss_server.sh"
15+
- ".github/scripts/uc_server.sh"
1616
- ".github/scripts/seed_uc_table.sh"
1717
pull_request:
1818
paths:
1919
- "unity-catalog-delta-client-api/**"
20-
- "unity-catalog-delta-rest-client/**"
21-
- ".github/workflows/unitycatalog_oss_test.yml"
20+
- "unity-catalog-delta-client-default/**"
21+
- ".github/workflows/unitycatalog_test.yml"
2222
- ".github/scripts/setup_unitycatalog.sh"
23-
- ".github/scripts/uc_oss_server.sh"
23+
- ".github/scripts/uc_server.sh"
2424
- ".github/scripts/seed_uc_table.sh"
2525
workflow_dispatch:
2626

@@ -35,8 +35,8 @@ env:
3535
UC_DIR: ${{ github.workspace }}/.uc-server
3636

3737
jobs:
38-
uc-oss-test:
39-
name: UC OSS server integration test
38+
uc-test:
39+
name: UC server integration test
4040
runs-on: ubuntu-latest
4141
timeout-minutes: 45
4242
steps:
@@ -68,13 +68,13 @@ jobs:
6868
~/.ivy2
6969
~/.sbt
7070
~/.cache/coursier
71-
key: uc-oss-${{ runner.os }}-${{ env.UC_COMMIT }}-${{ hashFiles('.github/scripts/setup_unitycatalog.sh') }}
71+
key: uc-${{ runner.os }}-${{ env.UC_COMMIT }}-${{ hashFiles('.github/scripts/setup_unitycatalog.sh') }}
7272

7373
- name: Build Unity Catalog server
7474
run: bash .github/scripts/setup_unitycatalog.sh
7575

7676
- name: Start Unity Catalog server
77-
run: bash .github/scripts/uc_oss_server.sh start
77+
run: bash .github/scripts/uc_server.sh start
7878

7979
# Seed a catalog-managed (MANAGED) Delta table so the load_table live test has a table to read.
8080
- name: Seed a managed table
@@ -84,16 +84,16 @@ jobs:
8484
env:
8585
UC_SERVER_URL: http://localhost:8080
8686
UC_TOKEN: not-used
87-
# Fresh OSS server seeds the `unity` catalog with a `default` schema.
87+
# Fresh server seeds the `unity` catalog with a `default` schema.
8888
UC_TEST_CATALOG: unity
8989
UC_TEST_SCHEMA: default
9090
UC_TEST_TABLE: smoke_test_table
9191
UC_CREATE: "1"
92-
run: cargo nextest run --locked -p unity-catalog-delta-rest-client --features integration-test -E 'test(live_)'
92+
run: cargo nextest run --locked -p unity-catalog-delta-client-default --features integration-test -E 'test(live_)'
9393

9494
- name: Stop Unity Catalog server
9595
if: always()
96-
run: bash .github/scripts/uc_oss_server.sh stop
96+
run: bash .github/scripts/uc_server.sh stop
9797

9898
- name: Dump server log on failure
9999
if: failure()

‎CHANGELOG.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -857,7 +857,7 @@
857857
2. Add create many API to engine ([#2070])
858858
- Adds `create_many` method to `ParquetHandler` trait. Implementors must add this method. See the trait rustdocs for details.
859859
3. Rename uc-catalog and uc-client crates ([#2136])
860-
- `delta-kernel-uc-catalog` renamed to `delta-kernel-unity-catalog`. `delta-kernel-uc-client` renamed to `unity-catalog-delta-rest-client`. Update `Cargo.toml` dependencies accordingly.
860+
- `delta-kernel-uc-catalog` renamed to `delta-kernel-unity-catalog`. `delta-kernel-uc-client` renamed to `unity-catalog-delta-client-default`. Update `Cargo.toml` dependencies accordingly.
861861
4. Checksum and checkpoint APIs return updated Snapshot ([#2182])
862862
- `Snapshot::checkpoint()` and checksum APIs now return the updated `Snapshot`. Callers must handle the returned value.
863863
5. Add P&M to CommitMetadata and enforce committer/table type matching ([#2250])

‎CLAUDE.md‎

Lines changed: 17 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -48,19 +48,19 @@ cargo +nightly fmt \
4848

4949
### Crate Names for `-p` Flag
5050

51-
| Crate | Directory | Description |
52-
|------------------------------------------|--------------------------------------------|---------------------------------------------------------|
53-
| `delta_kernel` | `kernel/` | Core library |
54-
| `delta_kernel_default_engine` | `default-engine/` | Default Arrow/Tokio `Engine` implementation |
55-
| `delta_kernel_ffi` | `ffi/` | C/C++ FFI bindings |
56-
| `delta_kernel_derive` | `derive-macros/` | Proc macros |
57-
| `acceptance` | `acceptance/` | Acceptance tests (DAT) |
58-
| `test_utils` | `test-utils/` | Shared test utilities |
59-
| `delta_kernel_workloads` | `workloads/` | Shared workload spec types + SQL predicate parser |
60-
| `feature_tests` | `feature-tests/` | Feature flag tests |
61-
| `delta-kernel-unity-catalog` | `delta-kernel-unity-catalog/` | Unity Catalog integration (UCKernelClient, UCCommitter) |
62-
| `unity-catalog-delta-client-api` | `unity-catalog-delta-client-api/` | Unity Catalog client traits and shared models |
63-
| `unity-catalog-delta-rest-client` | `unity-catalog-delta-rest-client/` | Unity Catalog REST client |
51+
| Crate | Directory | Description |
52+
|-----------------------------------|------------------------------------|-----------------------------------------------------------|
53+
| `delta_kernel` | `kernel/` | Core library |
54+
| `delta_kernel_default_engine` | `default-engine/` | Default Arrow/Tokio `Engine` implementation |
55+
| `delta_kernel_ffi` | `ffi/` | C/C++ FFI bindings |
56+
| `delta_kernel_derive` | `derive-macros/` | Proc macros |
57+
| `acceptance` | `acceptance/` | Acceptance tests (DAT) |
58+
| `test_utils` | `test-utils/` | Shared test utilities |
59+
| `delta_kernel_workloads` | `workloads/` | Shared workload spec types + SQL predicate parser |
60+
| `feature_tests` | `feature-tests/` | Feature flag tests |
61+
| `delta-kernel-unity-catalog` | `delta-kernel-unity-catalog/` | Unity Catalog integration (UCCommitter, snapshot helpers) |
62+
| `unity-catalog-delta-client-api` | `unity-catalog-delta-client-api/` | Unity Catalog client traits and shared models |
63+
| `unity-catalog-delta-client-default` | `unity-catalog-delta-client-default/` | Unity Catalog REST client |
6464

6565
### Feature Flags
6666

@@ -332,6 +332,10 @@ Keep this list updated when new protocol features are added to kernel.
332332
data-dependent schema manipulation.
333333
- NEVER panic in production code -- use errors instead. Panicking
334334
(including `unwrap()`, `expect()`, `panic!()`, `unreachable!()`, etc) is acceptable in test code only.
335+
- Order a file so the most important APIs and impls come first; put private helper functions
336+
toward the bottom. A reader scanning top to bottom should hit the public surface before the
337+
private plumbing. (Order-sensitive items like `macro_rules!` used within the file are exempt --
338+
they must precede their use.)
335339

336340
## Comment & Doc Style
337341

‎Cargo.lock‎

Lines changed: 3 additions & 10 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Cargo.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ members = [
1212
"workloads",
1313
"feature-tests",
1414
"mem-test",
15-
"unity-catalog-delta-rest-client", # WIP: this is an experimental UC client for catalog-managed table work
15+
"unity-catalog-delta-client-default", # WIP: this is an experimental UC client for catalog-managed table work
1616
"delta-kernel-unity-catalog", # WIP: this is an experimental UC catalog implementation
1717
"unity-catalog-delta-client-api", # WIP: transport-agnostic UC client API traits
1818
]

‎benchmarks/Cargo.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,7 @@ serde = { version = "1", features = ["derive"] }
3232
serde_json = "1"
3333
tokio = { version = "1", features = ["rt-multi-thread"] }
3434
unity-catalog-delta-client-api = { path = "../unity-catalog-delta-client-api" }
35-
unity-catalog-delta-rest-client = { path = "../unity-catalog-delta-rest-client" }
35+
unity-catalog-delta-client-default = { path = "../unity-catalog-delta-client-default" }
3636
url = { version = "2", features = ["serde"] }
3737

3838
[dev-dependencies]

0 commit comments

Comments
 (0)