Skip to content

Cut iceberg test cases for remote catalogs. - #13962

Merged
liurenjie1024 merged 4 commits into
NVIDIA:release/25.12from
liurenjie1024:ray/13956
Dec 5, 2025
Merged

Cut iceberg test cases for remote catalogs.#13962
liurenjie1024 merged 4 commits into
NVIDIA:release/25.12from
liurenjie1024:ray/13956

Conversation

@liurenjie1024

Copy link
Copy Markdown
Collaborator

Related #13956.

Description

This pr has following changes:

  1. Cut a lot iceberg tests when remote catalog (rest catalog, s3tables) is enabled, while only keeping essential test cases for each operations. This reduced test time to 40 min with s3tables. This change is reasonable since for remote catalogs the key different is io and table commit operations, we don't need as much coverage as local disk tests.
  2. Further reduce eq deletion tests by reducing eq deletes combinations.

Checklists

  • This PR has added documentation for new or modified features or behaviors.
  • This PR has added new tests or modified existing tests to cover new code paths.
    (Please explain in the PR description how the new code paths are tested, such as names of the new/existing tests that cover them.)
  • Performance testing has been performed and its results are added in the PR description. Or, an issue has been filed with a link in the PR description.

Signed-off-by: liurenjie1024 <liurenjie2008@gmail.com>
Signed-off-by: liurenjie1024 <liurenjie2008@gmail.com>
@liurenjie1024 liurenjie1024 changed the title Ray/13956 Cut iceberg test cases for remote catalogs. Dec 5, 2025
Signed-off-by: liurenjie1024 <liurenjie2008@gmail.com>
@greptile-apps

greptile-apps Bot commented Dec 5, 2025

Copy link
Copy Markdown
Contributor

Greptile Overview

Greptile Summary

This PR optimizes Iceberg integration test execution for remote catalogs (REST catalog, S3 Tables) to address critical timeout issues. The changes implement strategic test reduction by adding conditional skipping of comprehensive test coverage when remote catalogs are detected, while preserving essential functionality tests. The key modifications include splitting existing comprehensive tests into basic versions (that run for all catalogs) and full coverage versions (skipped for remote catalogs), extracting helper functions to reduce code duplication, and reducing equality delete column combinations from 3 to 2. This optimization reduces test execution time from 24+ hours to approximately 40 minutes for S3 Tables. The approach is architecturally sound because remote catalogs primarily differ in I/O and table commit operations rather than core GPU acceleration logic, making extensive combinatorial testing less valuable while maintaining adequate coverage for the differentiating functionality.

Important Files Changed

Filename Score Overview
integration_tests/src/main/python/iceberg/iceberg_ctas_test.py 4/5 Refactors CTAS tests by splitting comprehensive partition tests into basic and full coverage versions with conditional remote catalog skipping
integration_tests/src/main/python/iceberg/iceberg_update_test.py 4/5 Implements strategic test skipping for remote catalogs and splits partition tests into basic/full coverage variants
integration_tests/src/main/python/iceberg/iceberg_append_test.py 5/5 Adds conditional test skipping for remote catalogs with clean separation of basic vs comprehensive partition test coverage
integration_tests/src/main/python/iceberg/iceberg_rtas_test.py 4/5 Optimizes RTAS tests by extracting helper functions and splitting partition tests for conditional remote catalog execution
integration_tests/src/main/python/iceberg/iceberg_overwrite_static_test.py 5/5 Refactors static overwrite tests with helper function extraction and conditional skipping for improved remote catalog performance
integration_tests/src/main/python/iceberg/iceberg_overwrite_dynamic_test.py 5/5 Implements clean test optimization with basic/full coverage split and strategic remote catalog skipping
integration_tests/src/main/python/iceberg/iceberg_delete_test.py 4/5 Reduces delete test coverage for remote catalogs while maintaining essential functionality testing
integration_tests/src/main/python/iceberg/iceberg_merge_test.py 4/5 Optimizes merge tests by splitting into basic and comprehensive versions with remote catalog conditional skipping
integration_tests/src/main/python/iceberg/init.py 5/5 Reduces equality delete column combinations from 3 to 2 to address combinatorial explosion in test execution time

Confidence score: 4/5

  • This PR addresses a critical CI timeout issue with a well-reasoned approach that maintains essential test coverage while dramatically improving performance
  • Score reflects that while the changes are extensive across multiple test files, they follow a consistent pattern and the logic for remote catalog optimization is sound based on architectural understanding
  • Pay close attention to the test splitting logic in files like iceberg_merge_test.py and iceberg_delete_test.py to ensure the basic coverage tests still validate core functionality adequately

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

9 files reviewed, no comments

Edit Code Review Agent Settings | Greptile

@liurenjie1024

Copy link
Copy Markdown
Collaborator Author

build

@res-life res-life left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@liurenjie1024
liurenjie1024 merged commit 332ba8d into NVIDIA:release/25.12 Dec 5, 2025
101 of 106 checks passed
@sameerz sameerz added the test Only impacts tests label Dec 5, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test Only impacts tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants