Skip to content

[fast-ut] [reduced-it] [SkipRecovery] Restore Delta REORG coverage on Spark 4.1.1+ [databricks] #15480

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wjxiz1992 wants to merge 1 commit into
base: main
Choose a base branch
from
+5 −2
Open

[fast-ut] [reduced-it] [SkipRecovery] Restore Delta REORG coverage on Spark 4.1.1+ [databricks] #15480

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions integration_tests/src/main/python/delta_lake_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@
from marks import allow_non_gpu, delta_lake, ignore_order
from parquet_test import reader_opt_confs_no_native
from parquet_test_utils import parquet_row_group_midpoints
from spark_session import is_spark_411_or_later
from spark_session import with_cpu_session, with_gpu_session, is_databricks_runtime, \
is_spark_320_or_later, is_spark_340_or_later, supports_delta_lake_deletion_vectors, is_spark_401_or_later, \
is_before_spark_353, is_databricks173_or_later
Comment on lines +24 to 27
Expand Down Expand Up @@ -823,8 +824,10 @@ def do_delete_and_disable_DV(spark, data_path):
reason="Deletion vector scan is not supported on Databricks")
@pytest.mark.skipif(is_before_spark_353(),
reason="Spark-RAPIDS supports scan with deletion vectors starting in Spark 3.5.3")
@pytest.mark.skipif(is_spark_401_or_later(),
reason="REORG is not supported in Spark 4.0.1+ (https://github.com/delta-io/delta/issues/5690)")
@pytest.mark.skipif(is_spark_401_or_later() and not is_spark_411_or_later(),
reason="REORG requires Delta 4.1.0, available in "
"supported Spark 4.1.1+ profiles "
"(https://github.com/delta-io/delta/issues/5690)")
def test_delta_scan_split_with_DV_enabled_after_DVs_materialized(spark_tmp_path):
def do_delete_and_reorg(spark, data_path):
num_deleted = spark.sql(f"DELETE FROM delta.`{data_path}` WHERE a = 0").collect()[0][0]
Expand Down
Loading