[BUG] Delta Lake deletion vector tests failing on Spark != 3.5 due to version incompatibilities

[pxLi]

Describe the bug
Build:
rapids_it-matrix-pre_release-github/201
rapids_it-arm64-pre_release/282
rapids_it-AQE-pre_release-github/505

New Delta Lake deletion vector tests added in commit 9c95eaa are failing across multiple spark versions with version-specific errors:

• Spark 3.2.4: DELETE command returns empty list causing IndexError when accessing result[0][0]
• Spark 3.3.4: (a) Partition split assertion fails (2 vs 1 expected), (b) delta.enableDeletionVectors config not recognized, (c) REORG command syntax not supported
• Spark 3.4.4: IllegalArgumentException when reading __delta_internal_is_row_deleted metadata column due to ColumnarToRow execution plan issue
• Spark 4.0.0: test_delta_filter_out_metadata_col - deletion vector metadata column __delta_internal_is_row_deleted not found in CPU query plan
• Spark 4.0.1: test_delta_scan_split_with_DV_enabled_after_DVs_materialized - NoSuchMethodError for ParquetToSparkSchemaConverter during REORG PURGE

Tests affected:

• test_delta_scan_split_with_DV_enabled_with_DVs
• test_delta_scan_split_with_DV_disabled_with_DVs
• test_delta_scan_split_with_DV_enabled_after_DVs_materialized
• test_delta_filter_out_metadata_col

spark 3xx Error logs:

[Spark 3.2.4] IndexError: list index out of range at delta_lake_test.py:169
[Spark 3.3.4] AssertionError: Expected 1 partitions, got 2 at delta_lake_test.py:144
[Spark 3.3.4] AnalysisException: Unknown configuration: delta.enableDeletionVectors
[Spark 3.3.4] ParseException: Syntax error 'REORG' at delta_lake_test.py:195
[Spark 3.4.4] IllegalArgumentException: Part of plan not columnar ColumnarToRowExec

spark 40x Error logs:

Spark 4.0.0: assert '__delta_internal_is_row_deleted' in query plan failed
Spark 4.0.1: java.lang.NoSuchMethodError: 'void org.apache.spark.sql.execution.datasources.parquet.ParquetToSparkSchemaConverter.<init>(boolean, boolean, boolean, boolean, boolean)' at ReorgTableHelper.filterParquetFiles

Environment details
Spark versions: 3.2.x, 3.3.x, 3.4.x, 4.0.x
Test framework: pytest with GPU integration tests
Delta Lake version: varies by Spark version

[revans2]

@razajafri @jihoonson @gerashegalov this is blocking CI right now. I am going to put up a PR to disable the failing test. Not really sure what happened because I see 3 days ago CI/CD passed for this, even for open source versions that appear to be failing now.

[revans2]

@pxLi @razajafri @jihoonson @gerashegalov Some of the failing tests, like test_delta_filter_out_metadata_col were not changed as a part of 9c95eaa. That test was changed as a part of 01090d8

Which looks like there may have been a conflict between #13991 and #13980, but I am not sure. Both went in at the same time, which means they were not tested with each other in place.