Skip to content

[AutoSparkUT] Migrate ten Spark SQL suites #15452

Description

@wjxiz1992

Coverage gap

Ten eligible Apache Spark 3.3 SQL suites are not registered in the RAPIDS migrated-suite matrix:

  • DataFrameReaderWriterSuite
  • InsertSuite
  • BucketedReadWithoutHiveSupportSuite
  • InMemoryColumnarQuerySuite
  • PathFilterSuite
  • DataSourceV2Suite
  • JsonParsingOptionsSuite
  • PartitionedWriteSuite
  • V1WriteFallbackSuite
  • GlobalTempViewSuite

The suites exercise DataFrame/SQL read, write, cache, bucket, DataSource V2, JSON, partition, fallback, and global-view execution paths.

Local validation

The initial Spark 3.3 GPU run executed 250 tests: 221 succeeded, 29 failed, and 2 were ignored. The failures were isolated to CPU-specific exception, scan-plan, and cache-implementation assertions.

After remediation:

  • Added RAPIDS-aware replacements for the 3 InsertSuite exception cases and DataSourceV2 partitioning reporting.
  • Deferred 12 bucket scan-plan assertions and 13 Spark cache-serializer/statistics assertions with explicit P1 recovery triggers.
  • Final combined Spark 3.3 GPU gate: Tests: succeeded 225, failed 0, canceled 0, ignored 31, pending 0.

The 31 ignored cases comprise 2 upstream-preexisting ignores plus 29 explicitly registered migration exclusions; the 4 replaced cases execute and pass separately.

Intended scope

Add minimal RAPIDS wrappers and RapidsTestSettings registrations, preserve passing inherited coverage, add GPU-aware replacements where safe, and explicitly track CPU-private contracts that need dedicated follow-up.

Metadata

Metadata

Assignees

No one assigned

    Labels

    ? - Needs TriageNeed team to review and classifytestOnly impacts tests

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions