Skip to content

Fix a GBK test failure on DB runtimes[databricks] - #13808

Closed
firestarman wants to merge 1 commit into
NVIDIA:release/25.12from
firestarman:fix-db-gbk
Closed

Fix a GBK test failure on DB runtimes[databricks]#13808
firestarman wants to merge 1 commit into
NVIDIA:release/25.12from
firestarman:fix-db-gbk

Conversation

@firestarman

Copy link
Copy Markdown
Collaborator

close #13800

The root cause is missing the check for DB runtimes when specifying the legacy charset config for the CSV read.

Signed-off-by: Firestarman <firestarmanllc@gmail.com>
@firestarman firestarman changed the title Fix a GBK test failure on DB 143+[databricks] Fix a GBK test failure on DB runtimes[databricks] Nov 18, 2025
@firestarman

Copy link
Copy Markdown
Collaborator Author

build

@greptile-apps

greptile-apps Bot commented Nov 18, 2025

Copy link
Copy Markdown
Contributor

Greptile Summary

  • Adds is_databricks_runtime() check to correctly set spark.sql.legacy.javaCharsets=true for GBK CSV test
  • Fixes test failure on Databricks 14.3 runtime where legacy charset config was incorrectly set to false

Confidence Score: 5/5

  • This PR is safe to merge with minimal risk
  • The fix is a simple, targeted one-line change that adds a runtime check for Databricks. The change aligns with how Databricks backported the spark.sql.legacy.javaCharsets configuration from Spark 4.0.0, and directly addresses the reported test failure. The logic is correct and consistent with the existing pattern.
  • No files require special attention

Sequence Diagram

sequenceDiagram
    participant Test as "test_csv_read_gbk_encoded_data"
    participant Runtime as "Runtime Check"
    participant Config as "Spark Config"
    participant CSV as "CSV Reader"
    
    Test->>Runtime: "Check is_spark_400_or_later() or is_databricks_runtime()"
    alt Spark 4.0+ or Databricks
        Runtime->>Config: "Set spark.sql.legacy.javaCharsets=true"
    else Earlier Spark version
        Runtime->>Config: "Set spark.sql.legacy.javaCharsets=false"
    end
    Test->>CSV: "Read GBK encoded CSV with config"
    CSV->>Test: "Return data"
Loading

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 file reviewed, no comments

Edit Code Review Agent Settings | Greptile
React with 👍 or 👎 to share your feedback on this new summary format

@firestarman
firestarman changed the base branch from main to release/25.12 November 18, 2025 02:09
@firestarman

Copy link
Copy Markdown
Collaborator Author

build

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] test_csv_read_gbk_encoded_data fail Part of the plan is not columnar class of DB 14.3 runtime

2 participants