chore: add script to run gluten/spark ut with bolt and add to ci#586
Draft
zhangxffff wants to merge 4 commits into
Draft
chore: add script to run gluten/spark ut with bolt and add to ci#586zhangxffff wants to merge 4 commits into
zhangxffff wants to merge 4 commits into
Conversation
zhangxffff
force-pushed
the
run_gluten_ut
branch
2 times, most recently
from
d5f35a5 to
c16031f
Compare
… failures
run 27086097418 still exited 1: the blacklist was missing the two
GlutenDateExpressionsSuite#Gluten - {unix_timestamp,to_unix_timestamp}
cases (same timestamp-handling gap as the already-listed cast-to-timestamp
failures). Replaying run.sh's summary over the run's reports now yields
expected=17 unexpected=0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…red) run_one_suite appends the GLUTEN_UT_MVN_RC marker right after mvn's colorised output, which can end with a trailing ANSI reset and no newline. The marker line then reads "\e[0m\e[0mGLUTEN_UT_MVN_RC=0", so the summary's '^'-anchored `grep -m1 '^GLUTEN_UT_MVN_RC='` never matched. Under `set -euo pipefail` the failed `rc=$(grep|cut)` aborted the script on the first suite — right after the "Summary" banner — so the per-failure `! key` lines and the expected/unexpected counts were never printed and the step always exited 1, regardless of the blacklist. Fix both sides: write the marker on its own line (leading \n), and match it un-anchored + ANSI-tolerant with a `|| true` guard so a stray log can't kill the summary. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: close #588
Type of Change
Description
This PR introduce a script to run gluten ut with bolt backend and add it into bolt CI.
Added File:
scripts/gluten_ut/run.shto run gluten ut in parallel.scripts/gluten_ut/blacklist.txta blacklist contains known failure case.scripts/gluten_ut/run.shslow test suites in gluten that would dispatch first to reduce parallel run cost..github/workflows/bolt_spark_ut.ymlworkflows to run gluten ut in boltUsage
Key design
Gluten's
mvn testruns via thescalatest-maven-plugin, which doesn't parallelize across suites. The full UT matrix takes about 2 hours end-to-end — too slow for the Bolt CI loop. This PR runs each suite as its own mvn invocation in parallel, dropping wall time to roughly 30 minutes with 8 JOBS in CI.How it works
mvn install -DskipTestsproduces test-classes for every module.*/test-classes/*.class, and filter the test suites by pattern into a flat suite list.slow_suites.txt(suites running ≥ 3 min) dispatch first, preventing them from trailing on a single worker at the end of the run.xargs -P JOBSruns one mvn per suite. Each invocation is wrapped inbwrapwith a privatetarget/surefire,target/surefire-reports, and a--tmp-overlay-edtest-classes/, so concurrent suites can't race on shared write paths or on Gluten'sunit-tests-working-home/warehouse.*** FAILED ***/*** ABORTED ***markers, and look them up inblacklist.txt. Matches are expected and ignored; anything else is reported as an unexpected failure and fails the run, with the offending keys printed for triage.Performance Impact
No Impact: This change does not affect the critical path (e.g., build system, doc, error handling).
Positive Impact: I have run benchmarks.
Click to view Benchmark Results
Negative Impact: Explained below (e.g., trade-off for correctness).
Release Note
Please describe the changes in this PR
Release Note:
Checklist (For Author)
Breaking Changes
No
Yes (Description: ...)
Click to view Breaking Changes