Skip to content

Commit 0fbec92

Browse files
baibaichenCopilot
baibaichen
and
Copilot
authored
[GLUTEN-11550][UT] Add GlutenTestSetWithSystemPropertyTrait, enable 6 suites, remove 7 TestHiveSingleton suites, fix JobTagging flaky test (#11847)
* [GLUTEN-11550][UT] Add GlutenTestSetWithSystemPropertyTrait and enable 6 suites Add a reusable trait for test suites whose parent creates per-test SparkContext/SparkSession (via LocalSparkContext/LocalSparkSession). It injects GlutenPlugin config via system properties so per-test sessions inherit it, avoiding shared session conflicts. Enable 6 suites by switching to this trait: - GlutenSQLExecutionSuite (SQLExecutionSuite: new SparkContext internally) - GlutenSQLJsonProtocolSuite (SQLJsonProtocolSuite with LocalSparkSession) - GlutenShufflePartitionsUtilSuite (ShufflePartitionsUtilSuite with LocalSparkContext) - GlutenExternalAppendOnlyUnsafeRowArraySuite (with LocalSparkContext) - GlutenUnsafeRowSerializerSuite (UnsafeRowSerializerSuite with LocalSparkSession) - GlutenSparkSessionJobTaggingAndCancellationSuite (with LocalSparkContext) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [GLUTEN-11550][UT] Remove TestHiveSingleton suites that cannot load GlutenPlugin Remove 7 test suites (spark40 + spark41) that extend TestHiveSingleton. TestHiveSingleton.spark (protected val) conflicts with Gluten test traits' spark (implicit protected def), making it impossible to load GlutenPlugin. The non-Hive variants (e.g. GlutenBucketedReadWithoutHiveSupportSuite) are already enabled and cover the same functionality. Removed suites: - GlutenBucketedReadWithHiveSupportSuite - GlutenBucketedWriteWithHiveSupportSuite - GlutenDisableUnnecessaryBucketedScanWithHiveSupportSuite - GlutenCommitFailureTestRelationSuite - GlutenJsonHadoopFsRelationSuite - GlutenParquetHadoopFsRelationSuite - GlutenSimpleTextHadoopFsRelationSuite Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [GLUTEN-11550][UT] Fix JobTagging flaky test with fresh thread pool The original test uses ExecutionContext.global whose ForkJoinPool reuses threads created long before addTag(). InheritableThreadLocal (used by managedJobTags) only copies values at thread creation time, so reused threads see an empty tag map. This causes withSessionTagsApplied to not attach the user tag to the job, making cancelJobsWithTagWithFuture return empty. Fix: testGluten override that creates a fresh SingleThreadExecutor AFTER addTag(), ensuring the new thread inherits InheritableThreadLocal values. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent ed93689 commit 0fbec92

30 files changed

Lines changed: 225 additions & 370 deletions

‎gluten-ut/common/src/test/scala/org/apache/spark/sql/GlutenTestSetWithSystemPropertyTrait.scala‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
/*
2+
* Licensed to the Apache Software Foundation (ASF) under one or more
3+
* contributor license agreements. See the NOTICE file distributed with
4+
* this work for additional information regarding copyright ownership.
5+
* The ASF licenses this file to You under the Apache License, Version 2.0
6+
* (the "License"); you may not use this file except in compliance with
7+
* the License. You may obtain a copy of the License at
8+
*
9+
* http://www.apache.org/licenses/LICENSE-2.0
10+
*
11+
* Unless required by applicable law or agreed to in writing, software
12+
* distributed under the License is distributed on an "AS IS" BASIS,
13+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
* See the License for the specific language governing permissions and
15+
* limitations under the License.
16+
*/
17+
package org.apache.spark.sql
18+
19+
/**
20+
* A test trait for suites whose parent creates per-test SparkContext/SparkSession (via
21+
* LocalSparkContext, LocalSparkSession, or direct `new SparkContext`).
22+
*
23+
* Injects GlutenPlugin config via system properties so per-test sessions inherit it. SparkConf
24+
* reads system properties prefixed with "spark." as defaults during construction.
25+
*
26+
* Do NOT use GlutenTestsTrait or GlutenSQLTestsTrait for these suites -- they create a shared
27+
* SparkSession in beforeAll() that conflicts with per-test SparkContext creation.
28+
*/
29+
trait GlutenTestSetWithSystemPropertyTrait extends GlutenTestsCommonTrait {
30+
31+
override def beforeAll(): Unit = {
32+
System.setProperty("spark.plugins", "org.apache.gluten.GlutenPlugin")
33+
System.setProperty("spark.memory.offHeap.enabled", "true")
34+
System.setProperty("spark.memory.offHeap.size", "1024MB")
35+
System.setProperty(
36+
"spark.shuffle.manager",
37+
"org.apache.spark.shuffle.sort.ColumnarShuffleManager")
38+
super.beforeAll()
39+
}
40+
41+
override def afterAll(): Unit = {
42+
super.afterAll()
43+
System.clearProperty("spark.plugins")
44+
System.clearProperty("spark.memory.offHeap.enabled")
45+
System.clearProperty("spark.memory.offHeap.size")
46+
System.clearProperty("spark.shuffle.manager")
47+
}
48+
}

‎gluten-ut/excluded-spark-uts.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,4 +3,5 @@
33

44
| Suites | Versions | Reason to exclude |
55
|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
6-
| org.apache.spark.sql.SingleLevelAggregateHashMapSuite<br/>org.apache.spark.sql.TwoLevelAggregateHashMapSuite<br/>org.apache.spark.sql.TwoLevelAggregateHashMapWithVectorizedMapSuite | 4.0, 4.1 | This UTs is similar to `DataFrameAggregateSuite`.<br/>The only difference being that it contains variation for Spark codegen enabled and disabled and enabling single level or two level aggregate hash-maps which is also specific to Spark Aggregate implementation.<br/>We already run `GlutenDataFrameAggregateSuite` for `DataFrameAggregateSuite` so these additional suites doesn't need to run |
6+
| org.apache.spark.sql.SingleLevelAggregateHashMapSuite<br/>org.apache.spark.sql.TwoLevelAggregateHashMapSuite<br/>org.apache.spark.sql.TwoLevelAggregateHashMapWithVectorizedMapSuite | 4.0, 4.1 | This UTs is similar to `DataFrameAggregateSuite`.<br/>The only difference being that it contains variation for Spark codegen enabled and disabled and enabling single level or two level aggregate hash-maps which is also specific to Spark Aggregate implementation.<br/>We already run `GlutenDataFrameAggregateSuite` for `DataFrameAggregateSuite` so these additional suites doesn't need to run |
7+
| org.apache.spark.sql.sources.GlutenBucketedReadWithHiveSupportSuite<br/>org.apache.spark.sql.sources.GlutenBucketedWriteWithHiveSupportSuite<br/>org.apache.spark.sql.sources.GlutenDisableUnnecessaryBucketedScanWithHiveSupportSuite<br/>org.apache.spark.sql.sources.GlutenCommitFailureTestRelationSuite<br/>org.apache.spark.sql.sources.GlutenJsonHadoopFsRelationSuite<br/>org.apache.spark.sql.sources.GlutenParquetHadoopFsRelationSuite<br/>org.apache.spark.sql.sources.GlutenSimpleTextHadoopFsRelationSuite | 4.0, 4.1 | These suites extend `TestHiveSingleton` which provides a global HiveSession singleton. The `TestHiveSingleton.spark` (protected val) conflicts with Gluten test traits' `spark` (implicit protected def), making it impossible to load GlutenPlugin through any Gluten test trait. The non-Hive variants of these suites (e.g. `GlutenBucketedReadWithoutHiveSupportSuite`) are already enabled and cover the same functionality. |

‎gluten-ut/spark40/src/test/scala/org/apache/gluten/utils/velox/VeloxTestSettings.scala‎

Lines changed: 6 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -680,7 +680,7 @@ class VeloxTestSettings extends BackendTestSettings {
680680
// TODO: 4.x enableSuite[GlutenDataSourceScanExecRedactionSuite] // 2 failures
681681
// TODO: 4.x enableSuite[GlutenDataSourceV2ScanExecRedactionSuite] // 2 failures
682682
enableSuite[GlutenExecuteImmediateEndToEndSuite]
683-
// TODO: 4.x enableSuite[GlutenExternalAppendOnlyUnsafeRowArraySuite] // 14 failures
683+
enableSuite[GlutenExternalAppendOnlyUnsafeRowArraySuite]
684684
enableSuite[GlutenGlobalTempViewSuite]
685685
enableSuite[GlutenGlobalTempViewTestSuite]
686686
enableSuite[GlutenGroupedIteratorSuite]
@@ -696,18 +696,18 @@ class VeloxTestSettings extends BackendTestSettings {
696696
// TODO: 4.x enableSuite[GlutenRemoveRedundantProjectsSuite] // 14 failures
697697
// TODO: 4.x enableSuite[GlutenRemoveRedundantSortsSuite] // 1 failure
698698
enableSuite[GlutenRowToColumnConverterSuite]
699-
// TODO: 4.x enableSuite[GlutenSQLExecutionSuite] // 1 failure
699+
enableSuite[GlutenSQLExecutionSuite]
700700
enableSuite[GlutenSQLFunctionSuite]
701-
// TODO: 4.x enableSuite[GlutenSQLJsonProtocolSuite] // 1 failure
702-
// TODO: 4.x enableSuite[GlutenShufflePartitionsUtilSuite] // 1 failure
701+
enableSuite[GlutenSQLJsonProtocolSuite]
702+
enableSuite[GlutenShufflePartitionsUtilSuite]
703703
// TODO: 4.x enableSuite[GlutenSimpleSQLViewSuite] // 1 failure
704704
// TODO: 4.x enableSuite[GlutenSparkPlanSuite] // 1 failure
705705
enableSuite[GlutenSparkPlannerSuite]
706706
enableSuite[GlutenSparkScriptTransformationSuite]
707707
enableSuite[GlutenSparkSqlParserSuite]
708708
enableSuite[GlutenUnsafeFixedWidthAggregationMapSuite]
709709
enableSuite[GlutenUnsafeKVExternalSorterSuite]
710-
// TODO: 4.x enableSuite[GlutenUnsafeRowSerializerSuite] // 1 failure
710+
enableSuite[GlutenUnsafeRowSerializerSuite]
711711
// TODO: 4.x enableSuite[GlutenWholeStageCodegenSparkSubmitSuite] // 1 failure
712712
// TODO: 4.x enableSuite[GlutenWholeStageCodegenSuite] // 24 failures
713713
enableSuite[GlutenBroadcastExchangeSuite]
@@ -790,14 +790,7 @@ class VeloxTestSettings extends BackendTestSettings {
790790
enableSuite[GlutenSaveLoadSuite]
791791
enableSuite[GlutenTableScanSuite]
792792
// Generated suites for org.apache.spark.sql.sources
793-
// TODO: 4.x enableSuite[GlutenBucketedReadWithHiveSupportSuite] // 2 failures
794-
// TODO: 4.x enableSuite[GlutenBucketedWriteWithHiveSupportSuite] // 1 failure
795-
// TODO: 4.x enableSuite[GlutenCommitFailureTestRelationSuite] // 2 failures
796793
enableSuite[GlutenDataSourceAnalysisSuite]
797-
// TODO: 4.x enableSuite[GlutenDisableUnnecessaryBucketedScanWithHiveSupportSuite] // 2 failures
798-
// TODO: 4.x enableSuite[GlutenJsonHadoopFsRelationSuite] // 2 failures
799-
// TODO: 4.x enableSuite[GlutenParquetHadoopFsRelationSuite] // 2 failures
800-
// TODO: 4.x enableSuite[GlutenSimpleTextHadoopFsRelationSuite] // 2 failures
801794
// Generated suites for org.apache.spark.sql
802795
enableSuite[GlutenCacheManagerSuite]
803796
enableSuite[GlutenDataFrameShowSuite]
@@ -823,6 +816,7 @@ class VeloxTestSettings extends BackendTestSettings {
823816
// TODO: 4.x enableSuite[GlutenSetCommandSuite] // 1 failure
824817
enableSuite[GlutenSparkSessionBuilderSuite]
825818
enableSuite[GlutenSparkSessionJobTaggingAndCancellationSuite]
819+
.exclude("Tags set from session are prefixed with session UUID")
826820
enableSuite[GlutenTPCDSCollationQueryTestSuite]
827821
enableSuite[GlutenTPCDSModifiedPlanStabilitySuite]
828822
enableSuite[GlutenTPCDSModifiedPlanStabilityWithStatsSuite]

‎gluten-ut/spark40/src/test/scala/org/apache/spark/sql/GlutenSparkSessionJobTaggingAndCancellationSuite.scala‎

Lines changed: 66 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,71 @@
1616
*/
1717
package org.apache.spark.sql
1818

19+
import org.apache.spark.SparkContext
20+
import org.apache.spark.scheduler.{SparkListener, SparkListenerJobStart}
21+
import org.apache.spark.sql.execution.SQLExecution
22+
import org.apache.spark.util.ThreadUtils
23+
24+
import java.util.concurrent.{Executors, Semaphore, TimeUnit}
25+
26+
import scala.concurrent.{ExecutionContext, Future}
27+
import scala.concurrent.duration._
28+
1929
class GlutenSparkSessionJobTaggingAndCancellationSuite
2030
extends SparkSessionJobTaggingAndCancellationSuite
21-
with GlutenTestsTrait {}
31+
with GlutenTestSetWithSystemPropertyTrait {
32+
33+
// Override with testGluten because the original test uses ExecutionContext.global whose
34+
// ForkJoinPool reuses threads created before addTag("one"). InheritableThreadLocal (used by
35+
// managedJobTags) only copies values at thread creation time, so reused threads see an empty
36+
// tag map. This causes withSessionTagsApplied to not attach the user tag to the job, making
37+
// cancelJobsWithTagWithFuture return empty. We fix this by creating a fresh thread pool AFTER
38+
// addTag so that new threads inherit the InheritableThreadLocal values.
39+
testGluten("Tags set from session are prefixed with session UUID") {
40+
sc = new SparkContext("local[2]", "test")
41+
val session = classic.SparkSession.builder().sparkContext(sc).getOrCreate()
42+
import session.implicits._
43+
44+
val sem = new Semaphore(0)
45+
sc.addSparkListener(new SparkListener {
46+
override def onJobStart(jobStart: SparkListenerJobStart): Unit = {
47+
sem.release()
48+
}
49+
})
50+
51+
session.addTag("one")
52+
53+
// Use a fresh thread pool created AFTER addTag so that new threads inherit
54+
// InheritableThreadLocal values (managedJobTags, threadUuid).
55+
val threadPool = Executors.newSingleThreadExecutor()
56+
implicit val ec: ExecutionContext = ExecutionContext.fromExecutor(threadPool)
57+
try {
58+
Future {
59+
session.range(1, 10000).map { i => Thread.sleep(100); i }.count()
60+
}
61+
62+
assert(sem.tryAcquire(1, 1, TimeUnit.MINUTES))
63+
val activeJobsFuture =
64+
session.sparkContext.cancelJobsWithTagWithFuture(
65+
session.managedJobTags.get()("one"),
66+
"reason")
67+
val activeJob = ThreadUtils.awaitResult(activeJobsFuture, 60.seconds).head
68+
val actualTags = activeJob.properties
69+
.getProperty(SparkContext.SPARK_JOB_TAGS)
70+
.split(SparkContext.SPARK_JOB_TAGS_SEP)
71+
assert(
72+
actualTags.toSet == Set(
73+
session.sessionJobTag,
74+
s"${session.sessionJobTag}-thread-${session.threadUuid.get()}-one",
75+
SQLExecution.executionIdJobTag(
76+
session,
77+
activeJob.properties
78+
.get(SQLExecution.EXECUTION_ROOT_ID_KEY)
79+
.asInstanceOf[String]
80+
.toLong)
81+
))
82+
} finally {
83+
threadPool.shutdownNow()
84+
}
85+
}
86+
}

‎gluten-ut/spark40/src/test/scala/org/apache/spark/sql/execution/GlutenExternalAppendOnlyUnsafeRowArraySuite.scala‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -16,8 +16,8 @@
1616
*/
1717
package org.apache.spark.sql.execution
1818

19-
import org.apache.spark.sql.GlutenTestsTrait
19+
import org.apache.spark.sql.GlutenTestSetWithSystemPropertyTrait
2020

2121
class GlutenExternalAppendOnlyUnsafeRowArraySuite
2222
extends ExternalAppendOnlyUnsafeRowArraySuite
23-
with GlutenTestsTrait {}
23+
with GlutenTestSetWithSystemPropertyTrait {}

‎gluten-ut/spark40/src/test/scala/org/apache/spark/sql/execution/GlutenSQLExecutionSuite.scala‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,6 @@
1616
*/
1717
package org.apache.spark.sql.execution
1818

19-
import org.apache.spark.sql.GlutenTestsTrait
19+
import org.apache.spark.sql.GlutenTestSetWithSystemPropertyTrait
2020

21-
class GlutenSQLExecutionSuite extends SQLExecutionSuite with GlutenTestsTrait {}
21+
class GlutenSQLExecutionSuite extends SQLExecutionSuite with GlutenTestSetWithSystemPropertyTrait {}

‎gluten-ut/spark40/src/test/scala/org/apache/spark/sql/execution/GlutenSQLJsonProtocolSuite.scala‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,8 @@
1616
*/
1717
package org.apache.spark.sql.execution
1818

19-
import org.apache.spark.sql.GlutenTestsTrait
19+
import org.apache.spark.sql.GlutenTestSetWithSystemPropertyTrait
2020

21-
class GlutenSQLJsonProtocolSuite extends SQLJsonProtocolSuite with GlutenTestsTrait {}
21+
class GlutenSQLJsonProtocolSuite
22+
extends SQLJsonProtocolSuite
23+
with GlutenTestSetWithSystemPropertyTrait {}

‎gluten-ut/spark40/src/test/scala/org/apache/spark/sql/execution/GlutenShufflePartitionsUtilSuite.scala‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,8 @@
1616
*/
1717
package org.apache.spark.sql.execution
1818

19-
import org.apache.spark.sql.GlutenTestsTrait
19+
import org.apache.spark.sql.GlutenTestSetWithSystemPropertyTrait
2020

21-
class GlutenShufflePartitionsUtilSuite extends ShufflePartitionsUtilSuite with GlutenTestsTrait {}
21+
class GlutenShufflePartitionsUtilSuite
22+
extends ShufflePartitionsUtilSuite
23+
with GlutenTestSetWithSystemPropertyTrait {}

‎gluten-ut/spark40/src/test/scala/org/apache/spark/sql/execution/GlutenUnsafeRowSerializerSuite.scala‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,8 @@
1616
*/
1717
package org.apache.spark.sql.execution
1818

19-
import org.apache.spark.sql.GlutenTestsTrait
19+
import org.apache.spark.sql.GlutenTestSetWithSystemPropertyTrait
2020

21-
class GlutenUnsafeRowSerializerSuite extends UnsafeRowSerializerSuite with GlutenTestsTrait {}
21+
class GlutenUnsafeRowSerializerSuite
22+
extends UnsafeRowSerializerSuite
23+
with GlutenTestSetWithSystemPropertyTrait {}

‎gluten-ut/spark40/src/test/scala/org/apache/spark/sql/sources/GlutenBucketedReadWithHiveSupportSuite.scala‎

Lines changed: 0 additions & 23 deletions
This file was deleted.

0 commit comments

Comments
 (0)