Skip to content

Commit 6f4a41e

Browse files
AnhTtissunchao
AnhTtis
authored and
sunchao
committed
[SPARK-59043][SQL] Fix SimplifyCaseConversionExpressions to preserve Unicode case-conversion semantics
### What changes were proposed in this pull request? In Catalyst Optimizer, the rule `SimplifyCaseConversionExpressions` previously simplified nested mixed case conversions: - `Upper(Lower(child))` -> `Upper(child)` - `Lower(Upper(child))` -> `Lower(child)` However, in the Unicode standard and Java's case mapping semantics (`UTF8String`), mixed case conversion is not idempotent and not symmetric for several Unicode characters: - For `ı` (U+0131 LATIN SMALL LETTER DOTLESS I): `upper('ı')` = `'I'`, `lower('I')` = `'i'`. Therefore `lower(upper('ı'))` = `'i'`, but `lower('ı')` = `'ı'`. - For `µ` (U+00B5 MICRO SIGN): `upper('µ')` = `'Μ'` (U+039C), `lower('Μ')` = `'μ'` (U+03BC). Therefore `lower(upper('µ'))` = `'μ'`, but `lower('µ')` = `'µ'`. - For `ß` (U+00DF LATIN SMALL LETTER SHARP S): `upper('ß')` = `'SS'`, `lower('SS')` = `'ss'`. Therefore `lower(upper('ß'))` = `'ss'`, but `lower('ß')` = `'ß'`. - For `K` (U+212A KELVIN SIGN): `lower('K')` = `'k'`, `upper('k')` = `'K'`. Therefore `upper(lower('K'))` = `'K'`, but `upper('K')` = `'K'`. Consequently, simplifying `Lower(Upper(child))` to `Lower(child)` or `Upper(Lower(child))` to `Upper(child)` leads to silent data correctness bugs and causes queries to return different results depending on whether `SimplifyCaseConversionExpressions` is enabled. This PR fixes the issue by removing the mixed case simplification rules from `SimplifyCaseConversionExpressions`, retaining only same-case idempotent transformations (`Upper(Upper(child))` -> `Upper(child)` and `Lower(Lower(child))` -> `Lower(child)`), which are 100% idempotent across all Unicode code points. Fixes [SPARK-59043](https://issues.apache.org/jira/browse/SPARK-59043). ### Why are the changes needed? To prevent incorrect query results and maintain Unicode case-conversion semantic correctness under SQL expression optimization. ### Does this PR introduce _any_ user-facing change? Yes. Queries with nested mixed case conversions (e.g. `lower(upper(str))`) on Unicode characters now correctly preserve Unicode semantics and return consistent results regardless of optimizer configuration. ### How was this patch tested? - Updated optimizer unit tests in `SimplifyStringCaseConversionSuite.scala` verifying that mixed case expressions are preserved and same-case expressions are simplified. - Added end-to-end SQL regression tests in `StringFunctionsSuite.scala` covering Unicode edge cases (`'ı'`, `'µ'`, `'ß'`, `'K'`) under both default optimizer configuration and with rule excluded. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #58376 from AnhTtis/SPARK-59043-fix-case-conversion-unicode-semantics. Authored-by: AnhTtis <nguyenhuuanhtri866@gmail.com> Signed-off-by: Chao Sun <chao@openai.com>
1 parent a6e7607 commit 6f4a41e

4 files changed

Lines changed: 101 additions & 8 deletions

File tree

‎docs/sql-migration-guide.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@ license: |
2424

2525
## Upgrading from Spark SQL 4.3 to 4.4
2626

27+
- Since Spark 4.4, the optimizer rule `SimplifyCaseConversionExpressions` no longer simplifies mixed case conversions `LOWER(UPPER(str))` to `LOWER(str)` or `UPPER(LOWER(str))` to `UPPER(str)`. This preserves standard Unicode case mapping semantics for non-ASCII characters such as Latin small letter dotless i (`ı` U+0131) and micro sign (`µ` U+00B5).
2728
- Since Spark 4.4, for storage-partitioned joins, `spark.sql.requireAllClusterKeysForCoPartition` requires every join key to be covered by some partition key instead of matching the partition keys positionally. As a result, a join-key column partitioned by more than one transform no longer prevents shuffle elimination, and `spark.sql.sources.v2.bucketing.allowKeysSubsetOfPartitionKeys.enabled` no longer additionally requires `spark.sql.requireAllClusterKeysForCoPartition` to be `false` when the join keys are a subset of the partition keys. As before, when the partition keys cover only part of the join keys, eliminating the shuffle still requires `spark.sql.requireAllClusterKeysForCoPartition` to be `false`.
2829

2930
## Upgrading from Spark SQL 4.2 to 4.3

‎sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala‎

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1184,17 +1184,20 @@ object SimplifyCasts extends Rule[LogicalPlan] {
11841184

11851185

11861186
/**
1187-
* Removes the inner case conversion expressions that are unnecessary because
1188-
* the inner conversion is overwritten by the outer one.
1187+
* Removes redundant same-case conversion expressions (e.g. UPPER(UPPER(x)) or LOWER(LOWER(x)))
1188+
* that are unnecessary because the case conversion operation is idempotent.
1189+
*
1190+
* Note: Cross-case conversions (UPPER(LOWER(x)) or LOWER(UPPER(x))) are NOT simplified
1191+
* because they are not semantics-preserving for some Unicode characters
1192+
* (e.g. U+0131 LATIN SMALL LETTER DOTLESS I, where LOWER(UPPER(U+0131)) yields 'i'
1193+
* while LOWER(U+0131) leaves the character unchanged).
11891194
*/
11901195
object SimplifyCaseConversionExpressions extends Rule[LogicalPlan] {
11911196
def apply(plan: LogicalPlan): LogicalPlan = plan.transformWithPruning(
11921197
_.containsPattern(UPPER_OR_LOWER), ruleId) {
11931198
case q: LogicalPlan => q.transformExpressionsUpWithPruning(
11941199
_.containsPattern(UPPER_OR_LOWER), ruleId) {
11951200
case Upper(Upper(child)) => Upper(child)
1196-
case Upper(Lower(child)) => Upper(child)
1197-
case Lower(Upper(child)) => Lower(child)
11981201
case Lower(Lower(child)) => Lower(child)
11991202
}
12001203
}

‎sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/SimplifyStringCaseConversionSuite.scala‎

Lines changed: 36 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -48,28 +48,28 @@ class SimplifyStringCaseConversionSuite extends PlanTest {
4848
comparePlans(optimized, correctAnswer)
4949
}
5050

51-
test("simplify UPPER(LOWER(str))") {
51+
test("SPARK-59043: do not simplify UPPER(LOWER(str)) to preserve Unicode semantics") {
5252
val originalQuery =
5353
testRelation
5454
.select(Upper(Lower($"a")) as "u")
5555

5656
val optimized = Optimize.execute(originalQuery.analyze)
5757
val correctAnswer =
5858
testRelation
59-
.select(Upper($"a") as "u")
59+
.select(Upper(Lower($"a")) as "u")
6060
.analyze
6161

6262
comparePlans(optimized, correctAnswer)
6363
}
6464

65-
test("simplify LOWER(UPPER(str))") {
65+
test("SPARK-59043: do not simplify LOWER(UPPER(str)) to preserve Unicode semantics") {
6666
val originalQuery =
6767
testRelation
6868
.select(Lower(Upper($"a")) as "l")
6969

7070
val optimized = Optimize.execute(originalQuery.analyze)
7171
val correctAnswer = testRelation
72-
.select(Lower($"a") as "l")
72+
.select(Lower(Upper($"a")) as "l")
7373
.analyze
7474

7575
comparePlans(optimized, correctAnswer)
@@ -87,4 +87,36 @@ class SimplifyStringCaseConversionSuite extends PlanTest {
8787

8888
comparePlans(optimized, correctAnswer)
8989
}
90+
91+
test("SPARK-59043: simplify deeply nested same-case expressions") {
92+
val nestedUpper = testRelation.select(Upper(Upper(Upper($"a"))) as "u")
93+
val optimizedUpper = Optimize.execute(nestedUpper.analyze)
94+
val expectedUpper = testRelation.select(Upper($"a") as "u").analyze
95+
comparePlans(optimizedUpper, expectedUpper)
96+
97+
val nestedLower = testRelation.select(Lower(Lower(Lower($"a"))) as "l")
98+
val optimizedLower = Optimize.execute(nestedLower.analyze)
99+
val expectedLower = testRelation.select(Lower($"a") as "l").analyze
100+
comparePlans(optimizedLower, expectedLower)
101+
}
102+
103+
test("SPARK-59043: mixed case expressions with multiple layers") {
104+
// Upper(Lower(Upper(str))) is preserved as there are no adjacent same-case operations
105+
val query1 = testRelation.select(Upper(Lower(Upper($"a"))) as "res")
106+
val optimized1 = Optimize.execute(query1.analyze)
107+
val expected1 = testRelation.select(Upper(Lower(Upper($"a"))) as "res").analyze
108+
comparePlans(optimized1, expected1)
109+
110+
// Lower(Upper(Upper(str))) simplifies inner Upper(Upper) to Upper -> Lower(Upper(str))
111+
val query2 = testRelation.select(Lower(Upper(Upper($"a"))) as "res")
112+
val optimized2 = Optimize.execute(query2.analyze)
113+
val expected2 = testRelation.select(Lower(Upper($"a")) as "res").analyze
114+
comparePlans(optimized2, expected2)
115+
116+
// Upper(Lower(Lower(str))) simplifies inner Lower(Lower) to Lower -> Upper(Lower(str))
117+
val query3 = testRelation.select(Upper(Lower(Lower($"a"))) as "res")
118+
val optimized3 = Optimize.execute(query3.analyze)
119+
val expected3 = testRelation.select(Upper(Lower($"a")) as "res").analyze
120+
comparePlans(optimized3, expected3)
121+
}
90122
}

‎sql/core/src/test/scala/org/apache/spark/sql/StringFunctionsSuite.scala‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1564,4 +1564,61 @@ class StringFunctionsSuite extends SharedSparkSession {
15641564
)
15651565
}
15661566
}
1567+
1568+
test("SPARK-59043: SimplifyCaseConversionExpressions preserves Unicode semantics") {
1569+
val excludedConf = "org.apache.spark.sql.catalyst.optimizer.SimplifyCaseConversionExpressions"
1570+
Seq(true, false).foreach { optimizerEnabled =>
1571+
val confModifier = if (optimizerEnabled) {
1572+
Map.empty[String, String]
1573+
} else {
1574+
Map(SQLConf.OPTIMIZER_EXCLUDED_RULES.key -> excludedConf)
1575+
}
1576+
1577+
withSQLConf(confModifier.toSeq: _*) {
1578+
// scalastyle:off
1579+
// non ascii characters are not allowed in the code, so we disable the scalastyle here.
1580+
// 1. Turkish dotless i (U+0131 LATIN SMALL LETTER DOTLESS I)
1581+
checkAnswer(
1582+
sql("SELECT lower(upper('ı')) AS result"),
1583+
Row("i") :: Nil
1584+
)
1585+
checkAnswer(
1586+
sql("SELECT lower(upper(s)) FROM (VALUES ('ı')) AS t(s)"),
1587+
Row("i") :: Nil
1588+
)
1589+
checkAnswer(
1590+
sql(
1591+
"""SELECT lower(upper(s2)) AS result
1592+
|FROM (VALUES ('ı')) AS t(s)
1593+
|LATERAL VIEW explode(array(s)) e AS s2""".stripMargin),
1594+
Row("i") :: Nil
1595+
)
1596+
1597+
// 2. Micro sign (U+00B5 MICRO SIGN)
1598+
checkAnswer(
1599+
sql("SELECT lower(upper('µ')) AS result"),
1600+
Row("μ") :: Nil
1601+
)
1602+
1603+
// 3. German Sharp S (U+00DF LATIN SMALL LETTER SHARP S)
1604+
checkAnswer(
1605+
sql("SELECT lower(upper('ß')) AS result"),
1606+
Row("ss") :: Nil
1607+
)
1608+
1609+
// 4. Kelvin Sign (U+212A KELVIN SIGN)
1610+
checkAnswer(
1611+
sql("SELECT upper(lower('K')) AS result"),
1612+
Row("K") :: Nil
1613+
)
1614+
// scalastyle:on
1615+
1616+
// 5. Verify same-case idempotent operations still simplify
1617+
checkAnswer(
1618+
sql("SELECT upper(upper('abc')), lower(lower('XYZ'))"),
1619+
Row("ABC", "xyz") :: Nil
1620+
)
1621+
}
1622+
}
1623+
}
15671624
}

0 commit comments

Comments
 (0)