[BUG] Handle null regex captures in interval and regexp_extract_all (#15280)

**JaCoCo sql-plugin line coverage: +25 lines** (8 interval-parsing lines
covered by `CsvScanForIntervalSuite`, shim 351; 17 `regexp_extract_all`
lines covered by `test_regexp_extract_all_idx_positive`, shim 401)

Closes #15275.
Closes #15281.

## Root cause

[rapidsai/cudf#23123](rapidsai/cudf#23123)
changed regex extraction so optional capture groups that do not
participate in a match return null instead of an empty string. Two
cudf-spark paths relied on the previous representation:

- day-time interval parsing treated optional sign and fractional-seconds
captures as non-null values;
- `regexp_extract_all` removed null list elements, which lost Spark's
required empty-string placeholders for unmatched captures and could
change list cardinality.

Because both failures come from the same cuDF behavior change and
independently break premerge, this PR contains the two focused
compatibility fixes so the complete Blossom matrix can validate and
merge them together.

## Fix

Due to the nature of the premerge CI, I have to merge 2 different
fixes(though they are caused by the same upstream change) in 1 PR:

### Day-time intervals

- Normalize missing sign comparisons to `false` before combining the two
signs with XOR.
- Normalize a missing fractional-seconds capture to numeric zero before
decimal conversion.

### `regexp_extract_all`

- Preserve one output list element per regex match.
- Convert an unmatched requested capture from null to Spark's empty
string.
- Preserve empty lists for non-null inputs with no matches and null
lists for null inputs.
- Restore the input row count when cuDF returns a zero-row list column
for an all-no-match batch.

The longer-term proposal to move Spark-specific regex result
normalization behind the JNI boundary is tracked in
[NVIDIA/cudf-spark-jni#4821](NVIDIA/cudf-spark-jni#4821).
This PR keeps the current ownership boundaries and fixes the two
affected consumers without adding a temporary compatibility switch.

## Validation

- Scala 2.12 / Spark 3.5.1 `CsvScanForIntervalSuite`: `Tests: succeeded
6, failed 0, canceled 1, ignored 0, pending 0`; reactor `BUILD SUCCESS`.
- Scala 2.13 / Spark 4.0.1 production and integration-test build:
reactor `BUILD SUCCESS`.
- Spark 4.0.1 `regexp_test.py::test_regexp_extract_all_idx_positive`: `3
passed, 39627 deselected`.
- The same regex IT with forced OOM injection: `3 passed, 39627
deselected`.
- JaCoCo fix-line intersection: 8 of 16 added interval lines plus 17 of
30 added regex lines covered, for 25 unique added production lines.
- `scripts/check-shim-coverage.sh`: passed.

## Performance impact

The interval change adds null normalization only in the specialized
string-to-day-time-interval path. The regex change removes the previous
max-list-width expansion and reconstructs the existing variable-length
list column in place; the focused microbenchmark measured 0.538 s before
versus 0.464 s after (about 13.7% faster). No new regex match pass is
introduced.

Documentation
- [ ] Updated for new or modified user-facing features or behaviors
- [x] No user-facing change

Testing
- [ ] Added or modified tests to cover new code paths
- [x] Covered by existing tests
(`CsvScanForIntervalSuite` and
`regexp_test.py::test_regexp_extract_all_idx_positive`, including forced
OOM injection.)
- [ ] Not required

Performance
- [x] Tests ran and results are added in the PR description
- [ ] Issue filed with a link in the PR description
- [ ] Not required

---------

Signed-off-by: Allen Xu <allxu@nvidia.com>

Author: wjxiz1992

sql-plugin/src/main/scala/com/nvidia/spark/rapids/shims/GpuIntervalUtilsBase.scala

@@ -286,7 +286,14 @@ trait GpuIntervalUtilsBase {
       firstSignInTable: ColumnVector, secondSignInTable: ColumnVector): ColumnVector = {
     val negatives = withResource(Scalar.fromString("-")) { negScalar =>
       withResource(Seq(firstSignInTable, secondSignInTable).safeMap(negScalar.equalTo)) {
-        case Seq(neg1, neg2) => neg1.bitXor(neg2)
+        signMatches =>
+          // cuDF returns null for optional capture groups that did not participate in the match.
+          // A missing sign is positive, so normalize those null comparisons to false before XOR.
+          withResource(Scalar.fromBool(false)) { falseScalar =>
+            withResource(signMatches.safeMap(_.replaceNulls(falseScalar))) {
+              case Seq(neg1, neg2) => neg1.bitXor(neg2)
+            }
+          }
       }
     }
 
@@ -307,8 +314,14 @@ trait GpuIntervalUtilsBase {
   protected def getMicrosFromDecimal(sign: ColumnVector, decimal: ColumnVector): ColumnVector = {
     val decimalType64_6 = DType.create(DType.DTypeEnum.DECIMAL64, -6)
     val timesMillion = withResource(Scalar.fromLong(1000000L)) { million =>
-      withResource(decimal.castTo(decimalType64_6)) {
-        _.mul(million)
+      // An absent optional fractional-seconds capture is null. It represents zero micros rather
+      // than a failed parse, so normalize it before the decimal conversion.
+      withResource(Scalar.fromString("0")) { zero =>
+        withResource(decimal.replaceNulls(zero)) { normalizedDecimal =>
+          withResource(normalizedDecimal.castTo(decimalType64_6)) {
+            _.mul(million)
+          }
+        }
       }
     }
     val timesMillionLongs = withResource(timesMillion) {

sql-plugin/src/main/scala/org/apache/spark/sql/rapids/stringFunctions.scala

@@ -37,6 +37,7 @@ import com.nvidia.spark.rapids.jni.RegexRewriteUtils
 import com.nvidia.spark.rapids.shims.{NullIntolerantShim, ShimExpression, SparkShimImpl}
 
 import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.util.GenericArrayData
 import org.apache.spark.sql.errors.ConvUtils
 import org.apache.spark.sql.rapids.catalyst.expressions._
 import org.apache.spark.sql.types._
@@ -1753,58 +1754,44 @@ case class GpuRegExpExtractAll(
           EnumSet.of(RegexFlag.EXT_NEWLINE), CaptureGroups.NON_CAPTURE)
         str.getBase.extractAllRecord(prog, 0)
       case _ =>
-        // Extract matches corresponding to idx. cuDF's extract_all_record does not support
-        // group idx, so we must manually extract the relevant matches. Example:
-        // Given the pattern (\d+)-(\d+) and idx=1
-        //
-        // |      Input      |      Java       |               cuDF             |
-        // |-----------------|-----------------|--------------------------------|
-        // | '1-2, 3-4, 5-6' | ['1', '3', '5'] | ['1', '2', '3', '4', '5', '6'] |
-        //
-        // Since idx=1 and the pattern has 2 capture groups, we take the 1st element and every
-        // 2nd element afterwards from the cuDF list
-
         val rowCount = str.getRowCount
         val prog = new RegexProgram(cudfRegexPattern, EnumSet.of(RegexFlag.EXT_NEWLINE))
 
-        val extractedWithNulls = withResource(
-          // Now the index is always 1 because we have transpiled all the capture groups to the
-          // single group that we care about, so we just have to handle the idx = 1 case here
-          str.getBase.extractAllRecord(prog, 1)) { allExtracted =>
-            withResource(allExtracted.countElements) { listSizes =>
-              withResource(listSizes.max) { maxSize =>
-                val maxSizeInt = maxSize.getInt
-                val stringCols = Range(0, maxSizeInt, 1).safeMap {
-                  i =>
-                    allExtracted.extractListElement(i)
-                }
-                withResource(stringCols) { _ =>
-                  ColumnVector.makeList(rowCount, DType.STRING, stringCols: _*)
-                }
+        // The transpiler leaves only the requested group as a capture group, so cuDF already
+        // returns one list element per regex match. Align the remaining result semantics with
+        // Spark: an unmatched capture is an empty string, no matches is an empty list, and a
+        // null input is a null list.
+        withResource(str.getBase.extractAllRecord(prog, 1)) { extracted =>
+          val noMatchesAsEmptyLists = withResource(GpuScalar.from(
+            new GenericArrayData(Array.empty[Any]), dataType)) { emptyStringList =>
+            // cuDF returns a zero-row list column when the entire input has no matches.
+            // Restore the input row count before applying Spark's null-input semantics.
+            if (extracted.getRowCount == 0) {
+              ColumnVector.fromScalar(emptyStringList, rowCount.toInt)
+            } else {
+              val capturesWithEmptyStrings = withResource(extracted.getChildColumnView(0)) {
+                captures =>
+                  withResource(Scalar.fromString("")) { emptyString =>
+                    withResource(captures.replaceNulls(emptyString)) { normalizedCaptures =>
+                      withResource(extracted.replaceListChild(normalizedCaptures)) {
+                        _.copyToColumnVector()
+                      }
+                    }
+                  }
               }
-            }
-          }
-        // Filter out null values in the lists
-        val extractedStrings = withResource(extractedWithNulls) { _ =>
-          val booleanMask = withResource(extractedWithNulls.getListOffsetsView) { offsetsCol =>
-            withResource(extractedWithNulls.getChildColumnView(0)) { stringCol =>
-              withResource(stringCol.isNotNull) { isNotNull =>
-                isNotNull.makeListFromOffsets(rowCount, offsetsCol)
+              withResource(capturesWithEmptyStrings) { normalized =>
+                withResource(normalized.isNull) { noMatchesOrNullInput =>
+                  noMatchesOrNullInput.ifElse(emptyStringList, normalized)
+                }
               }
             }
           }
-          withResource(booleanMask) {
-            extractedWithNulls.applyBooleanMask
-          }
-        }
-
-        // If input is null, output should also be null
-        withResource(extractedStrings) { s =>
-          withResource(GpuScalar.from(null, DataTypes.createArrayType(DataTypes.StringType))) {
-            nullStringList =>
+          withResource(noMatchesAsEmptyLists) { normalized =>
+            withResource(GpuScalar.from(null, dataType)) { nullStringList =>
               withResource(str.getBase.isNull) { isInputNull =>
-                isInputNull.ifElse(nullStringList, s)
+                isInputNull.ifElse(nullStringList, normalized)
               }
+            }
           }
         }
     }