Refactor regex group parsing and explicitly reject unsupported group types

[igorpeshansky]

Description

Refactors regex group handling to use an explicit RegexGroup.Type representation instead of separate capture and lookahead fields.

The regex parser now recognizes additional valid Java group constructs:

• Positive and negative lookbehind groups
• Independent groups
• Named capture groups

These constructs remain unsupported by cuDF and are explicitly rejected during transpilation so they reliably fall back to CPU. The change also prevents unsupported groups adjacent to $ or \Z from being silently discarded.

Compatibility documentation now explicitly lists all unsupported group types, including lookahead groups.

Testing includes:

• Parser AST coverage for each group type
• Transpiler fallback coverage for standalone, quantified, and anchor-adjacent unsupported groups
• Integration coverage confirming RLike falls back to CPU

Related issues: #14946, #14947.

Checklists

Documentation

• Updated for new or modified user-facing features or behaviors
• No user-facing change

Testing

• Added or modified tests to cover new code paths
• Covered by existing tests
• Not required

Performance

• Tests ran and results are added in the PR description
• Issue filed with a link in the PR description
• Not required

[greptile-apps]

Greptile Summary

Refactors RegexGroup from (capture: Boolean, lookahead: Option[RegexLookahead]) fields to an explicit sealed RegexGroup.Type ADT, and extends the parser to recognize lookbehind, independent, and named capture groups — all of which are now explicitly rejected during transpilation instead of silently discarded or producing a misleading parse error.

• Parser extension: parseGroup now detects (?<, (?>, and disambiguates lookbehind vs. named-capture by inspecting the char after <; all six new group types parse to proper AST nodes.
• Transpiler guards: Two new early-rejection arms catch (a) any unsupported group type adjacent to $/\Z and (b) any quantified unsupported group before the existing repetition logic, fixing the silent-discard bug mentioned in the PR description.
• Rewrite tightening: removeBrackets, getMultipleContainsLiterals, and isWildcard now restrict unwrapping to Capturing | NonCapturing only, so named-capture and lookaround groups are no longer misidentified as Contains patterns.

Confidence Score: 5/5

Safe to merge — all new group types are explicitly rejected by the transpiler, triggering reliable CPU fallback with clear error messages.

The refactoring is mechanical and internally consistent: every call site that previously destructured the old three-field RegexGroup has been updated to the new groupType field. The two new early-rejection arms in the transpiler close the silent-discard gaps described in the PR. The rewrite optimizer guards are correctly tightened to Capturing | NonCapturing. Parser edge cases are handled. Test coverage spans parser AST assertions, transpiler rejection, rewrite optimizer, and integration fallback.

No files require special attention.

Important Files Changed

Filename	Overview
sql-plugin/src/main/scala/com/nvidia/spark/rapids/RegexParser.scala	Core change: RegexGroup sealed ADT replaces boolean fields; parser and transpiler updated correctly; edge-case EOF handling in parseGroup verified.
sql-plugin/src/main/scala/org/apache/spark/sql/rapids/stringFunctions.scala	Pattern matches updated to new RegexGroup.Type API; countGroups and getChoicesFromRegex correctly restricted to Capturing/NonCapturing.
tests/src/test/scala/com/nvidia/spark/rapids/RegularExpressionTranspilerSuite.scala	New tests cover lookahead, lookbehind, independent, named-capture rejections — standalone, quantified, and anchor-adjacent variants all exercised.
tests/src/test/scala/com/nvidia/spark/rapids/RegularExpressionRewriteSuite.scala	New test confirms unsupported group types are not misidentified as Contains/MultipleContains/StartsWith by the rewrite optimizer.
tests/src/test/scala/com/nvidia/spark/rapids/RegularExpressionParserSuite.scala	Parser AST assertions updated from old boolean fields to new RegexGroup.Type values; new test covers all six group types in one parse.
integration_tests/src/main/python/regexp_test.py	Integration test extended to cover lookbehind, independent, and named-capture fallback via assert_gpu_fallback_collect.
docs/compatibility.md	Documentation updated to enumerate all newly-recognized but unsupported group types.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A["parseGroup()"] --> B{"starts with '?'\nand second char in ':!=<>'?"}
    B -- No --> C[RegexGroup.Capturing]
    B -- Yes --> D["consume '?', then next char"]
    D --> E{Char?}
    E -- "':'" --> F[RegexGroup.NonCapturing]
    E -- "'='" --> G[RegexGroup.PositiveLookahead]
    E -- "'!'" --> H[RegexGroup.NegativeLookahead]
    E -- "'>'" --> I[RegexGroup.Independent]
    E -- "'<'" --> J{"pos < length?"}
    J -- No --> K["throw: trailing '<'"]
    J -- Yes --> L["consume next char"]
    L --> M{Char?}
    M -- "'='" --> N[RegexGroup.PositiveLookbehind]
    M -- "'!'" --> O[RegexGroup.NegativeLookbehind]
    M -- "isLetter" --> P["parse name loop\nthen consume '>'"]
    P --> Q[RegexGroup.Named]
    M -- "other" --> R["throw: unexpected char"]

    subgraph Transpiler ["CudfRegexTranspiler.rewrite()"]
        T1{"groupType?"}
        T1 -- "Capturing / NonCapturing" --> T2[rewrite term and emit group]
        T1 -- "Lookahead / Lookbehind\nIndependent / Named" --> T3["throw RegexUnsupportedException\n→ CPU fallback"]
    end

    C --> T1
    F --> T1
    G --> T1
    H --> T1
    I --> T1
    N --> T1
    O --> T1
    Q --> T1

Loading

_{Reviews (6): Last reviewed commit: "Merge branch 'main' into regex-transpile..." | Re-trigger Greptile}

[greptile-apps]

Want your agent to iterate on Greptile's feedback? Try greploops.

[igorpeshansky]

@greptile please re-review.

[igorpeshansky]

build

[igorpeshansky]

build

[revans2]

If I really want to pick some nits we could have a few more tests for things beyond RLIKE, but I think that is not really needed as most of the code is common between them all.

[igorpeshansky]

If I really want to pick some nits we could have a few more tests for things beyond RLIKE, but I think that is not really needed as most of the code is common between them all.

Yeah, I was piling new cases onto the previously existing test, which only covered RLIKE. We can look into adding more coverage in a follow-up if we choose to.