Support case-insensitive inline / scoped flags (?i) / (?i:…) in regex patterns

[igorpeshansky]

Fixes #14947.

Description

rlike / regexp_like, regexp_extract, and regexp_replace fell back to the CPU whenever a pattern used inline or scoped flag syntax ((?i) or (?i:…)). The RegexParser only special-cased (?:...) non-capturing groups; any other (? sequence that was not a lookahead was rejected, so case-insensitive patterns never ran on the GPU.

This PR adds transpiler support for the case-insensitive flag, both the inline (?i) and the scoped (non-capturing) form (?i:…). ASCII letters in case-insensitive runs (parts of the pattern where the i flag is in effect) are rewritten to match either case: literal characters and character-class ranges are folded (e.g. (?i)abc → [aA][bB][cC], (?i)[a-c] → [a-cA-C]). Scoping follows Java's rules: an inline flag applies from its position to the end of the enclosing group, (?-i) turns it back off, and (?i:...) is only in scope within its group. This covers all three placement shapes from the issue: flag at the start of the pattern, mid-pattern, and inside an alternation branch. Results are unchanged (identical to the CPU); only the execution location changes (GPU instead of CPU fallback).

Anything that cannot be folded safely or easily still falls back to the CPU:

• Any positive flag other than i (e.g. (?m), (?s)).
• A case-insensitive scope that would extend across alternatives, like a(?i)b|c|d(?-i)e, where carrying the flag state across | would require a significant rearchitecture of the transpiler. Note that the equivalent scoped form a(?i:b)|(?i:c)|(?i:d)e will run on the GPU.
• Case-insensitive matching of a letter-valued hex or octal escape such as (?i)\x61, which cannot be folded to a character class.

A negated non-i flag (e.g. (?-m)) is treated as a no-op, since those flags are off by default, and there is no mechanism to turn them on.

Tip

The three implementation commits in this change (pre-refactoring, parser support, transpiler support) can be reviewed in order. Each builds independently.

Testing

• Parser unit tests for the bare and scoped inline-flag forms, including edge cases like (?i-), (?-), (?-:), and (?i-:).
• Transpiler unit tests for case-insensitive folding across find, replace, and split modes (literals, positive and negated character classes, nested and scoped flags), for regexp_extract group extraction, and for each fallback rule above.
• Rewrite-suite coverage confirming the fast path still applies through case-sensitive scoped-flag groups.
• Integration tests in regexp_test.py comparing GPU vs CPU for rlike, regexp_extract, and regexp_replace with case-insensitive inline and scoped flags, plus fallback coverage for unsupported flags, choice-crossing flags, and anchor combinations.

Checklists

Documentation

• Updated for new or modified user-facing features or behaviors
• No user-facing change

Testing

• Added or modified tests to cover new code paths
• Covered by existing tests
  (Please provide the names of the existing tests in the PR description.)
• Not required

Performance

• Tests ran and results are added in the PR description
• Issue filed with a link in the PR description
• Not required

[igorpeshansky]

build

[greptile-apps]

Greptile Summary

The PR extends the regex parser and transpiler to support Java case-insensitive inline and scoped flags while retaining CPU fallback for unsupported flag combinations.

• Parses inline (?i) and scoped (?i:…) flag forms.
• Tracks flag scope through groups and alternatives and folds ASCII literals and character classes.
• Adds Scala and Python coverage for matching, extraction, replacement, optimization, and fallback behavior.
• Updates regex compatibility documentation.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
sql-plugin/src/main/scala/com/nvidia/spark/rapids/RegexParser.scala	Adds inline-flag AST nodes, Java-compatible scope tracking, ASCII case folding, and explicit fallback for unsupported forms.
integration_tests/src/main/python/regexp_test.py	Adds end-to-end CPU/GPU equivalence and fallback coverage for case-insensitive regex operations.
tests/src/test/scala/com/nvidia/spark/rapids/RegularExpressionParserSuite.scala	Adds parser assertions for bare and scoped inline-flag syntax.
tests/src/test/scala/com/nvidia/spark/rapids/RegularExpressionRewriteSuite.scala	Verifies scoped flags do not enable unsafe literal rewrite optimizations.
tests/src/test/scala/com/nvidia/spark/rapids/RegularExpressionTranspilerSuite.scala	Exercises case folding, flag scoping, extraction, replacement, split behavior, and fallback conditions.
docs/compatibility.md	Documents supported case-insensitive flags and the patterns that continue to fall back.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Java/Spark regex pattern] --> B[Validate with Java Pattern]
    B --> C[Parse regex AST]
    C --> D{Supported inline flags and scope?}
    D -->|No| E[CPU fallback]
    D -->|Yes| F[Track case-insensitive state]
    F --> G[Fold ASCII literals and character classes]
    G --> H[Emit cuDF-compatible regex]
    H --> I[GPU execution]

Loading

_{Reviews (3): Last reviewed commit: "Document unsupported group and inline-fl..." | Re-trigger Greptile}

[igorpeshansky]

build

[igorpeshansky]

build