Description
split_record_re does not preserve reluctant/lazy quantifier semantics after the Glushkov regex
fast path was introduced in #21936.
For the pattern below, the reluctant \r+? should consume one carriage return. The Glushkov path
instead consumes both consecutive carriage returns, producing a different second token.
C++ reproducer
The following test can be added to cpp/tests/strings/split_tests.cpp:
TEST_F(StringsSplitTest, SplitRecordRegexLazyQuantifier)
{
auto const input = cudf::test::strings_column_wrapper({"\rbaab\r\ra"});
auto const sv = cudf::strings_column_view(input);
auto const prog = cudf::strings::regex_program::create(
"[^ \v\n\t\r\f]\r+?\n*",
cudf::strings::regex_flags::DEFAULT,
cudf::strings::capture_groups::NON_CAPTURE);
using LCW = cudf::test::lists_column_wrapper<cudf::string_view>;
auto const expected = LCW({LCW{"\rbaa", "\ra"}});
auto const result = cudf::strings::split_record_re(sv, *prog);
CUDF_TEST_EXPECT_COLUMNS_EQUIVALENT(result->view(), expected);
}Expected:
Actual:
Regression range
The same input and pattern were tested through the cuDF Java API, which directly invokes
split_record_re:
- cuDF
f566057450f9d0755d1196ac90a391fffa5207b0: passes
- cuDF
4cce29f72a92f4b1ab661daf697ccacdc591b2e9: passes
- cuDF
783e21425a69c82cc61f896ba71f64c2848a11c2: fails
The relevant change in this range is #21936 (709e821727158089cb3571e295dfa12acf58b127),
which enables the Glushkov path for split_re/split_record_re.
The #21936 description lists lazy quantifiers as unsupported and says unsupported patterns should
fall back to the Thompson NFA. However, build_glushkov_program does not appear to reject this
pattern, so \r+? is evaluated with greedy longest-match behavior.
Expected behavior
Patterns containing lazy quantifiers should either preserve Thompson-compatible reluctant
semantics or fall back to the Thompson NFA as described in #21936.
Description
split_record_redoes not preserve reluctant/lazy quantifier semantics after the Glushkov regexfast path was introduced in #21936.
For the pattern below, the reluctant
\r+?should consume one carriage return. The Glushkov pathinstead consumes both consecutive carriage returns, producing a different second token.
C++ reproducer
The following test can be added to
cpp/tests/strings/split_tests.cpp:Expected:
Actual:
Regression range
The same input and pattern were tested through the cuDF Java API, which directly invokes
split_record_re:f566057450f9d0755d1196ac90a391fffa5207b0: passes4cce29f72a92f4b1ab661daf697ccacdc591b2e9: passes783e21425a69c82cc61f896ba71f64c2848a11c2: failsThe relevant change in this range is #21936 (
709e821727158089cb3571e295dfa12acf58b127),which enables the Glushkov path for
split_re/split_record_re.The #21936 description lists lazy quantifiers as unsupported and says unsupported patterns should
fall back to the Thompson NFA. However,
build_glushkov_programdoes not appear to reject thispattern, so
\r+?is evaluated with greedy longest-match behavior.Expected behavior
Patterns containing lazy quantifiers should either preserve Thompson-compatible reluctant
semantics or fall back to the Thompson NFA as described in #21936.