fix(matcher): include last start in unicode substring and leading-digit ignore-case - #104
Open
SebTardif wants to merge 1 commit into
Open
Conversation
…it ignore-case Unicode and CRLF substring used an exclusive end one short of the last valid start, leftover from 3d46b62 while closing issue 34. Ignore-case ASCII needles whose first letter sits at index >= 2 searched the full needle in a too-short window and returned None. Needles in the new tests are already lowercase. This is not PR 100. Signed-off-by: Sebastien Tardif <sebtardif@ncf.ca>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Include the last valid start when searching Unicode (and CRLF) haystacks, and search the full last-valid-start window for ignore-case ASCII needles whose first letter is not at index 0.
Problem
Two substring false negatives on current
master.Unicode and CRLF suffix miss
substring_match_non_asciiiteratedhaystack[start..haystack.len() - needle.len()], so the last valid start (len - needle.len()) was never tried. ASCII haystacks were fine. Any Unicode haystack, or a\r\nthat forces the UnicodeUtf32Strvariant, missed a multi-character substring sitting at the end.This is leftover from #34. The exclusive end was introduced in
3d46b625while closing that issue (panic / false positives). The bound is one short of the last valid start. The ASCII prefilter already useshaystack.len() - needle.len() + 1.Leading-digit ignore-case window
When
ignore_caseis set and the firsta-zin the needle is at indexlen >= 2, the codememmems the full needle in a window ofhaystack.len() - needle.len() + len(too short) and returnsNonewith no fallback.Some(0)andSome(1)already use the correct window+ 1.This is not #100. The needles are already lowercase.
Pattern/Atomfold mixed-case input before callingMatcher. The miss happens for'12foovs bothxx12FOOyyandxx12fooyy.Change
In
matcher/src/exact.rs:haystack.len() - needle.len() + 1(last valid start is included).Some(len)forlen >= 1uses the same path asSome(1): search the first non-letter byte onhaystack[..haystack.len() - needle.len() + 1], then verify the tail withnormalize.Validation
Red (production from
origin/master, new tests only):tests::test_substring_unicode_suffix:"foo" did not match "üfoo"(NonevsSome(56))tests::test_substring_crlf_suffix:CRLF suffix should match: ["foo\nbar", "bar"]tests::test_substring_leading_digits_ignore_case:"12foo" did not match "xx12FOOyy"(NonevsSome(110))Green after the two window fixes:
cargo test --workspace: 28 nucleo-matcher lib tests, 8 nucleo tests, and 9 doc-tests passedcargo fmt --all --checkpassedClippy
-D warningsis already red onmaster(#98); not used as a gate.Refs
Ref #34