fix: do not flag a malformed-quote error when a \r\n line ending is s…#1127
Open
Jian-Zhang08 wants to merge 1 commit into
Open
fix: do not flag a malformed-quote error when a \r\n line ending is s…#1127Jian-Zhang08 wants to merge 1 commit into
Jian-Zhang08 wants to merge 1 commit into
Conversation
…plit across a chunk boundary (mholt#1103) When streaming, a chunk can end on the \r of a \r\n line ending, leaving a closing quote followed only by a lone \r. That \r matches neither the delimiter nor a full newline, so the parser reported a spurious "Trailing quote on quoted field is malformed" (InvalidQuotes) error even though the row is simply incomplete and is already deferred to the next chunk via _partialLine (no data is lost). Guard the InvalidQuotes branch: when ignoreLastRow is set (more chunks are coming) and there is no further newline left in this chunk, defer the row instead of emitting the error. Genuine malformed quotes are unaffected (there a newline still follows, or it is the final chunk). Adds a regression test that parses a quoted CSV with a \r\n line ending split across a chunk boundary and asserts no error is produced.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…plit across a chunk boundary (#1103)
When streaming, a chunk can end on the \r of a \r\n line ending, leaving a closing quote followed only by a lone \r. That \r matches neither the delimiter nor a full newline, so the parser reported a spurious "Trailing quote on quoted field is malformed" (InvalidQuotes) error even though the row is simply incomplete and is already deferred to the next chunk via _partialLine (no data is lost).
Guard the InvalidQuotes branch: when ignoreLastRow is set (more chunks are coming) and there is no further newline left in this chunk, defer the row instead of emitting the error. Genuine malformed quotes are unaffected (there a newline still follows, or it is the final chunk).
Adds a regression test that parses a quoted CSV with a \r\n line ending split across a chunk boundary and asserts no error is produced.