Lazy Evaluation with Streaming Parser for Large Files

[Kingflint]

Current findall() loads all matches into memory. For large files (GB+), this causes OOM errors.

Feature Request
Implement lazy evaluation for findall() that yields results without loading entire file
Add Parser.findall_lazy(file_handle, chunk_size=8192) method
Support backtracking across chunk boundaries for multi-line patterns
Handle partial matches at chunk edges
Maintain accurate span positions across chunks

Test Complexity
Mock file objects with gigabyte-size data
Edge cases: pattern split across chunks
Memory profiling tests
Performance benchmarks vs current implementation

Why Complex
Requires buffer management for patterns spanning chunk boundaries
Need to maintain accurate byte offsets
Must handle regex backtracking across chunks
Complex state machine for partial match handling

[jenisys]

Putting this functionality into the current Parser class is suboptimal in my opinion, because the Parser class does not support the file-reading-and-parsing functionality. IMHO an own FileParser/StreamParser class or utility function would be better suited.

QUESTION:

• Can you implement such a class (as specified above) based on the existing Parser class or parse module functionality ?

[Kingflint]

yes it is implementable and i have completed it will brief you more on it as this is a challenge to add new features to public repositories where impossible... you would notice that i would either be changing or adding comments to issues i cannot solve

[eddiethedean]

Hi! I wanted to share that lazy evaluation has been implemented in formatparse, a high-performance Rust-backed alternative to the parse library.

formatparse (https://github.com/eddiethedean/formatparse) includes:

• ✅ Lazy Results class - findall() returns a Results object that stores raw match data
• ✅ Lazy conversion - ParseResult objects are only created when accessed (via indexing or iteration)
• ✅ Batch conversion - On first iteration, all items are converted in a single batch (one GIL block)
• ✅ Memory efficient - For large result sets, this avoids creating thousands of Python objects upfront

Example:

from formatparse import findall

# findall() returns a Results object (lazy)
results = findall("ID:{id:d}", "ID:1 ID:2 ID:3 ... ID:10000")

# No conversion happens yet - just raw data stored
print(len(results))  # Fast - just returns length

# Conversion happens lazily on first access
first = results[0]  # Converts only this item
print(first.named['id'])  # 1

# Iteration triggers batch conversion
for result in results:  # Converts all items in one batch
    print(result.named['id'])

The Results class stores Vec<RawMatchData> (Rust types) and only converts to Python ParseResult objects when needed. This provides significant memory and performance benefits for large result sets.

Note: File streaming for very large files (GB+) is still a potential enhancement, but the core lazy evaluation is implemented and working well.

formatparse is a drop-in replacement for parse with significant performance improvements (often 2-10x faster) while maintaining API compatibility. If you're interested in trying it out or contributing, check out the repo!