Scan compressed pickle artifacts - #68
Conversation
massy-o
left a comment
massy-o
left a comment
There was a problem hiding this comment.
Self-review notes:
- The decompression step is gated by explicit compound pickle extensions only, so arbitrary
.gzfiles are not treated as pickles during directory/Hugging Face filtering. zipfile.is_zipfile()is still checked before the pickle fallback, preserving existing.npzand zip archive behavior.- I covered the three routes that previously skipped or misparsed compressed joblib payloads: direct file path, directory traversal, and compressed members inside zip archives.
- Full
tests/test_scanner.pypasses with the optional NumPy/7z dependencies installed.
| ".lzma": lzma.decompress, | ||
| ".xz": lzma.decompress, | ||
| } | ||
| _compressed_pickle_file_extensions = { |
There was a problem hiding this comment.
Self-review: compound extensions are generated from the existing pickle extension allowlist, so this expands coverage for compressed forms without making every compressed file type scan as pickle.
| """Disassemble a Pickle stream and report issues""" | ||
| _log.debug(f"scan_pickle_bytes({file_id})") | ||
|
|
||
| if file_ext in _compressed_pickle_file_extensions: |
There was a problem hiding this comment.
Self-review: decompression happens before pickle opcode parsing and decompression failures are surfaced as scan errors, avoiding the previous clean pass on gzip headers.
| _log.debug("Scanning file %s in zip archive %s", file_name, file_id) | ||
| with zip.open(file_name, "r") as file: | ||
| result.merge(scan_pickle_bytes(file, f"{file_id}:{file_name}")) | ||
| result.merge(scan_pickle_bytes(file, f"{file_id}:{file_name}", file_ext=file_ext)) |
There was a problem hiding this comment.
Self-review: this keeps archive member scanning on the same compound-extension path as direct files, which closes the equivalent bypass for .joblib.gz stored inside a zip.
| ) | ||
|
|
||
|
|
||
| def test_scan_compressed_joblib_file_path(tmp_path): |
There was a problem hiding this comment.
Self-review: these regression tests exercise the direct path, directory traversal, and zip-member path so the fix is covered where the scanner previously relied on single-suffix extension checks.
1 participant
Summary
.joblib.gzduring file, directory, archive, and Hugging Face model scansTesting
uv run --with-editable . --with pytest pytest tests/test_scanner.py::test_scan_compressed_joblib_file_path tests/test_scanner.py::test_scan_directory_path_includes_compressed_joblib tests/test_scanner.py::test_scan_zip_bytes_includes_compressed_joblib_memberuv run --with-editable . --with pytest --with numpy --with py7zr pytest tests/test_scanner.pyuv run python -m compileall src/picklescan tests/test_scanner.py