|
2 | 2 |
|
3 | 3 | All notable changes to this project will be documented in this file. |
4 | 4 |
|
5 | | -## Unreleased |
6 | | - |
7 | | -### ZIP64 metadata support (Phase 1) |
8 | | - |
9 | | -Sync ZIP APIs are now ZIP64-aware for archives and entries that still fit |
10 | | -inside the `Int`/`FixedArray`-bounded sync API budget. Larger archives |
11 | | -that need streaming continue to be deferred to Phase 2. |
12 | | - |
13 | | -#### Reader |
14 | | - |
15 | | -- **ZIP64 EOCD record + locator parsing** (PKWARE APPNOTE §4.3.14 / |
16 | | - §4.3.15) shared between `unzip_sync` and `unzip_list`. Multi-disk |
17 | | - archives are rejected at every layer (classic EOCD, ZIP64 EOCD record, |
18 | | - ZIP64 locator, per-entry disk number). |
19 | | -- **EOCD discovery** now requires the candidate's declared comment length |
20 | | - to match the candidate's distance from the end of the input, so a |
21 | | - `0x06054B50` byte sequence inside an EOCD comment can no longer be |
22 | | - mistaken for the record itself. |
23 | | -- **65 535-entry compatibility**: a classic archive whose only sentinel- |
24 | | - like field is the entry-count `0xFFFF`, with no ZIP64 locator present, |
25 | | - is parsed as a regular 65 535-entry classic archive instead of being |
26 | | - rejected. |
27 | | -- **ZIP64 extended-information extra field parsing** is now conditional |
28 | | - per APPNOTE §4.5.3: each 8-byte value is only consumed when its |
29 | | - matching classic 32-bit field is the sentinel, in spec-fixed order |
30 | | - (uncompressed, compressed, local header offset). Unrelated extras |
31 | | - before the ZIP64 entry are skipped. Sentinel values never propagate |
32 | | - out as real sizes. |
33 | | -- **Local-header validation**: the new `local_data_offset` checks the |
34 | | - local file header signature, the general-purpose bit flag (rejects |
35 | | - encryption), filename + extra bounds, and data range bounds before any |
36 | | - extraction. |
37 | | -- **Data-descriptor entries** (general-purpose bit 3) are accepted by the |
38 | | - sync reader using central-directory sizes and CRC-32 for bounds and |
39 | | - integrity checks, including both stored and deflated entries. |
40 | | -- **ZIP integrity and resource caps**: `unzip_sync` validates each |
41 | | - extracted entry against the central-directory CRC-32 and caps total |
42 | | - sync output. ZIP listing/extraction also rejects oversized sync input |
43 | | - and excessive entry fan-out before allocating result arrays. |
44 | | -- **Decompression cap fix**: `unzip_sync` now passes |
45 | | - `default_max_output_size` (not `default_max_input_size`) to the |
46 | | - inflater for the output cap, and applies the same cap to stored |
47 | | - entries before allocating a buffer. Deflated entries also reject an |
48 | | - uncompressed size above the cap before allocation. |
49 | | - |
50 | | -#### Writer |
51 | | - |
52 | | -- **ZIP64 emission**: `zip_sync` now automatically promotes archives to |
53 | | - ZIP64 when any per-entry size or local-header offset crosses |
54 | | - `0xFFFFFFFF`, when entry count crosses `0xFFFF`, or when the central |
55 | | - directory size or offset crosses `0xFFFFFFFF`. The promotion is |
56 | | - per-field — only overflowing classic fields use the sentinel value, |
57 | | - but a ZIP64 EOCD record + locator are written before the classic EOCD |
58 | | - whenever any field is promoted. |
59 | | -- **Per-entry ZIP64 extras** follow APPNOTE §4.4.8 / §4.5.3: the local |
60 | | - header carries a 16-byte payload with both 8-byte size values when |
61 | | - either size needs ZIP64 (never the local header offset); the central |
62 | | - directory entry carries only the 8-byte values for fields that |
63 | | - actually use the sentinel, in fixed order. |
64 | | -- **Version bytes** stamp the spec-low byte at 45 for any header that |
65 | | - carries ZIP64 sentinels or a ZIP64 extra field, and at 20 otherwise. |
66 | | - The high byte of the central directory's `version made by` field |
67 | | - continues to carry `opts.os` regardless of ZIP64 promotion. |
68 | | -- **Fixed-width metadata writes**: `zip_sync` now uses fixed-width |
69 | | - little-endian writers for ZIP signature, version, mtime, CRC, sizes, |
70 | | - attributes, lengths, and offset fields. The previous variable-width |
71 | | - `wbytes` writes worked only because the output buffer was zero- |
72 | | - initialized; the new writers do not depend on that assumption. |
73 | | - *Note*: this does not change the existing simplified `mtime` |
74 | | - semantics (`opts.mtime` is still stored as raw 4 bytes rather than |
75 | | - converted from Unix seconds to the DOS date+time pair). That bug is |
76 | | - tracked separately. |
77 | | -- **Reserved 0x0001 sanitization**: user-provided extra fields whose |
78 | | - header id is `0x0001` (ZIP64 extended information) are dropped before |
79 | | - the writer emits its own coherent ZIP64 extra payload, so an |
80 | | - attacker-supplied or accidentally-set `0x0001` cannot collide with |
81 | | - fzip's metadata. This is documented as reserved-id behavior. |
82 | | - |
83 | | -#### Cross-tool fixture coverage |
84 | | - |
85 | | -The `moon test` suite now exercises fzip's reader against ZIP64 |
86 | | -archives written by independent tools — both byte literals are |
87 | | -embedded in `src/zip64_fixtures_wbtest.mbt` so they run in CI without |
88 | | -requiring Python or Info-ZIP on the host: |
89 | | - |
90 | | -- Python `zipfile.ZipFile.open(..., force_zip64=True)` (278 bytes, |
91 | | - SHA-256 `3e15b0ee932b51a4...`). |
92 | | -- Info-ZIP `zip -X -fz` (378 bytes, SHA-256 `94463df34e356968b...`). |
93 | | - |
94 | | -The deterministic generator scripts under `tools/zip64-fixtures/` |
95 | | -reproduce the canonical bytes; SHA-256 comments next to the literals |
96 | | -catch silent drift if anyone re-embeds without regenerating. |
97 | | - |
98 | | -#### Public API additions |
99 | | - |
100 | | -- **`pub fn zip_sync_checked(files, opts?) -> FixedArray[Byte] raise |
101 | | - FzipError`** — recoverable-failure variant of `zip_sync`. The two |
102 | | - share the same builder; `zip_sync_checked` raises `FzipError` for |
103 | | - recoverable failures while `zip_sync` traps deterministically with a |
104 | | - stable abort message ("fzip.zip_sync failed; use zip_sync_checked |
105 | | - for recoverable errors: ...") rather than returning a partial or |
106 | | - corrupt archive. |
107 | | -- **`FzipErrorCode::Zip64ValueTooLarge`** — new error variant for |
108 | | - well-formed ZIP64 metadata whose count, size, offset, or final writer |
109 | | - layout cannot be represented or safely indexed by the current `Int` |
110 | | - / `FixedArray`-based sync API. Distinct from `InvalidZipData` (which |
111 | | - remains for malformed archives, unsafe paths, missing required ZIP64 |
112 | | - extras, multi-disk metadata, and similar policy violations). |
113 | | -- **`pub let zip64_eocd_signature : UInt`** — the actual ZIP64 |
114 | | - end-of-central-directory record signature (`0x06064B50U`). The |
115 | | - previously exported `zip64_eocd_locator_signature` was misnamed |
116 | | - (it carried the record signature value); it remains as a |
117 | | - `#alias(.., deprecated)` for source compatibility and will be |
118 | | - dropped in a later major release. |
119 | | -- **`pub let zip64_locator_signature : UInt`** — the ZIP64 EOCD |
120 | | - locator signature (`0x07064B50U`). |
121 | | - |
122 | | -#### Unsupported behavior, intentionally explicit |
123 | | - |
124 | | -- **Multi-disk archives** are rejected with `InvalidZipData` at every |
125 | | - point the reader could detect them (classic EOCD disk fields, ZIP64 |
126 | | - EOCD disk fields, locator disk fields, per-entry disk-number-start, |
127 | | - ZIP64 extra disk-start-number). |
128 | | -- **General-purpose bit 0 (encryption)** is rejected; fzip does not |
129 | | - implement ZIP encryption. |
130 | | -- **True large-file streaming** (>2 GiB entries or archives) remains |
131 | | - Phase 2 work. The sync APIs continue to require both the input and |
132 | | - output to fit in `FixedArray[Byte]`, indexed by `Int`. |
| 5 | +## v0.8.0 - 2026-05-20 |
| 6 | + |
| 7 | +### Added |
| 8 | + |
| 9 | +- ZIP64 metadata support for `zip_sync`, `unzip_sync`, and `unzip_list` when archives and entries still fit the current in-memory sync API limits. |
| 10 | +- ZIP writer emission of ZIP64 extra fields, ZIP64 EOCD records, and ZIP64 EOCD locators when classic ZIP fields need sentinel values. |
| 11 | +- `zip_sync_checked(files, opts?)`, a raising variant of `zip_sync` for recoverable ZIP writer validation errors. |
| 12 | +- `FzipErrorCode::Zip64ValueTooLarge` for ZIP64 values that are valid metadata but cannot be represented safely by the current `Int`/`FixedArray` sync APIs. |
| 13 | +- `zip64_eocd_signature` and `zip64_locator_signature`; the old `zip64_eocd_locator_signature` name remains as a deprecated alias. |
| 14 | +- ZIP data-descriptor entry support for the sync reader, using central-directory sizes and CRC-32. |
| 15 | + |
| 16 | +### Changed |
| 17 | + |
| 18 | +- ZIP reading now validates central-directory and local-header bounds before extraction, including EOCD comment length, ZIP64/classic EOCD consistency, extra-field length, local-header signature, and entry data range. |
| 19 | +- ZIP extraction now verifies each stored or deflated entry against the central-directory CRC-32 and caps total sync output and entry fan-out. |
| 20 | +- `str_from_u8` now rejects malformed UTF-8 according to RFC 3629 instead of accepting continuation-byte starts, bad continuation bytes, overlong encodings, surrogates, and out-of-range code points. |
| 21 | +- `gunzip_sync` now validates reserved GZIP flags, FEXTRA bounds, ISIZE range, ISIZE versus `max_output_size`, and final output length. |
| 22 | +- `zip_sync` now writes ZIP metadata with fixed-width little-endian helpers and removes user-provided extra fields with header id `0x0001` before emitting its own ZIP64 extra field. |
| 23 | + |
| 24 | +### Fixed |
| 25 | + |
| 26 | +- Fixed ZIP reader integer-overflow risks in central-directory, local-header, compression-ratio, and ZIP32 field handling. |
| 27 | +- Fixed ZIP64 EOCD locator detection and conditional ZIP64 extended-information extra-field parsing. |
| 28 | +- Fixed DEFLATE inflate handling for dictionary-backed LZ77 back-references that were fully satisfied by the dictionary. |
| 29 | +- Fixed malformed dynamic-Huffman handling by rejecting `HLIT > 286`, `HDIST > 30`, invalid repeat-16 placement, and code-length repeat overflows. |
| 30 | +- Fixed fixed-output-buffer inflate paths so undersized caller buffers return `FzipError` instead of writing past capacity. |
| 31 | + |
| 32 | +### Tests and docs |
| 33 | + |
| 34 | +- Added ZIP64 design documentation in `docs/zip64.md`. |
| 35 | +- Added embedded ZIP64 fixtures produced by Python `zipfile` and Info-ZIP, with generator scripts under `tools/zip64-fixtures/`. |
| 36 | +- Added regression tests for ZIP64 parsing/writing, ZIP bounds checks, GZIP header/ISIZE validation, DEFLATE malformed input handling, and strict UTF-8 decoding. |
133 | 37 |
|
134 | 38 | ## v0.7.0 - 2026-05-16 |
135 | 39 |
|
|
0 commit comments