Decide row streaming per workbook, and say when it costs something - #135
Merged
Conversation
write_xlsx() gains constant_memory and constant_memory_threshold. They sit
there rather than on xl_properties() because streaming governs how the file is
produced, not what the document contains -- use_zip64 is the same kind of knob.
Five rules, in order:
1. constant_memory = FALSE -> off, without evaluating anything else.
2. a blacklisted feature -> off, absolutely. TRUE cannot override it and
is warned about instead: three of the four fail loudly under streaming,
but a multi-cell array range writes a file that opens cleanly and is
quietly missing cells, which no warning makes acceptable.
3. constant_memory = TRUE -> on, overriding the size estimate.
4. estimated extra memory below the threshold -> off.
5. otherwise -> on.
Cells are summed across the workbook, not per sheet: libxlsxwriter holds every
sheet's cell table until the file is closed.
The 110 B/cell estimate is calibrated, not guessed -- measured against
nycflights13 at three sizes and three column mixes, from 37 B/cell for highly
repetitive text (the shared-string table deduplicates it) to 112 for mixed
columns at ~10M cells. The high end is used so the estimate errs towards
streaming, which at a million rows is 2.4x faster as well as 5x leaner.
Measuring that needed care: Rscript.exe and bin/R.exe are both launchers, so
peak working set has to be read from bin/x64/Rterm.exe or it reports a 7 MB
shim. Staged runs stopping before the .Call show where the memory goes -- R
holds 253 MB for 337k rows and libxlsxwriter adds nothing while streaming and
678 MB without it; at 1M rows, 463 MB against 2145 MB. Essentially all of the
difference is the C library's cell table.
The threshold changes the default for small workbooks from streaming to not,
which moves strings from inline to the shared table. Ten tests asserted the
inline spelling; each now asks for constant_memory = TRUE, which is what they
were really testing -- "does this feature force streaming off?" -- a question
the size rule had been answering for them by accident. Smaller workbooks also
come out 3-12% smaller this way, since repeated strings are shared.
billdenney
force-pushed
the
claude/constant-memory
branch
from
July 29, 2026 13:05
78ccd5c to
659d9f1
Compare
…stimate Three small corrections to the previous commit. The constant_memory_threshold documentation gave a byte count without saying it is estimated. The cost is inferred from the cell count and a fixed per-cell figure; the true cost varies with the data and is lowest for text that repeats. A reader could otherwise reasonably expect the threshold to be compared against something measured. The suggestion message is now emitted inside .resolve_constant_memory() rather than handed back for write_xlsx() to print. That function is the only place that knows both why streaming was refused and what it cost; returning the text made the caller responsible for a decision it could only re-derive. `note` is gone from the return value as a result. `reasons` stays, and is worth being explicit about: nothing in R/ reads it -- it exists so tests can assert *why* a decision was made, not merely what it was. Without it a test could not tell "off because of a merge" from "off because the workbook is small", and a rule misfiring in either direction would still pass. That is a real use, so it is documented as one rather than left looking like an oversight. Moving the message revealed that two tests were emitting it incidentally; they now suppress it, and a new test asserts the decision is silent whenever there is nothing to report. write_xlsx() is called constantly on small frames, and a message on every one would be noise.
billdenney
added a commit
that referenced
this pull request
Jul 29, 2026
Brings in the constant_memory decision procedure (#135). One conflict, in R/write_xlsx.R: both sides inserted at the same point in the roxygen block, master adding the constant_memory and constant_memory_threshold @PARAM entries and this branch adding @family workbook settings. Both are wanted, so both are kept, with the params before the family tag as in every other file. Checked rather than assumed: all 28 @family tags survive with their feature groupings, none reverted to "writexl", and the merged files still carry the constant_memory and allow_sheet work from master.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
constant_memoryandconstant_memory_thresholdtowrite_xlsx(), andreplaces "stream unless a known-bad feature is present" with an explicit,
ordered decision.
They sit on
write_xlsx()rather thanxl_properties()because streaminggoverns how the file is produced, not what the document contains.
use_zip64is the same kind of knob.The rules
constant_memory = FALSE→ off, without evaluating anything else.TRUEcannot override it, andis warned about instead.
constant_memory = TRUE→ on, overriding the size estimate.constant_memory_threshold(default 128 MiB)→ off.
Rule 2 is deliberately not overridable. Three of the four blacklisted features
fail loudly under streaming — tables are refused outright, merges and embedded
images hit dimension checks. But a multi-cell array formula writes a file
that opens cleanly in Excel and is quietly missing cells, which is
jmcnamara/libxlsxwriter#522, closed upstream as won't-fix. No warning makes
that acceptable, so the request is refused rather than honoured.
Rule 4 is new. Streaming only pays for itself on large data, and not streaming
also produces a smaller file, since repeated strings go to the shared string
table instead of being written inline.
Where the estimate comes from
110 bytes per cell, measured rather than guessed — against
nycflights13atthree sizes and three column mixes:
Repetitive text is cheapest, because the shared string table deduplicates it.
The high end is used, so the estimate errs towards keeping streaming on — which
at scale is the faster path as well as the leaner one.
Cells are summed across the workbook, not per sheet: libxlsxwriter holds
every sheet's cell table until the file is closed.
What streaming actually costs
Peak process memory, staged so R's share can be separated from the C library's.
The
preprow runs everythingwrite_xlsx()does up to but not including the.Call:.CallEssentially all of the difference is libxlsxwriter's cell table: R's whole
contribution is 253/463 MB, streaming adds nothing measurable, and not
streaming adds 678 MB and 2,145 MB. At 1M rows the C library wants 4.6× what
R does, and takes 65s against 27s.
(Measuring this needs
bin/x64/Rterm.exe— bothRscript.exeandbin/R.exeare launchers and report a 7 MB shim.)
The suggestion
When a blacklisted feature turns streaming off and the workbook is large
enough for it to matter,
write_xlsx()now says so, naming the feature and theestimated cost. Silent below the threshold, where there is nothing to act on.
Test churn, and why
The threshold changes the default for small workbooks from streaming to not,
which moves strings from inline to the shared table. Ten tests asserted the
inline spelling.
Each now passes
constant_memory = TRUE, because that is what they wereactually testing — does this feature force streaming off? — a question the
size rule had been answering for them by accident.
request = TRUEisolatesthe feature's effect from the size rule, which is a sharper test than before.
Also included
Two things that rode along with this work rather than belonging to it:
Sheet-qualified ranges.
.xl_resolve_range()now parsesData!A1:B5,'My Sheet'!A1:B5and'It''s'!A1, and asheetelement in the list form.It is opt-in via
allow_sheet: an autofilter, merge, validation,conditional format, table, image anchor and print area all apply to the sheet
they are on, so a sheet name there is a misunderstanding to report, not a
refinement to honour. Only chart series can genuinely point elsewhere, which
is what this is for. Everything else now rejects one with a clear message
instead of ignoring it.
lxw_table_column.formatis ignored for data written by the caller jmcnamara/libxlsxwriter#520 resolved. Not a bug — a documentedlimitation, and the maintainer's recommended fix is what writexl already
does. The actionable detail is that both the column plan and the table
column's own format are needed, the latter becoming
dataDxfId, "for strictcorrectness with Excel". A test now pins both, because after the column-plan
fix the
dataDxfIdlooks like the redundant one.Verification
R CMD check: 0 errors, 0 warnings, 1 NOTE — the pre-existing spellingwordlist.