Skip to content

Decide row streaming per workbook, and say when it costs something - #135

Merged
billdenney merged 2 commits into
masterfrom
claude/constant-memory
Jul 29, 2026
Merged

Decide row streaming per workbook, and say when it costs something#135
billdenney merged 2 commits into
masterfrom
claude/constant-memory

Conversation

@billdenney

Copy link
Copy Markdown
Collaborator

Adds constant_memory and constant_memory_threshold to write_xlsx(), and
replaces "stream unless a known-bad feature is present" with an explicit,
ordered decision.

They sit on write_xlsx() rather than xl_properties() because streaming
governs how the file is produced, not what the document contains.
use_zip64 is the same kind of knob.

The rules

  1. constant_memory = FALSE → off, without evaluating anything else.
  2. A blacklisted feature → off, absolutely. TRUE cannot override it, and
    is warned about instead.
  3. constant_memory = TRUE → on, overriding the size estimate.
  4. Estimated extra memory below constant_memory_threshold (default 128 MiB)
    → off.
  5. Otherwise → on.

Rule 2 is deliberately not overridable. Three of the four blacklisted features
fail loudly under streaming — tables are refused outright, merges and embedded
images hit dimension checks. But a multi-cell array formula writes a file
that opens cleanly in Excel and is quietly missing cells, which is
jmcnamara/libxlsxwriter#522, closed upstream as won't-fix. No warning makes
that acceptable, so the request is refused rather than honoured.

Rule 4 is new. Streaming only pays for itself on large data, and not streaming
also produces a smaller file, since repeated strings go to the shared string
table instead of being written inline.

Where the estimate comes from

110 bytes per cell, measured rather than guessed — against nycflights13 at
three sizes and three column mixes:

shape cells bytes/cell
repetitive text 200k → 2M 37 → 82
numeric 700k → 7M 79 → 107
mixed 950k → 9.5M 87 → 112

Repetitive text is cheapest, because the shared string table deduplicates it.
The high end is used, so the estimate errs towards keeping streaming on — which
at scale is the faster path as well as the leaner one.

Cells are summed across the workbook, not per sheet: libxlsxwriter holds
every sheet's cell table until the file is closed.

What streaming actually costs

Peak process memory, staged so R's share can be separated from the C library's.
The prep row runs everything write_xlsx() does up to but not including the
.Call:

stage 337k rows 1.01M rows
R + package, no data 92 MB 93 MB
data frame loaded 253 MB 365 MB
all R-side resolution, before .Call 253 MB 463 MB
write, streaming on 253 MB 458 MB
write, streaming off 931 MB 2,608 MB

Essentially all of the difference is libxlsxwriter's cell table: R's whole
contribution is 253/463 MB, streaming adds nothing measurable, and not
streaming adds 678 MB and 2,145 MB. At 1M rows the C library wants 4.6× what
R does, and takes 65s against 27s.

(Measuring this needs bin/x64/Rterm.exe — both Rscript.exe and bin/R.exe
are launchers and report a 7 MB shim.)

The suggestion

When a blacklisted feature turns streaming off and the workbook is large
enough for it to matter, write_xlsx() now says so, naming the feature and the
estimated cost. Silent below the threshold, where there is nothing to act on.

Test churn, and why

The threshold changes the default for small workbooks from streaming to not,
which moves strings from inline to the shared table. Ten tests asserted the
inline spelling.

Each now passes constant_memory = TRUE, because that is what they were
actually testing — does this feature force streaming off? — a question the
size rule had been answering for them by accident. request = TRUE isolates
the feature's effect from the size rule, which is a sharper test than before.

Also included

Two things that rode along with this work rather than belonging to it:

  • Sheet-qualified ranges. .xl_resolve_range() now parses Data!A1:B5,
    'My Sheet'!A1:B5 and 'It''s'!A1, and a sheet element in the list form.
    It is opt-in via allow_sheet: an autofilter, merge, validation,
    conditional format, table, image anchor and print area all apply to the sheet
    they are on, so a sheet name there is a misunderstanding to report, not a
    refinement to honour. Only chart series can genuinely point elsewhere, which
    is what this is for. Everything else now rejects one with a clear message
    instead of ignoring it.

  • lxw_table_column.format is ignored for data written by the caller jmcnamara/libxlsxwriter#520 resolved. Not a bug — a documented
    limitation, and the maintainer's recommended fix is what writexl already
    does. The actionable detail is that both the column plan and the table
    column's own format are needed, the latter becoming dataDxfId, "for strict
    correctness with Excel". A test now pins both, because after the column-plan
    fix the dataDxfId looks like the redundant one.

Verification

  • 1747 tests passing, 0 failures, 0 warnings.
  • R CMD check: 0 errors, 0 warnings, 1 NOTE — the pre-existing spelling
    wordlist.
  • 13 files changed, +275 / −27.

write_xlsx() gains constant_memory and constant_memory_threshold.  They sit
there rather than on xl_properties() because streaming governs how the file is
produced, not what the document contains -- use_zip64 is the same kind of knob.

Five rules, in order:

  1. constant_memory = FALSE -> off, without evaluating anything else.
  2. a blacklisted feature   -> off, absolutely.  TRUE cannot override it and
     is warned about instead: three of the four fail loudly under streaming,
     but a multi-cell array range writes a file that opens cleanly and is
     quietly missing cells, which no warning makes acceptable.
  3. constant_memory = TRUE  -> on, overriding the size estimate.
  4. estimated extra memory below the threshold -> off.
  5. otherwise -> on.

Cells are summed across the workbook, not per sheet: libxlsxwriter holds every
sheet's cell table until the file is closed.

The 110 B/cell estimate is calibrated, not guessed -- measured against
nycflights13 at three sizes and three column mixes, from 37 B/cell for highly
repetitive text (the shared-string table deduplicates it) to 112 for mixed
columns at ~10M cells.  The high end is used so the estimate errs towards
streaming, which at a million rows is 2.4x faster as well as 5x leaner.

Measuring that needed care: Rscript.exe and bin/R.exe are both launchers, so
peak working set has to be read from bin/x64/Rterm.exe or it reports a 7 MB
shim.  Staged runs stopping before the .Call show where the memory goes -- R
holds 253 MB for 337k rows and libxlsxwriter adds nothing while streaming and
678 MB without it; at 1M rows, 463 MB against 2145 MB.  Essentially all of the
difference is the C library's cell table.

The threshold changes the default for small workbooks from streaming to not,
which moves strings from inline to the shared table.  Ten tests asserted the
inline spelling; each now asks for constant_memory = TRUE, which is what they
were really testing -- "does this feature force streaming off?" -- a question
the size rule had been answering for them by accident.  Smaller workbooks also
come out 3-12% smaller this way, since repeated strings are shared.
@billdenney
billdenney force-pushed the claude/constant-memory branch from 78ccd5c to 659d9f1 Compare July 29, 2026 13:05
…stimate

Three small corrections to the previous commit.

The constant_memory_threshold documentation gave a byte count without saying
it is estimated.  The cost is inferred from the cell count and a fixed per-cell
figure; the true cost varies with the data and is lowest for text that repeats.
A reader could otherwise reasonably expect the threshold to be compared against
something measured.

The suggestion message is now emitted inside .resolve_constant_memory() rather
than handed back for write_xlsx() to print.  That function is the only place
that knows both why streaming was refused and what it cost; returning the text
made the caller responsible for a decision it could only re-derive.  `note` is
gone from the return value as a result.

`reasons` stays, and is worth being explicit about: nothing in R/ reads it --
it exists so tests can assert *why* a decision was made, not merely what it
was.  Without it a test could not tell "off because of a merge" from "off
because the workbook is small", and a rule misfiring in either direction would
still pass.  That is a real use, so it is documented as one rather than left
looking like an oversight.

Moving the message revealed that two tests were emitting it incidentally; they
now suppress it, and a new test asserts the decision is silent whenever there
is nothing to report.  write_xlsx() is called constantly on small frames, and a
message on every one would be noise.
@billdenney
billdenney merged commit f314564 into master Jul 29, 2026
12 checks passed
@billdenney
billdenney deleted the claude/constant-memory branch July 29, 2026 13:40
billdenney added a commit that referenced this pull request Jul 29, 2026
Brings in the constant_memory decision procedure (#135).

One conflict, in R/write_xlsx.R: both sides inserted at the same point in the
roxygen block, master adding the constant_memory and
constant_memory_threshold @PARAM entries and this branch adding
@family workbook settings.  Both are wanted, so both are kept, with the params
before the family tag as in every other file.

Checked rather than assumed: all 28 @family tags survive with their feature
groupings, none reverted to "writexl", and the merged files still carry the
constant_memory and allow_sheet work from master.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant