Skip to content

feat(concat_on_disk): support pairwise=True - #2636

Open
alejandro-publius wants to merge 3 commits into
scverse:mainfrom
alejandro-publius:feat/concat-on-disk-pairwise
Open

alejandro-publius wants to merge 3 commits into
scverse:mainfrom
alejandro-publius:feat/concat-on-disk-pairwise

Conversation

@alejandro-publius

Copy link
Copy Markdown

concat_on_disk raised NotImplementedError for pairwise=True and never wrote the concatenation-axis pairwise group. It now writes .{obs,var}p as the block diagonal of each object's pairwise elements, matching concat_pairwise_mapping used by the in-memory concat: keys are intersected or unioned according to join, and an object lacking a key contributes an empty block.

The block diagonal is streamed one object at a time rather than assembled whole: row block i is that object's (n_i, n_i) element padded with zeros to (n_i, N) and appended to the output dataset, so only a single object's pairwise element is resident at once. The result dtype is resolved across all objects before the first block is written, since that write fixes the dtype of the dataset the remaining blocks are appended to; otherwise a leading empty stand-in block would pin the output to bool.

Tests check the result against concat for both file formats and both joins, cover objects that lack a key, and confirm .obsp is left out unless pairwise=True is passed.

This touches the same file as #2558 but is independent of it; whichever lands second will need a trivial rebase.

@codecov

codecov Bot commented Sep 6, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 85.87%. Comparing base (4ded337) to head (68ef855).
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2636      +/-   ##
==========================================
- Coverage   87.64%   85.87%   -1.77%     
==========================================
  Files          49       49              
  Lines        8125     8151      +26     
==========================================
- Hits         7121     7000     -121     
- Misses       1004     1151     +147     
Files with missing lines Coverage Δ
src/anndata/experimental/merge.py 90.33% <100.00%> (+1.85%) ⬆️

... and 8 files with indirect coverage changes

alejandro-publius and others added 3 commits September 14, 2026 23:22
Closes scverse#2414. `concat_on_disk` raised `NotImplementedError` for `pairwise=True`
and never wrote the concatenation-axis pairwise group at all.

It now writes `.{obs,var}p` as the block diagonal of each object's pairwise
elements, matching `concat_pairwise_mapping` used by the in-memory `concat`:
keys are intersected or unioned according to `join`, and an object lacking a key
contributes an empty block.

The block diagonal is built one object at a time rather than assembled whole: row
block `i` is that object's `(n_i, n_i)` element padded with zeros to `(n_i, N)`
and appended to the output dataset, so only a single object's pairwise element is
resident at a time. The result's dtype is resolved across all objects before the
first block is written, since that write fixes the dtype of the dataset the
remaining blocks are appended to -- otherwise a leading empty stand-in block would
pin the output to `bool`.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KAdFWr2Fo9HAtSiSWF4KPU
towncrier fragments are keyed by PR number and use the `feat` type.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KAdFWr2Fo9HAtSiSWF4KPU
…test

`_adatas_to_paths` was replaced by `_adatas_to_stores` plus `_make_store`, so
the default-off test raised NameError. It now builds stores the way the other
tests in this module do.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@alejandro-publius
alejandro-publius force-pushed the feat/concat-on-disk-pairwise branch from 562cea6 to 68ef855 Compare September 16, 2026 01:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support pairwise for concat_on_disk

1 participant