v0.37.0

BREAKING CHANGE

• requires genoray>=3; VCF/PGEN _chunk_ranges_with_length now
  both yield (data, end, chunk_idxs).

Feat

• write: add structured progress reporting
• svar2: route INFO/FORMAT fields into variant-windows output
• svar2: route INFO/FORMAT fields into RaggedVariants
• svar2: discover store INFO/FORMAT fields and skip SVAR1 lazy-loading
• svar2: decode_variants_from_svar2_readbound returns INFO/FORMAT field buffers
• svar2: field-byte gather with vk_src/dense provenance in split decode
• svar2: Svar2Store retains store_path for field-sidecar paths
• adopt genoray 3.0.0 API (privatized SVAR2 FFI, available_samples, no Reader, ContigNormalizer moved)
• consume reconciled genoray chunk_idxs contract (genoray 3.0)
• svar2: unphased_union for variant-windows (ploidy-1 fold)
• svar2: unphased_union for variants mode (ploidy-1 fold)
• svar2: variant-windows read path (ref=window, alt window/allele)
• dataset: svar2 haplotype-realigned tracks via HapsTracks
• dataset: Svar2Haps reconstructor + svar2 read dispatch (haplotypes, variants)
• rust: read-bound svar2 per-hap diffs FFI (for jitter shift computation)
• rust: read-bound svar2 variants/variant-windows decode
• rust: read-bound svar2 track re-alignment kernel
• svar2: read-bound haplotype kernel (one all-Rust FFI call)
• rust: link genoray_core (query-only) + Svar2Store pyclass
• write: _write_from_svar2 6-array ranges cache + dispatch
• dataset: Svar2Link resolution/fingerprint + Metadata.svar2_link
• svar2: additive Python SVAR2 reconstruction adapter (gvl M6b)
• svar2: PyO3 fused wrappers for two-source reconstruct/realign (gvl M6b)
• svar2: two-source track realign driver (gvl M6b)
• svar2: two-source reconstruct driver via closure source (gvl M6b)
• svar2: per-hap var_key⋈dense merge + key decode source (gvl M6b)
• threads: add GVL_FORCE_PARALLEL to bypass the size gate

Fix

• deps: bump genoray to Windows-compatible rev
• write: harden structured progress publication
• svar2: global fill-seed for multi-contig FlankSample tracks (#267)
• svar2: raise (not assert) on max_ends packing-width overflow
• svar2: pack bounded tie-break in max_ends key (fix >2Mb overflow)
• svar2: reject extend_to_length=False for .svar2 sources
• svar2: drop over-strict anchor region guard; assert at read site
• svar2: raise PyValueError on non-contiguous / OOB Python input
• svar2: guard serial unsafe carve path with monotonicity debug_assert
• svar2: validate open(var_fields=), size buffered output, de-degenerate oracle
• svar2: honor Dataset.open(var_fields=) and cover multi-contig windows fields
• write: copy variant index across filesystems on EXDEV
• dataset: guard svar2 FlankSample multi-contig + annotated tracks
• dataset: svar2 min_af/max_af guard + variants jitter guard + coverage
• svar2: byte-size split_to_flat presence buffer; cover dense_snp + byte boundary
• threads: overwrite ambient RAYON_NUM_THREADS with resolved cap (#263)
• threads: honor CFS cpu quota in CPU detection (#263)

Refactor

• svar2: hoist shared present_bit to svar2::present_bit
• svar2: drop dead annot_* capability from readbound haps kernel
• svar2: move test-only oracles out of the shipped package
• svar2: drop unused region_starts from the ranges cache
• svar2: migrate gvl read-bound Rust to genoray Range API
• svar2: adopt genoray HapRanges for gather_haps_readbound
• svar2: hoist Any import + document union guard omission
• dataset: retire svar2 live overlap_batch dispatch (oracle-only)

Perf

• svar2: vectorize _svar2_region_max_ends (byte-identical)
• svar2: fuse variant decode, stream-merge from gather
• svar2: skip identity row-reorder for single-contig getitem
• svar2: read-bound getitem optimization results summary
• svar2: split_to_flat asm fix — hoist q=h/ploidy division out of hot loops (byte-identical)
• svar2: debug_assert guard for decode_variants_from_split get_unchecked invariant
• svar2: decode_variants_from_split asm fix — hoist q=h/ploidy div, unchecked present_bit (byte-identical)
• svar2: re-profile native layer after B1-B3; enumerate cargo-asm work-list
• svar2: compute the pos/ilen ragged reorder index once in variants decode
• svar2: pre-size split_to_flat + decode_variants_from_split allocations (byte-identical)
• svar2: skip redundant pre-reconstruct gather for deterministic haplotype reads
• svar2: native-layer baseline profile (perf DSO/symbol/callgraph + instr reference)
• svar2: Python-layer baseline profile (cProfile + pyinstrument)
• svar2: live read-bound Dataset.getitem profiling driver + pyinstrument
• ffi: release the GIL around rayon parallel regions

v0.36.0

v0.36.0 — The Rust Migration

This is the largest release since 2.0. GenVarLoader's entire core read and write path has been ported from Python/numba to a self-contained Rust crate behind a thin PyO3 (abi3) binding, and numba has been removed from GVL's own code entirely. Python keeps only the ergonomic surface — Dataset indexing, torch integration, validation and error messages — and dispatches into Rust for everything else.

The work landed as Phases 0–5 of the Rust migration roadmap (50+ commits, ~30k lines, mostly differential-parity tests and docs). Every kernel was ported under a strict contract: implement in Rust → differential-test byte-identical against the old numba implementation on property-generated inputs → flip the default and delete the numba version. main stayed shippable at every step.

---

⚠️ Breaking changes

On-disk dataset format → 2.0 (migration required)

Track intervals are now stored struct-of-arrays (starts.npy / ends.npy / values.npy sharing offsets.npy) instead of array-of-structs. This lets the memmaps cross the Python→Rust boundary zero-copy, and fixes an out-of-memory defect at sample-scale where the old layout forced a full contiguous copy on every batch.

Datasets written by gvl < 0.36 must be migrated. Opening one now raises:

ValueError: Dataset at <path> uses format version 1.x but this genvarloader
expects 2.0.0. Run `genvarloader.migrate('<path>')` to upgrade it in place.

Migration

import genvarloader as gvl

gvl.migrate("path/to/dataset")   # in place, then open as usual

gvl.migrate is in-place, streaming, and crash-safe (peak extra disk is a single track's interval store; metadata is bumped last via atomic replace). It's idempotent — a no-op on an already-2.0 dataset — and leaves genotypes, regions, and the reference untouched. No re-write() needed.

Other breaking changes

• GVL_BACKEND removed. The numba/rust runtime backend switch and _dispatch.py are gone — Python now calls Rust directly. If you set GVL_BACKEND anywhere, drop it (it's a no-op now).
• Unsupported track types now raise TypeError at write time (the dead custom IntervalTrack write path was removed).

---

✨ New

• gvl.migrate(path) — the in-place 1.x → 2.0 dataset migrator described above (new public API).
• Self-contained Rust crate. The core data structures and algorithms are now a cargo test-able Rust crate (114 Rust tests), usable from Rust directly and built on the pyo3-free seqpro-core Ragged layout — no reimplementation in GVL.
• Rust-accelerated variant-windows assembly — an entire Python assembly pass for windowed/bare-allele variant output now happens in a single Rust call.

---

⚡ Performance

Final single-thread rust-vs-numba Dataset.__getitem__ A/B (the last apples-to-apples comparison before numba was deleted; chr22_geuv, single thread). Rust is parity-or-better on every output mode:

Output mode	rust ÷ numba
tracks-only	1.07×
haplotypes / tracks+seqs	1.66×
annotated	1.43×
variants	1.38×
variant-windows	4.58×

Highlights:

• Five fused __getitem__ kernels (plain / annotated / spliced / annotated-spliced haplotypes, plus fused tracks) each cross the FFI boundary exactly once.
• gvl.write() / update() are fully Rust-backed: single-pass streaming bigWig writer and a COITrees-based table/annotation overlap engine. The bigWig write slice is ~1.88× faster with ~28% less allocation than the legacy path.
• Rayon batch parallelism on every read kernel, gated byte-identical to the serial result and only engaged above a size threshold (production-scale batches, e.g. SEQLEN ≥ 131072 or BATCH ≥ 256).
• GVL no longer contributes the ~3 GB llvmlite JIT cost it used to (note: seqpro still pulls numba transitively for now — its own numba removal is upstream).

---

📦 Dependencies

• Requires seqpro ≥ 0.20 (Rust-backed Ragged) and the seqpro-core 0.1.0 crate.
• awkward dropped from the hot paths.

---

🔬 Correctness notes

• The Rust port is byte-identical to the previous numba implementation across the differential-parity suite. Two narrow sub-domains are intentionally excluded from the parity oracle because numba was the buggy side there (the #242-family start >= contig_len interval clip, and a reconstruct trailing-fill under-write) — Rust is correct in both, and these are now covered by direct oracle tests.
• Verification at release: 973 pytest passed / 44 skipped / 5 xfailed / 0 failed; 114 cargo tests passed; ruff, pyrefly, and clippy clean.

---

Full changelog (conventional commits)

Feat

• rayon: parallelize get_diffs_sparse + intervals_to_tracks (C3)
• rayon: parallelize shift_and_realign_tracks_sparse and tracks_to_intervals (Task C2)
• rayon: parallelize reconstruct_haplotypes_from_sparse with rayon batch parallelism
• delete numba backend — rust-only read path (Phase 5 W5)
• rust: fuse annotated+spliced haplotype reconstruction into one FFI crossing (Phase 5 W3)
• register rc_alleles dispatch (rust default, seqpro reference)
• rust: rc_alleles PyO3 wrapper + registration
• rust: rc_alleles_inplace core for variant-allele RC
• rust: debug_assert to_rc mask length in kernel RC blocks
• fold strand RC into rust kernels; numba post-pass retained as oracle
• rust: optional in-kernel RC for spliced haplotype kernel
• rust: optional in-kernel RC for annotated haplotype kernel
• rust: optional in-kernel reverse for track realign kernel
• rust: optional in-kernel RC for reconstruct_haplotypes_fused
• rust: optional in-kernel RC for get_reference
• rust: in-place reverse/reverse-complement primitives for read path
• py: assemble_variant_buffers numba oracle, rust shim, and dict parity harness
• ffi: assemble_variant_buffers_{u8,i32} pyfunctions
• variants: assemble_windows_mode (token windows + bare alleles)
• variants: assemble_variants_mode (alt/ref bytes + flank tokens)
• variants: add fetch_windows reference-read helper
• variants: add tokenize/slice_flanks/assemble_alt_window cores
• ffi: zero-copy boundary guard for sample-scale memmaps
• migrate: add gvl.migrate for 1.x AoS -> 2.0 SoA
• open: gate dataset open on format_version major
• format: store track intervals as struct-of-arrays (gvl 2.0)
• intervals: route intervals_to_tracks through backend dispatch (default rust)
• ffi: Rust intervals_to_tracks + ffi seam module
• utils: route splits_sum_le_value through backend dispatch (default rust)
• ffi: Rust splits_sum_le_value + ffi seam module
• dispatch: backend registry for Rust migration strangler window
• ragged: route to_padded through seqpro-core Rust bridge
• rust: consume seqpro-core via rlib; add ragged_to_padded bridge

Fix

• rayon: debug_assert offset monotonicity in C1 carve; correct test comment
• env: keep conda numba pin (seqpro needs working libllvmlite); guard stays own-code
• threads: remove conditional numba import; update thread tests for pure-OS detection
• test: drop stale _recon_mod.intervals_to_tracks spy (B2 removed that import)
• test: restore generate_goldens regeneration; clean dead GVL_BACKEND in bench conftest (W5 B1)
• reconstruct,tracks: pad full tail in numba trailing-fill on ref overshoot
• reconstruct: pad full tail when ref exhausted, not from index 0
• test: add init.py to disambiguate test_write collision; ruff fmt
• variants: drop unused ArrayView2 import
• indexing: SpliceIndexer.parse_idx double-applies sample-subset map
• intervals: clip sub-query interval starts in both kernels (#242)
• tracks: clamp writable_ref when deletion extends past track end
• reconstruct: guard non-annotated parity test against numba SystemError; correct rayon-deferral comment
• reconstruct: strengthen SAFETY comments; rename batch test to match serial-only impl
• reconstruct: clamp writable_ref when ref_idx past contig end; skip numba annotated flake
• reference: revert padded_slice leniency — mirror numba's loud failure for start>=clen (parity twin)
• test: update stale _gather_v_idxs_ss import after Task 5 rename; lint/docstring cleanup
• variants: gather_rows must preserve data dtype (dosage/custom fields)
• bench: profile variants variable-length (with_len is meaningless for variants)
• stub: sync genvarloader.pyi — bigwig_intervals rename + intervals_to_tracks
• test: drop stale rv._rag access in flat_mode_equivalence after Ragged subclass refactor
• dispatch: guard isinstance(Ragged) sites for RaggedVariants subclass

Refactor

• delete numba kernels; numpy fallbacks for #231 dtype paths
• backend: B2 — collapse backend-conditional branches; delete GVL_BACKEND/_active_backend
• dispatch: B1 — replace all get() call sites with direct rust calls, delete _dispatch
• rust: extract reverse::rc_row shared helper
• write: delete dead legacy track path + splits_sum_le_value
• drop unreachable spliced variant-RC guard
• route variant-allele RC through dispatched rc_alleles kernel
• genotypes: delete dead filter_af kernel + its dead test (superseded by inline numpy)
• variants: RaggedVariants subclasses Ragged, drop _rag composition

Perf

• rust: fuse rc_alleles_inplace — 186→308 instrs (rc_row inlined), drop Vec alloc + rescan
• rust: tune reconstruct_haplotypes_from_sparse — 2839→1279 instrs, 0.655→0.589 rust÷numba
• rust: tune rc_flat_rows_inplace — 212→283 instrs (vectorized), 0.664→0.635 rust÷numba
• rust: tune assemble_alt_window — 518→727 asm lines (memcpy-expanded), 1.146→0.835 ms/batch
• rust: tune slice_flanks — 389→429 total instrs (hot-pa...

v0.35.0

Feat

• promote gvl.Table to public API; remove experimental subpackage
• route Table/annot writes through Rust; vectorize contig norm
• back gvl.Table with Rust COITrees engine; drop polars-bio
• rust: streaming Table writer + PyO3 RustTable methods
• rust: materialize ordered intervals from offsets
• rust: COITrees overlap count for RustTable
• rust: table interval store + RustTable::build
• warn when opening datasets with variant-truncated track windows (#233)
• route annotation bigWig writes through rust behind switch
• route per-sample bigWig writes through rust behind GVL_RUST_BIGWIG_WRITE
• PyO3 binding for bigwig_write_track
• rust single-pass streaming bigWig write_track

Fix

• rag-variants: string-key field access + getattr for record fields
• rag-variants: getitem preserves leading fixed axes for slice/array indexing
• rag-variants: _share_offsets uses inner char offsets for opaque-string fields
• adapt GenVarLoader to rust-backed seqpro _core.Ragged backend
• un-skip annot e2e test; guard empty annot; prune dead Rust field
• floor track-write window at the input region (#233 follow-up)
• silence E402 for required sys.path bootstrap in bench corpus builder
• surface bigWig write_track I/O and contig errors as PyRuntimeError instead of panic

Refactor

• write: drop unreachable nomask branch; defensive max init; test empty-group region max
• write: per-region max + concat without awkward
• torch: to_nested_tensor accepts _core.Ragged only
• chunked: detect RaggedVariants by type, not ak.Array
• flat-variants: build RaggedVariants via _core.Ragged
• haps: allele layout returns _core.Ragged; AF filter without awkward
• splice: splice_map as _core.Ragged; drop awkward aggregations
• ragged: prepend_pad_itv via seqpro.rag.concatenate; drop awkward RC helpers
• shm: serialize RaggedVariants via _core buffers (no awkward)
• rag-variants: nested-tensor batch from char-view buffers; drop awkward walker
• rag-variants: pad via seqpro.rag.concatenate (empty-group sentinels)
• rag-variants: rc_ via flat allele-level reverse_complement_masked
• rag-variants: to_packed via record Ragged; drop awkward packing helpers
• rag-variants: record-Ragged wrapper skeleton (construction, access, indexing)
• make rust the default bigWig write path; delete legacy + switch

Perf

• vectorize _ragged_stack_tracks (remove per-batch Python loop on interval hot path)

v0.34.0

Feat

• gather custom FORMAT fields into RaggedVariants like dosage (#231)
• haps: load custom FORMAT fields into Haps.var_field_data (#231)
• discover genoray custom FORMAT fields in available_var_fields (#231)

Fix

• off-by-one in bigWig interval start collapsed wide annot tracks (#233)
• resolve svar override path in _lazy_load_custom_fields (#231)

v0.33.0

Feat

• realign_tracks setting decouples track re-alignment from seq mode
• flat interval reconstruction via FlatIntervals
• add FlatIntervals flat-buffer interval container

Fix

• update RefTracks→SeqsTracks unit tests; pin flat seqs; trim dead union

Refactor

• generalize RefTracks into SeqsTracks (seqs + un-realigned tracks)

v0.32.0

Feat

• add gvl.update for atomic post-hoc track addition
• parallelize gvl.write over track categories (variants first)
• gvl.write accepts annot_tracks
• polars-bio annotation extraction + bigwig annot source

Fix

• guard duplicate track names in update; flat_ends symmetry; test polish
• restore DATASET_FORMAT_VERSION attribute docstring placement
• pyrefly search-path resolves local source; drop stale ignore and unused import

Refactor

• remove Dataset.write_annot_tracks and _annot_to_intervals
• _write_track takes explicit out_dir; add _write_ragged_intervals

v0.31.0

Feat

• table: ship polars-bio as the table extra
• table: make gvl.Table an opt-in experimental feature

Fix

• write: raise clear ValueError when variant/track sample intersection is empty (#225)

v0.30.0

Feat

• dataset: fold haplotypes into ploidy-1 union in variant decode (#222)
• dataset: reject haplotypes/annotated output under unphased_union (#222)
• dataset: report ploidy=1 and fold n_variants under unphased_union (#222)
• dataset: add unphased_union flag on Haps + with_settings (#222)

Fix

• dataset: keep n_variants int32 under unphased_union fold (#222)

Refactor

• variants: drop retired order-dependent germline-CCF inference (#222)

Perf

• threads: tolerate malformed GVL_NUM_THREADS at import (#221)
• variant-windows: single fused fetch for both-window decode (#221)
• flanks: fuse 3 ref-window fetches into 1 via flank slicing (#221)
• reference: dispatch get_reference kernel serial/parallel (#221)
• reference: dispatch fetch kernel serial/parallel by per-thread bytes (#221)
• threads: cap numba workers to cgroup cores + add dispatch predicate (#221)

v0.29.0

Feat

• flat: dummy padding for flank_tokens and variant-windows in get_variants_flat
• flat: thread unknown_token onto Haps for dummy token fill
• flat: fill_empty_groups fills flank_tokens with unknown_token
• flat: _FlatVariantWindows.fill_empty_groups (all-unknown_token dummy)
• flat: _fill_empty_fixed kernel for flank_tokens empty-row dummy fill
• flat: double_buffered carries flat buffers without awkward re-wrap
• flat: slice_chunk handles flat containers
• flat: shm write/read flat types (kind 1/2/3) without awkward
• flat: instance-axis getitem on flat containers
• flat: with_settings(dummy_variant=...) + export DummyVariant + non-variant guard
• flat: Haps.dummy_variant + apply empty-group fill in get_variants_flat
• flat: DummyVariant + empty-group fill kernels + fill_empty_groups
• flat: export FlatVariantWindows and VarWindowOpt
• flat: VarWindowOpt-driven per-allele variant-windows (window|allele matrix)
• flat: VarWindowOpt + per-allele window/allele computation + optional window fields
• flat: register variant-windows kind end-to-end
• flat: attach flank_tokens / emit windows from get_variants_flat
• flat: ref/alt window assembly + tokenization
• flat: _FlatWindow + _FlatVariantWindows two-level token types
• flat: compute ride-along flank tokens from reference
• flat: thread flank_length + token LUT settings onto Haps
• flat: byte->int token LUT builder for flank tokenization
• flat: A flat variant decode (get_variants_flat) with no awkward
• flat: A0 flat passthrough for seqs/haps/annot/reference outputs
• flat: output_format field + with_output_format + QueryView.flat_output
• flat: export FlatRagged/FlatAnnotatedHaps/FlatVariants/FlatAlleles
• flat: _FlatVariants/_FlatAlleles types with to_ragged()

Fix

• torch: reject variant-windows output over buffered transport up front
• flat: allow dummy_variant with variant-windows output kind
• flat: _fill_empty_seq preserves input dtype for token windows
• flat: reject flat+buffered variants with flank tokens at construction
• flat: harden _FlatVariants slice for flank_tokens; test polish
• flat: with_settings(dummy_variant=False) is a no-op on non-variant datasets
• flat: narrow _seqs to Haps before flank-field replace (pyrefly)
• flat: clear error for variant-windows + active tracks; drop dead alias
• flat: clear error for VarWindowOpt(ref='allele') on REF-less dataset
• flat: clearer splice-unsupported messages for variant-windows
• flat: windows reshape must keep ploidy dim ([1:-1] not [1:-2])
• flat: guard flank-without-LUT, document Haps fields, test validation paths
• flat: shape-driven _FlatAlleles.to_ragged fixes scalar-scalar squeeze byte-identity
• flat: _FlatAlleles.reshape re-appends ragged axis; docstrings + 2D-index test
• flat: add _FlatAlleles.squeeze so _FlatVariants.squeeze works

Refactor

• flat: clarify flat-variant reader names; annotated shm round-trip test
• flat: retire awkward _get_variants; ragged variants decode via flat path
• flat: normalize _FlatVariants.reshape shape arg; drop redundant import

v0.28.0

Feat

• variants: numba allele-pack kernel + layout decomposition for non-canonical views
• write: reject symbolic/breakend variants from SVAR inputs
• write: reject symbolic/breakend variants from PGEN inputs
• write: reject symbolic/breakend variants from VCF inputs
• write: add consolidated unsupported-variant validator

Fix

• variants: rc_() on sliced/reordered views via materialized copy
• variants: to_packed() handles sliced/reordered alt/ref via numba kernel
• open: default to variants when genotypes have no reference
• write: validate VCF variants before creating output directory