MSL mzLib Spectrum Library indexed binary format by trishorts · Pull Request #1036 · smith-chem-wisc/mzLib

trishorts · 2026-03-14T22:15:27Z

Add `.msl` — mzLib Binary Spectral Library Format

This PR introduces .msl (mzLib Spectral Library), a new binary spectral library format designed as a high-performance alternative to the existing text-based .msp format for use in mzLib and MetaMorpheus workflows.

Motivation

The .msp format requires full sequential parsing on every load, allocates heavily due to string-based fragment annotations, and cannot support random-access fragment retrieval for large libraries. The .msl format addresses these limitations with a compact, structured binary layout that enables zero-copy struct reads, O(1) DDA lookups, zero-allocation m/z range queries, and an index-only load mode for multi-gigabyte libraries.

What's Included

New binary format (Omics + Readers projects)

MslFormat — all format constants, enums, and bitfield encode/decode helpers
MslStructs — five fixed-layout binary structs (Pack = 1) covering header, protein record, precursor record, fragment record, and footer
MslFragmentIon / MslLibraryEntry — rich in-memory models with full ion annotation (internal fragments, neutral losses, diagnostic ions, oligonucleotides, proteoforms)
MslWriter — two-pass atomic writer (temp file + rename); intensity normalization per precursor; fragment m/z sort
MslReader — binary reader with CRC-32 validation and both full-load and index-only modes
MslIndex — in-memory query engine: m/z-sorted array for range queries, O(1) sequence/charge dictionary, elution group map, LRU fragment cache
MslLibrary — top-level user façade with Load, LoadIndexOnly, Save, SaveFromLibrarySpectra, QueryMzWindow, QueryWindow (m/z + RT + IM filters), TryGetEntry, WithCalibratedRetentionTimes
MslFileTypeHandler — registers the .msl extension with mzLib's file-type routing
SpectralLibrary.cs — updated to route .msl files through MslLibrary transparently

Format highlights

Magic bytes MZLB + CRC-32 integrity check + footer trailing magic for truncation detection
Supports peptides, proteoforms, oligonucleotides, and glycopeptides (MoleculeType enum)
Protein table with accession / name / gene; string deduplication table eliminates redundant storage
Fragment flags encode internal ions, diagnostic ions, named neutral losses (H₂O, NH₃, H₃PO₄, HPO₃), and quantification exclusion in a single byte
Format version field for forward-compatible evolution

Tests

100+ NUnit 4 tests across 8 test classes covering writer round-trips, reader validation (including CRC corruption and truncation detection), index queries, regression against a known-good MSP reference file (42 precursors / 968 fragments), and edge cases (empty libraries, Unicode sequences, large fragment counts).

Benchmarks (BenchmarkDotNet, Intel Core i7-8550U, .NET 8, 1 K–500 K precursors at 10 fragments each)

Metric	MSL	MSP	Ratio
Full load — 1 K precursors	1.7 ms	5.6 ms	3.3× faster
Full load — 50 K precursors	174 ms	282 ms	1.6× faster
Allocation — full load	~50% lower	baseline
DDA lookup (O(1) dictionary)	~91 ns	file-seek
Single m/z window query	~113 ns	N/A	zero allocation
Write — 1 K / 50 K / 500 K	8 ms / 178 ms / 1.4 s	N/A	all within spec

codecov · 2026-03-14T22:25:50Z

Codecov Report

❌ Patch coverage is 93.34495% with 191 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.33%. Comparing base (bf1fb12) to head (1d096fc).

Files with missing lines	Patch %	Lines
mzLib/Readers/SpectralLibrary/MslConverter.cs	0.00%	44 Missing ⚠️
mzLib/Readers/SpectralLibrary/MslWriter.cs	95.41%	15 Missing and 26 partials ⚠️
mzLib/Omics/SpectralLibrary/MslFragmentIon.cs	89.23%	13 Missing and 8 partials ⚠️
mzLib/Readers/SpectralLibrary/MslReader.cs	95.37%	6 Missing and 14 partials ⚠️
mzLib/Omics/SpectralMatch/Mslindex.cs	95.20%	3 Missing and 10 partials ⚠️
...zLib/Readers/SpectralLibrary/MslProteoformIndex.cs	91.93%	3 Missing and 7 partials ⚠️
mzLib/Readers/SpectralLibrary/MslLibrary.cs	97.42%	4 Missing and 5 partials ⚠️
...Lib/Readers/SpectralLibrary/MslProteoformScorer.cs	93.43%	5 Missing and 4 partials ⚠️
mzLib/Readers/SpectralLibrary/MslMerger.cs	95.93%	2 Missing and 5 partials ⚠️
mzLib/Readers/SpectralLibrary/MslLibraryData.cs	89.28%	0 Missing and 6 partials ⚠️
... and 3 more

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1036      +/-   ##
==========================================
+ Coverage   81.64%   82.33%   +0.69%     
==========================================
  Files         369      382      +13     
  Lines       47666    50509    +2843     
  Branches     5662     6102     +440     
==========================================
+ Hits        38915    41586    +2671     
- Misses       7647     7734      +87     
- Partials     1104     1189      +85

Files with missing lines	Coverage Δ
mzLib/Omics/SpectralLibrary/MslSpectralLibrary.cs	`100.00% <100.00%> (ø)`
...SpectralMatch/IHasSequenceCoverageFromFragments.cs	`100.00% <ø> (ø)`
...zLib/Readers/SpectralLibrary/Mslfiletypehandler.cs	`100.00% <100.00%> (ø)`
...ts/Koina/AbstractClasses/FragmentIntensityModel.cs	`89.66% <89.65%> (+0.39%)`	⬆️
mzLib/Readers/SpectralLibrary/SpectralLibrary.cs	`89.03% <96.47%> (+3.65%)`	⬆️
mzLib/Omics/SpectralLibrary/MslStructs.cs	`66.66% <66.66%> (ø)`
mzLib/Readers/SpectralLibrary/MslLibraryData.cs	`89.28% <89.28%> (ø)`
mzLib/Readers/SpectralLibrary/MslMerger.cs	`95.93% <95.93%> (ø)`
mzLib/Readers/SpectralLibrary/MslLibrary.cs	`97.42% <97.42%> (ø)`
...Lib/Readers/SpectralLibrary/MslProteoformScorer.cs	`93.43% <93.43%> (ø)`
... and 6 more

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…mentIon ReadFragmentIon_InternalIon_StartAndEndResidue_Correct only checked the start residue (FragmentNumber). Add the secondary product type and end residue (SecondaryProductType / SecondaryFragmentNumber) asserts the name already promised, so the text-parse path is proven to preserve the full internal-ion annotation — not just the binary round-trip test. Addresses PR smith-chem-wisc#1036 review comment (internal-ion annotation completeness). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

MslProteoformScorer.MatchedFragments intentionally carries the library peak m/z/intensity, not the matched experimental peak — it is a library-vs-experiment scorer whose result identifies which library fragments matched. Experimental intensities feed only the spectral-angle score. Document this at the construction site to settle the ambiguity. Addresses PR smith-chem-wisc#1036 review comment (library vs experimental match values). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

SpectralLibrary.cs had been reindented spaces -> tabs across the whole file, which inflated the PR diff by ~730 whitespace-only lines and buried the real .msl routing / internal-ion changes. Repo style is 4-space (no .editorconfig; master uses spaces). Convert leading tabs back to spaces; the file is byte-identical ignoring whitespace, so behavior is unchanged. PR diff for this file drops from 1010/758 to ~252/24. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

TestAveragingSpectraWriteFile.cs (a spectra-averaging teardown rewrite) and TestDatabaseLoaders.cs (removal of an unused helper) are unrelated to the .msl spectral-library format and only widened this PR's surface. Revert both to master to keep the PR focused; the averaging-teardown improvement can land in its own PR. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Strip development-process residue that leaked into source: - Mslindex.cs: drop the "// New property/method to ADD alongside it" instruction comments and re-indent Count/GetEntryAt with tabs to match the file (they were pasted in space-indented); add real XML summaries. - MslLibrary.cs / MslReader.cs: reword four doc comments that referenced an internal "Prompt N / Prompt N Handoff" process, keeping the technical content. No behavior change. Addresses nbollis's smith-chem-wisc#1036 review (agent residue / mechanical assembly). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

… residue The MSL test suite encoded the agent's prompt-development sequence into 16 fixture names (TestMslPromptN...) and ~50 doc/category/string mentions of "Prompt N". Rename every fixture to describe what it tests (e.g. TestMslPrompt13GetAllLibrarySpectraFix -> TestMslGetAllLibrarySpectra; collisions resolved as TestMslMergerCoverage / TestMslCoverageGapsExtended), update all cross-references, drop the [Category("PromptN")] tags, de-prompt temp-dir names, and reword the doc/assert text. Also fixes one stale source reference in MslSpectralLibrary.cs to the renamed version-management test. Test logic and assertions are unchanged. Addresses nbollis's smith-chem-wisc#1036 review (agent residue / mechanical assembly). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

trishorts · 2026-06-10T23:35:58Z

Re: the #1036 review (the one posted on #1038)

Thanks @nbollis — went through every item. Dispositions below.

Addressed

Agent residue / "mechanical assembly" — you were right, and there was more than the one spot. Cleaned in d1eb53e and 92754ed:
- Removed the // New property/method to ADD alongside it instruction comments in Mslindex.cs (and re-indented Count/GetEntryAt, which had been pasted in space-indented; added real XML docs).
- Reworded 4 source doc comments that referenced an internal "Prompt N / Prompt N Handoff" process (MslLibrary.cs, MslReader.cs).
- Renamed all 16 TestMslPromptN… fixtures to descriptive names (e.g. TestMslPrompt13GetAllLibrarySpectraFix → TestMslGetAllLibrarySpectra), dropped the [Category("PromptN")] tags, de-prompted temp-dir names, and reworded ~50 doc/assert mentions. No Prompt N residue remains anywhere in the repo; test logic is unchanged.
Committed BenchmarkDotNet artifacts — untracked in 7ccacdf (the **/BenchmarkDotNet.Artifacts/ ignore rule keeps them out).
TestDatabaseLoaders unexplained deletion — reverted to master in f215f04 (it was unrelated scope-creep).

Already correct (no change needed)

FragmentIntensityModel intensity parsing — the outputIntensities (Outputs[2]) vs outputMZs (Outputs[1]) split is already correct in current code.
FragmentIntensityModel.Predict lock + Task.Run — kept (removing it risks reintroducing the sync-over-async deadlock it was added for); it's already documented at the call site, including the lock's purpose (serializing shared instance state) and the long-term async fix. Happy to split the Koina change into its own PR if you'd prefer.

Respectfully, not bugs — no change

The two 🔴 "span over local array" items (MslProteoformIndex.QueryMassWindow includeDecoys:false, Mslindex.GetElutionGroup). Returning a ReadOnlySpan<T> over a heap T[] (which .ToArray() produces) is safe: the span carries a managed reference that keeps the array GC-rooted for as long as the span is reachable. The lifetime hazard you're describing applies to spans over stackalloc/ref-locals — and the compiler enforces that (CS8347/8352), so a genuine escape wouldn't have built. GetElutionGroup's copy is even intentional (caller-isolation). Glad to add a clarifying comment if useful.

Since the review landed on #1038, flagging here so the threads track against the right PR. Happy to re-request review on #1036 once CI is green.

The PR had bumped the nuspec <version> to a dev/local-feed value (9.9.901) used while building local packages for the MetaMorpheus consumer PRs. Revert to master's 1.0.574 so the package version isn't inflated on merge; release versioning is handled separately. Addresses nbollis smith-chem-wisc#1036 review comment ("Change Nuspec back"). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…s override The override ran batched Torch forward passes for speed, but its results are not bit-identical to per-peptide prediction (~1e-7 drift from batched BatchNorm/matmul), which broke two pre-existing exact-equality tests (PredictRetentionTimeEquivalents_MatchesSinglePredictions and _LargeBatch) and contradicted its own "results identical to PredictCore" doc. It is not needed here: the method is declared on IRetentionTimePredictor and implemented in the base RetentionTimePredictor, so Chronologer now inherits the base implementation. The consumers (MetaMorpheus parallel search RT calibration) call the interface method and keep working; only a one-time calibration step loses the batched speedup. This file now matches master. Batched inference can return as its own PR with tolerance tests. Addresses nbollis smith-chem-wisc#1036 review ("Is this method needed? ... use the existing method"). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

… corruption Building spectral libraries calls Chronologer for millions of peptides. The ~30 tensor intermediates allocated inside each forward() were relying on the GC finalizer thread for disposal, which races libtorch's native allocator and corrupts the heap (0xC0000374) once enough inference has run. Wrap each Predict chunk (and the single-peptide path) in NewDisposeScope() so those tensors are freed deterministically on the calling thread. Also adds an opt-in CUDA path (MZLIB_CHRONOLOGER_CUDA=1; default CPU, numerically equivalent) behind a Device property; inert unless the env var is set. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The batched PredictRetentionTimeEquivalents override runs the model at batch size m, while PredictRetentionTimeEquivalent runs it at batch size 1. libtorch selects different conv/matmul kernels for the two paths, so predictions agree only to float32 precision (~1e-7 relative), not bit-exactly. Two assertions used Is.EqualTo with no tolerance and failed on that drift. Add .Within(1e-4) - orders of magnitude above the noise floor yet tight enough to still catch a genuine batch/single mismatch. Also soften the override docstring, which claimed the two paths give identical results. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The base RetentionTimePredictor.PredictRetentionTimeEquivalents parallelizes single-peptide PredictCore calls across CPU threads — each peptide is still a separate, lock-held, batch-of-1 libtorch forward pass. Override it for Chronologer with true tensor batching: format/encode the peptides in parallel on the CPU, then run the model in large batched forward passes (one [m,1] tensor per chunk, model lock taken once per chunk instead of once per peptide). The model is in eval mode (BatchNorm running stats), so each peptide's prediction is independent of the batch; results match PredictCore to float32 precision (libtorch picks different conv/matmul kernels at batch size m vs 1). Also wraps the forward pass in NewDisposeScope() so its native intermediates are freed deterministically. Test tolerance widened to allow the float32 drift. Extracted from smith-chem-wisc#1036 so the batched-inference work lands independently of the .msl format changes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…ictor Remove the MZLIB_CHRONOLOGER_CUDA opt-in and the Device plumbing: the model always runs on CPU and the batched override no longer moves tensors to a device. An env-gated execution mode is the wrong surface, the GPU path was never used in the production library build, and if GPU inference is wanted later it should be an explicit constructor parameter with benchmarks. Keeps the batched-CPU override and the NewDisposeScope() native-heap fix. Mirrors smith-chem-wisc#1077. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…Search Offline library compiler for MetaMorpheus's ManySearchTask: digests protein FASTAs once and writes compact indexed .msl libraries (precursor mass + charge + Chronologer iRT + sequence, lean / fragment-less by default) so the parallel search reads pre-built libraries instead of re-digesting thousands of FASTAs at search time. Supports per-database output, a merged db|accession-tagged index, and size-balanced sharded production builds with a global persisted iRT cache. Added as a new project under mzLib/MslDbBuilder referencing UsefulProteomics- Databases and Readers. Depends on the .msl writer/reader API from smith-chem-wisc#1036, so its base is mzlib_speclib. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…guard Document why MslFragmentIon is a standalone persistence/working model rather than a subclass of or interface over the MatchedFragmentIon domain type (different layer, mutable float32 storage record, no eager Product / neutral mass, carries storage-only ExcludeFromQuant; the two are bridged only by explicit conversion). Also document why the two annotation renderings intentionally differ. Add MslFragmentIonInteropTests: round-trips a diverse fragment set (terminal, neutral-loss, internal, diagnostic) through ToLibrarySpectrum/FromLibrarySpectrum asserting every mapped field survives, so the two models can't silently drift; and pins that ExcludeFromQuant is dropped by the domain projection (it round-trips only through the binary .msl format). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…Search Offline library compiler for MetaMorpheus's ManySearchTask: digests protein FASTAs once and writes compact indexed .msl libraries (precursor mass + charge + Chronologer iRT + sequence, lean / fragment-less by default) so the parallel search reads pre-built libraries instead of re-digesting thousands of FASTAs at search time. Supports per-database output, a merged db|accession-tagged index, and size-balanced sharded production builds with a global persisted iRT cache. Added as a new project under mzLib/MslDbBuilder referencing UsefulProteomics- Databases and Readers. Depends on the .msl writer/reader API from smith-chem-wisc#1036, so its base is mzlib_speclib. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…1077) The base RetentionTimePredictor.PredictRetentionTimeEquivalents parallelizes single-peptide PredictCore calls across CPU threads — each peptide is still a separate, lock-held, batch-of-1 libtorch forward pass. Override it for Chronologer with true tensor batching: format/encode the peptides in parallel on the CPU, then run the model in large batched forward passes (one [m,1] tensor per chunk, model lock taken once per chunk instead of once per peptide). The model is in eval mode (BatchNorm running stats), so each peptide's prediction is independent of the batch; results match PredictCore to float32 precision (libtorch picks different conv/matmul kernels at batch size m vs 1). Also wraps the forward pass in NewDisposeScope() so its native intermediates are freed deterministically. Test tolerance widened to allow the float32 drift. Extracted from #1036 so the batched-inference work lands independently of the .msl format changes. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Drop the ExternalReaders/Python and ExternalReaders/Rust reference readers (msl_reader.py, test_msl_reader.py, msl_reader.rs); they were interop reference code, not part of the C# build. Scrub the now-dangling version-sync references: trim the new-version checklist in MslSpectralLibrary.cs and repurpose the version canary test as FormatVersion_MustMatchExpectedConstant. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

trishorts · 2026-06-18T00:28:04Z

Removed the Python and Rust external MSL reader implementations (ExternalReaders/Python and ExternalReaders/Rust), and scrubbed the now-dangling version-sync references in MslSpectralLibrary.cs and the version canary test. Build is clean and the full unit suite passes (Test.dll: 4617 passed, 0 failed).

MICHAEL SHORTREED and others added 29 commits November 18, 2021 12:30

correct Within calculation

a6b1639

update unit tests

fa4da8b

conflicts resolved back to upstream

3246567

Merge remote-tracking branch 'upstream/master'

a018d4d

Merge remote-tracking branch 'upstream/master'

15a37d0

this is the spot

892fa45

Merge remote-tracking branch 'upstream/master'

211013c

Merge branch 'master' of https://github.com/trishorts/mzLib

68104ee

Merge remote-tracking branch 'upstream/master'

d715a08

Merge remote-tracking branch 'upstream/master'

3565522

Merge remote-tracking branch 'upstream/master'

72e7b53

Merge remote-tracking branch 'upstream/master'

593872a

Merge branch 'master' of https://github.com/trishorts/mzLib

42dd034

Merge remote-tracking branch 'upstream/master'

fbeaec0

Merge remote-tracking branch 'upstream/master'

614ded7

Merge branch 'master' of https://github.com/trishorts/mzLib

47307c8

Merge remote-tracking branch 'upstream/master'

28e05ae

Merge remote-tracking branch 'upstream/master'

0a7c609

Merge remote-tracking branch 'upstream/master'

630d8c7

Merge branch 'master' of https://github.com/trishorts/mzLib

f6a386b

Merge remote-tracking branch 'upstream/master'

d673800

Merge branch 'master' of https://github.com/trishorts/mzLib

675a0ae

Merge remote-tracking branch 'upstream/master'

15d4baf

Merge remote-tracking branch 'upstream/master'

03ca9f7

Merge remote-tracking branch 'upstream/master'

d0a4c79

Merge remote-tracking branch 'upstream/master'

894b998

Merge remote-tracking branch 'upstream/master'

dd0c8c9

Merge remote-tracking branch 'upstream/master'

c6310ae

Merge remote-tracking branch 'upstream/master' into mzlib_speclib

34063e7

trishorts and others added 4 commits June 10, 2026 17:36

trishorts changed the title ~~Mzlib speclib~~ MSL mzLib Spectrum Library indexed binary format Jun 10, 2026

nbollis reviewed Jun 10, 2026

View reviewed changes

Comment thread mzLib/mzLib.nuspec

Comment thread mzLib/Chromatography/RetentionTimePrediction/Chronologer/ChronologerRetentionTimePredictor.cs

trishorts and others added 2 commits June 10, 2026 18:13

trishorts and others added 4 commits June 10, 2026 18:37

This was referenced Jun 15, 2026

Add MslDbBuilder: FASTA→.msl spectral-library builder for ManySearchTask #1076

Open

Batched Chronologer retention-time inference (PredictRetentionTimeEquivalents override) #1077

Merged

Merge branch 'master' into mzlib_speclib

8fce55d

nbollis previously approved these changes Jun 16, 2026

View reviewed changes

trishorts dismissed nbollis’s stale review via 1d096fc June 18, 2026 00:26

pcruzparri approved these changes Jun 18, 2026

View reviewed changes

nbollis approved these changes Jun 18, 2026

View reviewed changes

trishorts merged commit 78275fb into smith-chem-wisc:master Jun 18, 2026
4 checks passed

trishorts mentioned this pull request Jun 19, 2026

chore(deps): bump mzLib to 1.0.580 smith-chem-wisc/MetaMorpheus#2673

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

MSL mzLib Spectrum Library indexed binary format#1036

MSL mzLib Spectrum Library indexed binary format#1036
trishorts merged 93 commits into
smith-chem-wisc:masterfrom
trishorts:mzlib_speclib

trishorts commented Mar 14, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Mar 14, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

trishorts commented Jun 10, 2026

Uh oh!

trishorts commented Jun 18, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

trishorts commented Mar 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Add .msl — mzLib Binary Spectral Library Format

Motivation

What's Included

Uh oh!

codecov Bot commented Mar 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

trishorts commented Jun 10, 2026

Re: the #1036 review (the one posted on #1038)

Addressed

Already correct (no change needed)

Respectfully, not bugs — no change

Uh oh!

trishorts commented Jun 18, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

trishorts commented Mar 14, 2026 •

edited

Loading

Add `.msl` — mzLib Binary Spectral Library Format

codecov Bot commented Mar 14, 2026 •

edited

Loading