Data pipeline#4
Merged
Merged
Conversation
The PR that fixed the stack extra (commit 4da34d3) updated pyproject.toml but didn't include the corresponding uv.lock regeneration. This commit catches the lock up: adds arc-stack and its transitive deps (torch, pytorch-lightning, scvi-tools, wandb, tensorboard, ...) so `uv sync --extra stack` reads from the lock instead of re-resolving each time. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
scripts/build/build_drug_xref.py resolves each drug name to a PubChem CID (canonical), InChIKey, and DrugBank ID via the PubChem PUG REST + UniChem APIs. Output: data/static/drug_xref.parquet (655 rows, sha256-tracked in data/static/manifest.json). Runtime loader fmharness.data.drug_xref.load_drug_xref() reads the parquet and verifies sha256 against the manifest; refuses to return on mismatch. Exposes resolve_cid(), canonical_cids() (vectorized), and overlap_report() for the substrate-gap analysis. .gitignore: whitelist data/static/ (asset dir, plan section 9), anchor the stock `build/` pattern to the project root so scripts/build/ is tracked, and add docs/datasets.md to the local-only list. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
src/fmharness/data/loaders/gdsc2_sarcoma.py:
- Verify data/raw/gdsc2_sarcoma/manifest.json shas (refuse on mismatch)
- Filter DepMap Model.csv to OncotreeLineage in {Soft Tissue, Bone}
with a populated COSMICID (join key into GDSC2 dose-response)
- Filter GDSC2 fitted_dose_response on sarcoma COSMIC set
- Slice DepMap RawReadCount csv to cohort ACH IDs
(IsDefaultEntryForModel="Yes" to dedupe; index_col=0 to absorb the
CSV's leading unnamed index column -- previously leaked through as
a 19,216th "Unnamed: 0" gene)
- pydeseq2 fit_size_factors -> median-of-ratios normalized matrix
(closes the upstream-pipeline confounder with Soragni's pre-computed
counts -- same normalization method)
- Resolve drug names via fmharness.data.drug_xref; attach PubChem CID
/ InChIKey / DrugBank ID per assay
- Emit one Patient + Sample + BaselineExpression per ACH ID, two
DrugAssay rows per (cell, drug) for IC50 and AUC
- Wrap in a content-hashed Tranche
Verified end-to-end: cohort is 28 cell lines x 295 drugs x 2 metrics
= 14,746 DrugAssay rows; 80% of drug names resolved to CIDs.
DepMap RNA-seq coverage is the rate-limiter (28 of 56 sarcoma lines
with COSMIC IDs also have RNA-seq in 26Q1). Subtypes: Osteo 10, Ewing
6, then 12 singleton/doubleton subtypes -- Day-6 LSO splits will need
to coarse-grain.
9 hermetic tests covering sarcoma filter, dedupe filter, xref
attachment, content-hash determinism, manifest mismatch refusal, and
two-metric emission.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
src/fmharness/data/loaders/soragni.py: read pre-computed normalized counts + drug screen + manifest, intersect to the matched cohort, emit a content-hashed Tranche + Patient/Sample/DrugAssay/ BaselineExpression objects. 12 hermetic tests cover ID variants, straggler-column filter, organoid-only attachment, xref attachment, content-hash determinism, and manifest mismatch refusal. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
src/fmharness/splits/ with three splitters sharing a Splitter Protocol + SplitFold dataclass: - StratifiedInDistribution: sklearn StratifiedKFold on Patient.subtype. - LeavePatientOut: N folds for N patients. - LeaveSubtypeOut: one fold per unique subtype label. granularity="fine" uses Patient.subtype verbatim; granularity="coarse" requires a subtype_map for collapsing of rare singletons into families (Embryonal Rhabdo + Alveolar Rhabdo -> Rhabdomyosarcoma, etc.). Every fold records the seed verbatim so it flows into the EnvironmentSnapshot of downstream PredictionRecords. SplitFold itself asserts no train/test patient_id overlap in __post_init__ -- the property the plan calls out. require_split(splitter) guards downstream callers (probe, evaluator) from running unsplit predictions. 22 tests: SplitFold invariant, LPO completeness, LSO no-subtype-overlap + coarse-mapping + missing-subtype handling + single-subtype rejection, Stratified determinism (same seed = same folds) + different-seed differs + rare-subtype collapse, require_split rejection, and a cross-splitter parametrized no-overlap test. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1. Silence the missing-stubs cascade in pyproject.toml. pandas, anndata,
pydeseq2, synapseclient -- none ship PEP-561 type stubs, and strict
mode treats every method call on them as reportUnknownMemberType /
reportUnknownArgumentType / etc. Disable those four cascade rules
(and reportMissingTypeStubs) while keeping strict mode's real checks
on our own code (typos, missing returns, literal-type mismatches,
schema constraints).
2. Fix the residual real type issues that surface once the cascade is
silenced:
- Extract maybe_int / maybe_str into src/fmharness/data/_pandas_utils.py
and handle np.integer / np.floating (parquet's nullable Int64 round-trips
as np.int64, which the prior isinstance(int) checks missed
- cast(pd.DataFrame, ...) where boolean indexing returns the
DataFrame | Series | ndarray union.
- list(...) instead of set in .isin() calls (pandas's stubs don't list
set as an acceptable values type).
- pd.Index(sample_ids) when passing list[str] to DataFrame.__init__.
- Splitter Protocol: list[SplittablePatient] -> Sequence[SplittablePatient]
so list[Patient] / list[_P] satisfy it under invariance.
- Test cleanups: type annotation on splitter_factory, removed unused
_covers helper, assert bundle.expression.X is not None before .dtype.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Scripts for downloading initial experiment data and for performing the test/training splits.