Restore custom loaders#6
Merged
Merged
Conversation
CoderData fed GDSC2 as gene-length-normalized TPM but Soragni as CPM -- an asymmetry that confounds Stack, a single-cell count model (log1p + NB decoder, expects length-free count-derived input). Restore the pre-swap loaders that read raw artifacts directly, on consistent length-free normalization: - gdscv2: DepMap raw counts -> DESeq2 median-of-ratios. Full cell-line panel (~700 lines) by default for transfer training; sarcoma_only=True -> 28 lines for within-sarcoma analyses. Raw counts kept in layers['raw_counts']. - sarcoma: Soragni deposited matrix is CPM (length-free, sums to 1e6, verified); normalization label fixed from the false 'tpm' to 'cpm', response metric corrected to 'viability' (CoderData mislabeled it 'auc'). load_tranche adapts the native bundles to the build_sample_design contract (Entrez var, improve_sample_id/model_type) and dispatches sarcoma/gdscv2 to the native loaders, else CoderData. All 14 scripts + l1000 re-pointed. prep_stack_input now emits length-free CPM for both cohorts (GDSC2 raw counts->CPM, Soragni CPM), renormalized to 1e6 over the shared panel; Stack log1ps internally so no manual log1p. Re-add pydeseq2/openpyxl; restore drug_xref + download scripts. Cohorts: GDSC2 700 full / 28 sarcoma; Soragni 17 patients. Downstream experiments must be re-run on the new length-free cohorts. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two coupled fixes for the GDSC2->Soragni transfer on the reverted loaders. CID drug key: native loaders use per-dataset drug ids (Soragni names, GDSC2 numeric DRUG_IDs) with no shared namespace, so every cross-dataset join was empty. Thread PubChem CID as the canonical key -- build_sample_design gains drug_key='pubchem_cid'; both transfers, the Morgan-fingerprint table, and the L1000 pert maps now key on CID (15 shared GDSC2<->Soragni drugs). Sarcoma-only default: with the join fixed, the drug-specific organoid x drug interaction transfers to Soragni only when trained in-domain (sarcoma cell lines): expr/per_drug interaction +0.17..+0.30, p<=0.035, robust across n_components and frozen-test permutation null. The full pan-cancer panel washes it out (-0.04, n.s.); the general-sensitivity substrate gap (global rho ~ -0.13) is the same either way. transfer_gdsc_soragni now defaults to sarcoma-only (--pan-cancer opts back in); prep_stack_input regains --sarcoma-only so Stack's in-context prompt can be sarcoma-only too. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…sult The GDSC2->Soragni transfer fed log1p(bundle.X), but the reverted loaders leave GDSC2 on DESeq2 median-of-ratios and Soragni on CPM -- a train/test normalization mismatch (the PCA/NMF basis is fit on one scale, applied to the other). Add evaluation.cpm_bundle (raw counts -> CPM when present, else renormalize CPM) and run both cohorts through it, so the transfer is log1p(CPM) end to end. prep_stack_input imports the same helper. This corrects 746e317: the drug-specific interaction it reported (+0.17..+0.30, p<=0.035, "robust across k") was largely a normalization artifact. Under consistent CPM it is not robust -- across k=5/10/15, expr/pca p={0.44,0.11,0.01}, expr/nmf p={0.30,0.03,0.11}, stack/pca p={0.15,0.08,0.10}; the sub-0.05 cells are isolated and inconsistent (noise across 9 tests). Stack tracks PCA-of-expression at every k (no FM advantage). The robust result is the substrate gap: neither general sensitivity nor drug-specific interaction transfers cell-line -> PDTO. Also add an NMF reducer row and drop the shared-slope rows (per-drug only). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Precision-oncology comparison the global models lacked: for the actionable drugs in the Soragni panel, score a single pre-specified biomarker per drug -- target-gene log1p(CPM) expression, or a sensitizing/resistance alteration from WES (SNV/CNV) -- against the organoid's response to the matched drug, and pit it against the global PCA-of-expression model on the same cells. The BIOMARKERS table is pre-specified biological priors (nothing selected on the response data); small cohort (per-biomarker n=3-15) so the pooled p is marginal. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Factor SimpleProbe's per-drug mean and PCA/NMF reduction into a shared ProbeBase, then add KernelProbe: an RBF kernel-ridge residual on the same reduced scores, (alpha, gamma) chosen per drug by leave-one-out. make_head() builds a probe factory by name so a --head flag swaps linear<->kernel without touching the CV harness. Lets the Stack-vs-expression comparison be reported across head families, testing whether the negative result is head-invariant. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The [z, g, z(x)g] construction was copy-pasted in transfer_pharmaformer_lite and per_patient_eval; move it to fmharness.bilinear.bilinear_features as the single source of truth. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
transfer_gdsc_soragni gains --head/--all-heads and writes per (head, rep) metric rows to results/head_invariance.csv; transfer_pharmaformer_lite appends its bilinear row to the same table and uses the shared bilinear_features. The table shows Stack <= expression holds across linear, kernel, and bilinear heads. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Leave-one-patient-out regret@k/recall@k recommendation eval with negative (within-drug permutation) and positive (planted-interaction) controls. The transcriptome policies take --head (linear ridge or RBF kernel), so the positive control exercises the nonlinear apparatus too. Uses the shared bilinear_features helper. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
scripts/_plotting.py centralizes the aggregation process and savefig script (de-dups the two existing plot scripts). plot_data.py emits the data figures (cohort composition, response distributions, organoid x drug heatmap, shared panel) and the head-invariance figure, with no conclusion annotations, into tracked docs/figures/. Add matplotlib as a viz extra. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds Mermaid style evaluation flowchart, results, the head-invariance table/figure, and the data figures. docs/models.md describes each model operationally (input, normalization, scoring contract, coverage), cross-linking the adapter contract and the companion manuscript. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Pre-existing ruff errors (B905, E501) so the tree passes ruff check repo-wide. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR restores native (raw-artifact) dataset loaders for Soragni and GDSC2 instead of relying on the CoderData-only ingestion path, and updates the evaluation harness to use a canonical PubChem CID drug key and a shared CPM normalization so cross-substrate comparisons (GDSC2 → Soragni) are apples-to-apples. It also introduces a nonlinear kernel head (RBF kernel ridge) alongside the existing linear ridge head, and updates scripts/docs/tests accordingly.
Changes:
- Add native Soragni + GDSC2 loaders with manifest verification, drug xref attachment, and adapters into the existing
CoderDataBundledownstream contract. - Add a shared probe base class + head registry, and introduce
KernelProbe(RBF kernel ridge) to support head-invariance experiments. - Update evaluation utilities and scripts to use CPM normalization and PubChem CID as the cross-dataset drug key; expand docs/README and add comprehensive hermetic tests.
Reviewed changes
Copilot reviewed 46 out of 53 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/test_soragni_loader.py | New hermetic end-to-end tests for the Soragni native loader (IDs, cohort intersection, xref, manifest verification). |
| tests/test_l1000.py | Updates L1000 mapping test to reflect PubChem-CID-keyed drug mappings. |
| tests/test_kernel.py | New unit tests for KernelProbe behavior and head registry wiring. |
| tests/test_gdsc2_sarcoma_loader.py | New hermetic end-to-end tests for the GDSC2 native loader (mini DepMap/GDSC2 fixtures + manifest verification). |
| tests/test_drug_xref.py | New hermetic tests for static drug crosswalk loading, validation, and helpers. |
| src/fmharness/schema/assays.py | Extends allowed normalization vocabulary to include cpm. |
| src/fmharness/probe/simple.py | Refactors SimpleProbe to inherit shared logic from ProbeBase. |
| src/fmharness/probe/kernel.py | Adds KernelProbe implementing per-drug (or shared) RBF kernel-ridge residual modeling. |
| src/fmharness/probe/heads.py | Adds head registry and make_head() factory for swapping heads by name. |
| src/fmharness/probe/base.py | Introduces shared probe scaffolding: input checks, drug-mean base, PCA/NMF reduction, constants. |
| src/fmharness/probe/init.py | Exposes KernelProbe and make_head at package level. |
| src/fmharness/l1000.py | Switches to load_tranche and PubChem CID drug keys for L1000 joins/mappings. |
| src/fmharness/evaluation.py | Adds cpm_bundle() and extends build_sample_design() to support configurable drug keys (incl. pubchem_cid). |
| src/fmharness/data/loaders/soragni.py | New Soragni native loader: manifest verification, ID canonicalization, AnnData build, xref attachment, schema emission. |
| src/fmharness/data/loaders/gdsc2_sarcoma.py | New GDSC2 native loader: Model/GDSC2/DepMap joins, DESeq2 normalization, raw_counts layer, xref attachment, schema emission. |
| src/fmharness/data/loaders/adapt.py | Adapts native bundles to the downstream CoderDataBundle contract (Entrez var, improve_sample_id, model_type). |
| src/fmharness/data/loaders/init.py | Re-exports new loader/adapters and establishes load_tranche as the preferred entry point. |
| src/fmharness/data/drug_xref.py | Adds static drug xref loader with manifest sha verification plus helper utilities. |
| src/fmharness/bilinear.py | Extracts shared bilinear feature construction used by structure-aware transfer models. |
| scripts/transfer_pharmaformer_lite.py | Updates structure-aware transfer script to join on PubChem CID and reuse bilinear_features; adds optional CSV output. |
| scripts/transfer_gdsc_soragni.py | Updates transfer benchmark to use load_tranche, CPM normalization, CID drug keys, and swappable heads (--head/--all-heads). |
| scripts/score_viability_adapters.py | Switches Soragni target metric to viability via load_tranche. |
| scripts/prep_stack_input.py | Updates Stack input prep to use CPM (length-free) consistently; adds panel renormalization and improves docs/flags. |
| scripts/predict_expression_baselines.py | Updates Soragni target metric to viability and switches to load_tranche. |
| scripts/plot_pearson.py | Switches to load_tranche + viability and consolidates plotting boilerplate into _plotting. |
| scripts/plot_global_rho.py | Switches to load_tranche + viability and consolidates plotting boilerplate into _plotting. |
| scripts/plot_data.py | New script to generate README data figures and (optionally) head-invariance figure. |
| scripts/per_patient_eval.py | New per-patient clinical-metric evaluation script (regret/recall), with head-invariance and controls. |
| scripts/gdsc_representation_increment.py | Switches to load_tranche for GDSC2 representation-increment analysis. |
| scripts/download/download_soragni.py | New Synapse downloader for Soragni tables with shared manifest schema. |
| scripts/download/download_gdsc2_sarcoma.py | New downloader for GDSC2 + DepMap artifacts with manifest recording/verification. |
| scripts/download/_utils.py | Shared manifest utilities used by download scripts. |
| scripts/check_data_repro.py | Switches to load_tranche + viability for reproducibility checks. |
| scripts/build/build_drug_xref.py | New builder for data/static/drug_xref.parquet with manifest update. |
| scripts/build_stack_panel.py | Switches to load_tranche + viability for measured-gene intersection when building Stack panel. |
| scripts/build_l1000_context.py | Switches to load_tranche + viability for Soragni drug set selection. |
| scripts/bridge_generated_to_auc.py | Switches to load_tranche + viability for bridging generated deltas to Soragni labels. |
| scripts/biomarker_anchored.py | New biomarker-anchored baseline evaluation script for actionable drugs. |
| scripts/benchmark_soragni.py | Switches to load_tranche and selects metric based on dataset (viability vs auc). |
| scripts/baselines_soragni.py | Switches to CID-keyed joins and load_tranche; updates L1000 pert mapping to CID. |
| scripts/_plotting.py | New shared plotting helper module (Agg backend + consistent savefig). |
| README.md | Updates project framing, adds evaluation-design diagram, embeds new data figures, and documents head-invariance workflow. |
| pyproject.toml | Adds runtime deps required by new loaders (pydeseq2, openpyxl) and optional viz extra (matplotlib). |
| docs/models.md | New/expanded model catalogue documenting representations/heads and shared guarantees. |
| data/static/manifest.json | Adds static asset manifest entry for drug_xref.parquet (sha/metadata). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This branch restores custom loaders instead of using the CoderData approach. This was done to match the input transformation between the organoid datasets and the STACK model. In addition, this PR adds a kernel adapter head to adjust for the nonlinear nature of the embedding architecture. It adds more description to the model docs, the adapter contracts, and the README results.