Data downloaders#3
Merged
Merged
Conversation
Two downloaders under scripts/download/ that fetch raw artifacts to data/raw/. Broaden .gitignore to exclude all of data/. Add openpyxl. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Arc Institute's perturbation foundation model is named STACK. Update the docs (adapter contract, environment) and the optional-dependency extra in pyproject.toml from `state` (pulling arc-state) to `stack` (pulling stack from the ArcInstitute GitHub repo). Adapter contract also gains a note on STACK's native predict_in_context() zero-shot path as the predict_native() example. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three changes shipped together because they share one goal -- make the
two downloaders interchangeable for the loaders and the verifier:
- Extract sha256_file, manifest read/write, skip-on-sha-match, and
verify-many-manifests into scripts/download/_utils.py. Both downloaders
now produce manifests sharing one top-level shape (dataset, release,
updated_utc, files{sha256, bytes, source_uri, ...extras}).
- Soragni gains --verify-only (walks _test/, tables/, and root manifests).
--tables and --test now skip-on-matching-sha256 so reruns are cheap and
silently changed Synapse Table content fails loudly.
- DepMap file swap: replace log2(TPM+1) protein-coding with the raw-counts
variant (OmicsExpressionRawReadCountHumanProteinCodingGenes.csv, same
canonical-id .27). The GDSC2 loader will run pydeseq2 median-of-ratios
on this so both cohorts share Soragni's normalization scheme.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The MVP uses the pre-computed normalized counts at syn64333318 instead of local quantification from FASTQ (documented in plan section 5/6/11), so --test (smoke FASTQ download) and the default --full sync of syn55180195 (~50+ GB) are out of scope. Strip both modes plus the helpers they relied on (FASTQ_SUFFIXES, collect_file_index, walk_files' loose form). What's left: --list (inspect the project), --verify-only (re-check tables/manifest.json shas), and a tables fetch as the new default. FASTQ fallback per the section 10 risk row would require reintroducing a FASTQ-pull path. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two adjustments so `uv sync --extra stack` resolves: - The dependency spec must use the upstream package's distribution name (arc-stack, declared in github.com/ArcInstitute/stack pyproject), not its Python import name (stack). - Hatchling refuses direct URL references in dependencies unless explicitly opted in, so add [tool.hatch.metadata] allow-direct-references = true. Verified: `uv sync` clean, `uv sync --extra stack` builds arc-stack from the GitHub repo. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adding scripts for reproducible downloads of published PDTO data and GDSC2 sarcoma cell lines