Skip to content

Commit 3efef61

Browse files
antonkulagaantonkulaga
committed
added summaries
1 parent 37ef2cd commit 3efef61

9 files changed

Lines changed: 2829 additions & 36 deletions

File tree

‎batch_extract.json‎

Lines changed: 1102 additions & 0 deletions
Large diffs are not rendered by default.

‎docs/DATASET_CARD.md‎

Lines changed: 113 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,113 @@
1+
---
2+
license: cc-by-4.0
3+
task_categories:
4+
- text-generation
5+
- token-classification
6+
- text-classification
7+
language:
8+
- en
9+
tags:
10+
- biology
11+
- single-cell
12+
- genomics
13+
- gene-expression
14+
- cell2sentence
15+
- age-prediction
16+
- longevity
17+
size_categories:
18+
- 1M<n<10M
19+
---
20+
21+
## Dataset Card: longevity-genie/cell2sentence4longevity-data
22+
23+
### Summary
24+
This repository contains preprocessed single-cell RNA-seq (scRNA‑seq) datasets prepared as “cell sentences” for training and evaluation of cells2sentence-style models. Each cell is represented as a space‑separated sequence of top expressed gene symbols, enabling language‑model style training for tasks such as biological age prediction and other downstream applications.
25+
26+
This dataset targets fine‑tuning and evaluation of models inspired by cells2sentence approaches for cellular phenotyping, including age prediction as described in the preprint: [cells2sentence: Sequence models on gene expression](https://www.biorxiv.org/content/10.1101/2025.04.14.648850v3.full).
27+
28+
### What are “cell sentences”?
29+
For each cell, we rank genes by expression and keep the top N (default 2000). We filter out Ensembl IDs and keep valid gene symbols, then serialize them as a whitespace‑separated string. This converts a numeric high‑dimensional cell profile into a token sequence amenable to language‑model training.
30+
31+
### Supported tasks and use cases
32+
- Age prediction from single‑cell expression profiles
33+
- Tissue/organ classification
34+
- Cell type labeling and transfer
35+
- Condition/disease stratification and dataset harmonization
36+
- Few‑shot or instruction‑style fine‑tuning of sequence models on cells
37+
38+
### Data sources and provenance
39+
- Source data are public scRNA‑seq h5ad datasets, primarily from the CZI CellxGene collections.
40+
- When a dataset is detected as CellxGene (by UUID), we add `dataset_id` and, where available via cached collections metadata, join publication information:
41+
- `collection_id`, `publication_title`, `publication_doi`, `publication_description`, `publication_contact_name`, `publication_contact_email`.
42+
- The pipeline is streaming and memory‑efficient, and uses Polars for processing.
43+
44+
### Repository structure
45+
Each source dataset is organized under its own subfolder. There are two common layouts:
46+
- Train/test split (default):
47+
- `<dataset_name>/train/chunk_*.parquet`
48+
- `<dataset_name>/test/chunk_*.parquet`
49+
- Single split (if train/test split is disabled):
50+
- `<dataset_name>/chunk_*.parquet` or `<dataset_name>/chunks/chunk_*.parquet`
51+
52+
### Data fields (columns)
53+
Columns are inherited from the input AnnData `.obs` table, plus generated fields:
54+
- `cell_sentence` (string): space‑separated gene symbols for the cell (top‑N expression).
55+
- `age` (float): numeric age extracted from `development_stage` where parsable (years). Cells with null age are filtered by default for training splits.
56+
- `dataset_id` (string, optional): CellxGene dataset UUID when detected.
57+
- Publication fields (optional, when join succeeds): `collection_id`, `publication_title`, `publication_doi`, `publication_description`, `publication_contact_name`, `publication_contact_email`.
58+
- Other `.obs` fields (optional, dataset‑specific): e.g., `organism`, `tissue`, `cell_type`, `assay`, `sex`, `disease`, etc.
59+
60+
Notes:
61+
- In current train/test outputs, the standardized column is `age` (years) when extractable from `development_stage`. Some upstream datasets encode mouse age in months; those may not map into `age` unless present in a parsable “year‑old” format.
62+
63+
### Preparation pipeline (high level)
64+
1. Read h5ad in backed mode (streaming).
65+
2. Map genes to symbols (HGNC lookup where helpful); filter out Ensembl IDs from sentences.
66+
3. Build `cell_sentence` from top expressed genes per cell (default top‑N = 2000).
67+
4. Extract `age` from `development_stage` when available (numeric years).
68+
5. Optionally add `dataset_id` and join publication metadata if the dataset is found in CellxGene collections cache.
69+
6. Filter cells with null `age` by default (for consistent age‑based tasks).
70+
7. Write Parquet chunks and, by default, produce train/test split stratified by `age` (~95/5).
71+
72+
### How to use
73+
Below is an example for downloading the repository snapshot and loading with Polars. This approach is scalable and keeps a local cache.
74+
75+
```python
76+
from pathlib import Path
77+
import polars as pl
78+
from huggingface_hub import snapshot_download
79+
80+
repo_id = "longevity-genie/cell2sentence4longevity-data"
81+
local_dir = Path(snapshot_download(repo_id=repo_id, repo_type="dataset"))
82+
83+
# Example: load train split for one dataset folder
84+
dataset_name = "10cc50a0-af80-4fa1-b668-893dd5c0113a" # replace with any available subfolder
85+
train_glob = local_dir / dataset_name / "train" / "chunk_*.parquet"
86+
test_glob = local_dir / dataset_name / "test" / "chunk_*.parquet"
87+
88+
train_df = pl.scan_parquet(str(train_glob)).collect()
89+
test_df = pl.scan_parquet(str(test_glob)).collect()
90+
91+
# Basic checks
92+
assert "cell_sentence" in train_df.columns
93+
assert "age" in train_df.columns
94+
```
95+
96+
You can iterate across all dataset subfolders to build training mixtures, or concatenate multiple datasets at scan‑time for large‑scale training pipelines.
97+
98+
### Limitations and caveats
99+
- Not all datasets provide a reliably parsable human age; cells with null `age` are filtered for the default split.
100+
- For mouse datasets that encode months (e.g., “24m”), month handling may appear in metadata extraction utilities but train/test outputs standardize on `age` when parsable as years.
101+
- `.obs` schema varies across sources; presence of optional fields is dataset‑dependent.
102+
103+
### Licensing
104+
- This repository aggregates preprocessed derivatives of public scRNA‑seq datasets. The original data remain under their respective licenses (see the source collection pages on CellxGene and corresponding publications). Please respect upstream licensing and citation requirements when using the data.
105+
- The dataset card and pipeline code are provided under the project’s license; data licensing follows the upstream sources.
106+
107+
### Citation
108+
If you use this dataset, please cite:
109+
- cells2sentence preprint: “Sequence models on gene expression.” BioRxiv, 2025. [Link](https://www.biorxiv.org/content/10.1101/2025.04.14.648850v3.full)
110+
- CellxGene data portal and the individual source publications for datasets included in this collection.
111+
112+
### Contact
113+
Maintainer: `longevity-genie` on Hugging Face. Issues and improvements are welcome.

‎docs/EXPLORE_CLI.md‎

Lines changed: 134 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,134 @@
1+
# Explore CLI - AnnData Metadata Extraction Tool
2+
3+
A memory-efficient CLI tool for extracting metadata fields from h5ad AnnData files into Polars DataFrames saved as Parquet files.
4+
5+
## Installation
6+
7+
The `explore` command is automatically installed with the project:
8+
9+
```bash
10+
uv sync
11+
```
12+
13+
## Usage
14+
15+
### Single File Extraction
16+
17+
Extract specific metadata fields from a single h5ad file:
18+
19+
```bash
20+
uv run explore extract <h5ad_file> -f <field1> -f <field2> -f <field3>
21+
```
22+
23+
**Example:**
24+
```bash
25+
uv run explore extract data/input/sample.h5ad \
26+
-f development_stage \
27+
-f cell_type \
28+
-f tissue \
29+
-o data/output/sample_meta.parquet
30+
```
31+
32+
**Options:**
33+
- `--field`, `-f`: Field name to extract (can be specified multiple times) - **required**
34+
- `--output`, `-o`: Output parquet file path (default: `<h5ad_name>_meta.parquet`)
35+
- `--chunk-size`: Number of rows to process at a time (default: 10000)
36+
- `--compression`: Compression algorithm (default: zstd)
37+
- `--compression-level`: Compression level (default: 3)
38+
- `--log-dir`: Directory for log files
39+
40+
### Batch Extraction
41+
42+
Extract metadata fields from multiple h5ad files in a directory:
43+
44+
```bash
45+
uv run explore batch <input_dir> <output_dir> -f <field1> -f <field2>
46+
```
47+
48+
**Example:**
49+
```bash
50+
uv run explore batch data/input data/output/metadata \
51+
-f development_stage \
52+
-f cell_type \
53+
-f tissue \
54+
-f donor_id \
55+
--log-dir logs/batch_extract
56+
```
57+
58+
**Options:**
59+
- `--field`, `-f`: Field name to extract (can be specified multiple times) - **required**
60+
- `--chunk-size`: Number of rows to process at a time (default: 10000)
61+
- `--compression`: Compression algorithm (default: zstd)
62+
- `--compression-level`: Compression level (default: 3)
63+
- `--log-dir`: Directory for log files
64+
- `--skip-existing` / `--overwrite`: Skip files that already have output (default: skip-existing)
65+
66+
## Output Format
67+
68+
All output files are saved as Parquet files with the `_meta` suffix:
69+
- Single file: `<input_name>_meta.parquet`
70+
- Batch mode: `<input_name>_meta.parquet` for each input file
71+
72+
The output files contain only the requested metadata fields as columns, extracted from the `adata.obs` DataFrame.
73+
74+
## Memory Efficiency
75+
76+
The tool is designed to be memory-efficient:
77+
1. **Backed mode**: H5ad files are opened in `backed='r'` mode, keeping data on disk
78+
2. **Chunked processing**: Data is processed in chunks (default: 10,000 rows at a time)
79+
3. **Streaming**: Row slices are extracted first before selecting columns
80+
4. **Direct conversion**: Data flows from AnnData → Polars → Parquet without intermediate copies
81+
82+
This allows processing very large h5ad files (millions of cells) on systems with limited RAM.
83+
84+
## Examples
85+
86+
### Extract age-related metadata
87+
```bash
88+
uv run explore extract data.h5ad -f development_stage -f age -f donor_id
89+
```
90+
91+
### Extract cell type annotations
92+
```bash
93+
uv run explore extract data.h5ad \
94+
-f cell_type \
95+
-f cell_type_ontology_term_id \
96+
-f tissue \
97+
-f organ
98+
```
99+
100+
### Batch extract from multiple datasets
101+
```bash
102+
uv run explore batch ./datasets ./metadata \
103+
-f development_stage \
104+
-f cell_type \
105+
-f tissue \
106+
-f disease \
107+
--skip-existing \
108+
--log-dir ./logs
109+
```
110+
111+
### Check what fields are available
112+
To see what fields are available in your h5ad file, you can use Python:
113+
```python
114+
import anndata as ad
115+
adata = ad.read_h5ad('your_file.h5ad', backed='r')
116+
print(adata.obs.columns.tolist())
117+
adata.file.close()
118+
```
119+
120+
## Logging
121+
122+
The tool uses Eliot for structured logging. When you specify `--log-dir`, it creates:
123+
- `extract.json` / `batch_extract.json`: Machine-readable JSON logs
124+
- `extract.log` / `batch_extract.log`: Human-readable rendered logs
125+
126+
Without `--log-dir`, logs are written to stdout and a JSON file in the current directory.
127+
128+
## Error Handling
129+
130+
- If a requested field doesn't exist, the tool will warn you and extract only the available fields
131+
- In batch mode, errors in one file don't stop processing of other files
132+
- Failed files are reported in the final summary
133+
134+

0 commit comments

Comments
 (0)