Skip to content

Commit 20e1be8

Browse files
TKassisTKassis
authored
Merge pull request #80 from fedorov/update-idc-1.4.0
Update imaging-data-commons skill to v1.4.0
2 parents d965eb8 + 0c4a7ea commit 20e1be8

2 files changed

Lines changed: 155 additions & 6 deletions

File tree

‎scientific-skills/imaging-data-commons/SKILL.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -3,9 +3,9 @@ name: imaging-data-commons
33
description: Query and download public cancer imaging data from NCI Imaging Data Commons using idc-index. Use for accessing large-scale radiology (CT, MR, PET) and pathology datasets for AI training or research. No authentication required. Query by metadata, visualize in browser, check licenses.
44
license: This skill is provided under the MIT License. IDC data itself has individual licensing (mostly CC-BY, some CC-NC) that must be respected when using the data.
55
metadata:
6-
version: 1.3.1
6+
version: 1.4.0
77
skill-author: Andrey Fedorov, @fedorov
8-
idc-index: "0.11.9"
8+
idc-index: "0.11.10"
99
idc-data-version: "v23"
1010
repository: https://github.com/ImagingDataCommons/idc-claude-skill
1111
---
@@ -25,7 +25,7 @@ Use the `idc-index` Python package to query and download public cancer imaging d
2525
```python
2626
import idc_index
2727

28-
REQUIRED_VERSION = "0.11.9" # Must match metadata.idc-index in this file
28+
REQUIRED_VERSION = "0.11.10" # Must match metadata.idc-index in this file
2929
installed = idc_index.__version__
3030

3131
if installed < REQUIRED_VERSION:
@@ -229,7 +229,7 @@ print(client.get_idc_version()) # Should return "v23"
229229
```
230230
If you see an older version, upgrade with: `pip install --upgrade idc-index`
231231

232-
**Tested with:** idc-index 0.11.9 (IDC data version v23)
232+
**Tested with:** idc-index 0.11.10 (IDC data version v23)
233233

234234
**Optional (for data analysis):**
235235
```bash

‎scientific-skills/imaging-data-commons/references/digital_pathology_guide.md‎

Lines changed: 151 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# Digital Pathology Guide for IDC
22

3-
**Tested with:** IDC data version v23, idc-index 0.11.9
3+
**Tested with:** IDC data version v23, idc-index 0.11.10
44

55
For general IDC queries and downloads, use `idc-index` (see main SKILL.md). This guide covers slide microscopy (SM) imaging, microscopy bulk simple annotations (ANN), and segmentations (SEG) in the context of digital pathology in IDC.
66

@@ -10,7 +10,7 @@ Five specialized index tables provide curated metadata without needing BigQuery:
1010

1111
| Table | Row Granularity | Description |
1212
|-------|-----------------|-------------|
13-
| `sm_index` | 1 row = 1 SM series | Slide Microscopy series metadata: lens power, pixel spacing, image dimensions |
13+
| `sm_index` | 1 row = 1 SM series | Slide Microscopy series metadata: container/slide ID, tissue type, anatomic structure, diagnosis, lens power, pixel spacing, image dimensions |
1414
| `sm_instance_index` | 1 row = 1 SM instance | Instance-level (SOPInstanceUID) metadata for individual slide images |
1515
| `seg_index` | 1 row = 1 SEG series | DICOM Segmentation metadata: algorithm, segment count, reference to source series. Used for both radiology and pathology — filter by source Modality to find pathology-specific segmentations |
1616
| `ann_index` | 1 row = 1 ANN series | Microscopy Bulk Simple Annotations series metadata; includes `referenced_SeriesInstanceUID` linking to the annotated slide |
@@ -57,6 +57,109 @@ client.sql_query("""
5757
""")
5858
```
5959

60+
### Filter by specimen preparation
61+
62+
The `sm_index` includes staining, embedding, and fixative metadata. These columns are **arrays** (e.g., `[hematoxylin stain, water soluble eosin stain]` for H&E) — use `array_to_string()` with `LIKE` or `list_contains()` to filter.
63+
64+
```python
65+
# Find H&E-stained slides in a collection
66+
client.fetch_index("sm_index")
67+
client.sql_query("""
68+
SELECT
69+
i.PatientID,
70+
s.staining_usingSubstance_CodeMeaning as staining,
71+
s.embeddingMedium_CodeMeaning as embedding,
72+
s.tissueFixative_CodeMeaning as fixative
73+
FROM sm_index s
74+
JOIN index i ON s.SeriesInstanceUID = i.SeriesInstanceUID
75+
WHERE i.collection_id = 'tcga_brca'
76+
AND array_to_string(s.staining_usingSubstance_CodeMeaning, ', ') LIKE '%hematoxylin%'
77+
LIMIT 10
78+
""")
79+
```
80+
81+
```python
82+
# Compare FFPE vs frozen slides across collections
83+
client.sql_query("""
84+
SELECT
85+
i.collection_id,
86+
s.embeddingMedium_CodeMeaning as embedding,
87+
COUNT(*) as slide_count
88+
FROM sm_index s
89+
JOIN index i ON s.SeriesInstanceUID = i.SeriesInstanceUID
90+
GROUP BY i.collection_id, embedding
91+
ORDER BY i.collection_id, slide_count DESC
92+
""")
93+
```
94+
95+
## Identifying Tumor vs Normal Slides
96+
97+
The `sm_index` table provides two ways to identify tissue type:
98+
99+
| Column | Use Case |
100+
|--------|----------|
101+
| `primaryAnatomicStructureModifier_CodeMeaning` | Structured tissue type from DICOM specimen metadata (e.g., `Neoplasm, Primary`, `Normal`, `Tumor`, `Neoplasm, Metastatic`). Works across all collections with SM data. |
102+
| `ContainerIdentifier` | Slide/container identifier. For TCGA collections, contains the [TCGA barcode](https://docs.gdc.cancer.gov/Encyclopedia/pages/TCGA_Barcode/) where the [sample type code](https://gdc.cancer.gov/resources-tcga-users/tcga-code-tables/sample-type-codes) (positions 14-15) encodes tissue origin: `01`-`09` = tumor, `10`-`19` = normal. |
103+
104+
### Using structured tissue type metadata
105+
106+
```python
107+
from idc_index import IDCClient
108+
client = IDCClient()
109+
client.fetch_index("sm_index")
110+
111+
# Discover tissue type values across all SM data
112+
client.sql_query("""
113+
SELECT
114+
s.primaryAnatomicStructureModifier_CodeMeaning as tissue_type,
115+
COUNT(*) as slide_count
116+
FROM sm_index s
117+
WHERE s.primaryAnatomicStructureModifier_CodeMeaning IS NOT NULL
118+
GROUP BY tissue_type
119+
ORDER BY slide_count DESC
120+
""")
121+
```
122+
123+
#### Example: Tumor vs normal slides in TCGA-BRCA
124+
125+
```python
126+
# Tissue type breakdown for TCGA-BRCA
127+
client.sql_query("""
128+
SELECT
129+
s.primaryAnatomicStructureModifier_CodeMeaning as tissue_type,
130+
COUNT(*) as slide_count,
131+
COUNT(DISTINCT i.PatientID) as patient_count
132+
FROM sm_index s
133+
JOIN index i ON s.SeriesInstanceUID = i.SeriesInstanceUID
134+
WHERE i.collection_id = 'tcga_brca'
135+
GROUP BY tissue_type
136+
ORDER BY slide_count DESC
137+
""")
138+
# Returns: Neoplasm, Primary (2704 slides), Normal (399 slides)
139+
```
140+
141+
### Using TCGA barcode (TCGA collections only)
142+
143+
For TCGA collections, `ContainerIdentifier` contains the slide barcode (e.g., `TCGA-E9-A3X8-01A-03-TSC`). Extract the sample type code to classify tissue:
144+
145+
```python
146+
# Parse sample type from TCGA barcode
147+
client.sql_query("""
148+
SELECT
149+
SUBSTRING(SPLIT_PART(s.ContainerIdentifier, '-', 4), 1, 2) as sample_type_code,
150+
s.primaryAnatomicStructureModifier_CodeMeaning as tissue_type,
151+
COUNT(*) as slide_count
152+
FROM sm_index s
153+
JOIN index i ON s.SeriesInstanceUID = i.SeriesInstanceUID
154+
WHERE i.collection_id = 'tcga_brca'
155+
GROUP BY sample_type_code, tissue_type
156+
ORDER BY sample_type_code
157+
""")
158+
# Returns: 01 → Neoplasm, Primary (2704), 06 → None (8), 11 → Normal (399)
159+
```
160+
161+
The barcode approach catches cases where structured metadata is NULL (e.g., `06` = Metastatic slides have `primaryAnatomicStructureModifier_CodeMeaning` = NULL in TCGA-BRCA).
162+
60163
## Annotation Queries (ANN)
61164

62165
DICOM Microscopy Bulk Simple Annotations (Modality = 'ANN') are annotations **on** slide microscopy images. They appear in `ann_index` (series-level) and `ann_group_index` (group-level detail). Each ANN series references the slide it annotates via `referenced_SeriesInstanceUID`.
@@ -134,6 +237,52 @@ client.sql_query("""
134237
""")
135238
```
136239

240+
## Finding Pre-Computed Analysis Results
241+
242+
IDC hosts derived datasets (nuclei segmentations, TIL maps, AI annotations) identified by `analysis_result_id` in the main `index` table. Use `analysis_results_index` to discover what's available for pathology.
243+
244+
```python
245+
from idc_index import IDCClient
246+
client = IDCClient()
247+
client.fetch_index("analysis_results_index")
248+
249+
# Find analysis results that include pathology annotations or segmentations
250+
client.sql_query("""
251+
SELECT
252+
ar.analysis_result_id,
253+
ar.analysis_result_title,
254+
ar.Modalities,
255+
ar.Subjects,
256+
ar.Collections
257+
FROM analysis_results_index ar
258+
WHERE ar.Modalities LIKE '%ANN%' OR ar.Modalities LIKE '%SEG%'
259+
ORDER BY ar.Subjects DESC
260+
""")
261+
```
262+
263+
### Find analysis results for a specific slide
264+
265+
```python
266+
# Find all derived data (annotations, segmentations) for TCGA-BRCA slides
267+
client.fetch_index("ann_index")
268+
client.sql_query("""
269+
SELECT
270+
i.analysis_result_id,
271+
i.PatientID,
272+
a.referenced_SeriesInstanceUID as source_slide,
273+
g.AnnotationGroupLabel,
274+
g.NumberOfAnnotations,
275+
g.AlgorithmName
276+
FROM ann_group_index g
277+
JOIN ann_index a ON g.SeriesInstanceUID = a.SeriesInstanceUID
278+
JOIN index i ON a.SeriesInstanceUID = i.SeriesInstanceUID
279+
WHERE i.collection_id = 'tcga_brca'
280+
LIMIT 10
281+
""")
282+
```
283+
284+
Annotation objects can also contain per-annotation **measurements** (e.g., nucleus area, eccentricity) stored within the DICOM file. These are not in the index tables — extract them after download using [highdicom](https://github.com/ImagingDataCommons/highdicom) (`ann.get_annotation_groups()`, `group.get_measurements()`). See the [microscopy_dicom_ann_intro](https://github.com/ImagingDataCommons/IDC-Tutorials/blob/master/notebooks/pathomics/microscopy_dicom_ann_intro.ipynb) tutorial for a worked example including spatial analysis and cellularity computation.
285+
137286
## Filter by AnnotationGroupLabel
138287

139288
`AnnotationGroupLabel` is the most direct column for finding annotation groups by name or semantic content. Use `LIKE` with wildcards for text search.

0 commit comments

Comments
 (0)