-
Notifications
You must be signed in to change notification settings - Fork 12
Improvements to be done
This is a review of the imaging-data-commons Claude skill — a domain-specific skill for
querying and downloading public cancer imaging data from the NCI Imaging Data Commons. The goal is
to assess SKILL.md against best practices for skill authoring and LLM-efficient organization:
progressive disclosure, size/token budget, duplication/drift, and instructional clarity.
This revision refreshes an earlier review that was written against a 1,234-line version of
SKILL.md. Since then the file has been cut to 750 lines, most of the earlier extraction plan
has been carried out, the missing clinical_data_guide.md has been added, and a Quick Navigation
section with load-on-demand triggers has been introduced. The findings below reflect the current
state (SKILL.md at 750 lines, idc-index 0.12.3, IDC data v24).
Scope note: this pass focused on SKILL.md only. Reference files are inventoried but were not re-audited in depth.
The following items from the earlier review are now resolved and are recorded here so the history isn't lost:
| Prior finding | Status | Evidence |
|---|---|---|
| SKILL.md ~1,234 lines, far over budget | Improved — now 750 lines (~9.1K tokens) | Phase 1 extraction largely done |
clinical_data_guide.md missing |
Resolved — file present (329 lines) | references/clinical_data_guide.md |
| No internal navigation / TOC | Resolved — Quick Navigation added | SKILL.md:73–95 |
| Reference listings lack when-to-load triggers | Resolved — trigger column present | SKILL.md:84–95 |
| Frontmatter sparse |
Improved — versioned metadata block added |
version, idc-index, idc-data-version, repository
|
| 5 reference files | Expanded — now 10 files | see inventory below |
The pattern is established and working; the remaining work is finishing the size trim and cleaning up duplication/consistency that has crept in.
Finding: SKILL.md is 750 lines / ~36.5 KB / ~9.1K tokens, with ~39% of lines inside code blocks (17 blocks: 15 Python, 2 Bash). Anthropic's skill-authoring guidance targets under ~500 lines for the always-loaded SKILL.md, deferring detail to on-demand references. 750 is a large improvement over 1,234 but still ~50% over the guideline, and every line loads on each invocation.
Concrete extractions toward ~550 lines:
| Content | Current lines | Move to | Risk |
|---|---|---|---|
| "What's new in IDC vX" version-tracking block (3 full SQL queries) | 359–404 (~45) |
sql_patterns.md + 2-line pointer |
Low — specialized workflow |
Download variants (entire collection, 3-step series, download_dicom_series alt, GCS source, simplified/flat dirTemplate) |
421–489 (~6 blocks) | keep signature table + 1 canonical example; rest → use_cases.md
|
Low — keep the anti-footgun table inline |
| Inline Clinical Data Access block | 183–196 | trim to pointer (duplicates clinical_data_guide.md) |
Low |
Keep the primary API patterns (query, download signature table, visualize, license, citations) inline — moving those risks Claude answering IDC-specific APIs from general knowledge.
Finding: The indices_overview column-discovery loop appears twice, nearly verbatim — at
SKILL.md:127–139 (Index Tables) and again at SKILL.md:694–707 (Troubleshooting). Two copies of the
same ~6-line loop means two edit points and extra always-loaded tokens.
Fix: Keep one canonical copy; have the Troubleshooting entry point to it rather than restating the loop.
Finding: At SKILL.md:187–194 the example runs
SELECT DISTINCT table_name, column_label FROM clinical_index, then calls
client.get_clinical_table("table_name"). But get_clinical_table() expects the
short_table_name value, not table_name (the full BigQuery reference), per the column table in
clinical_data_guide.md. "table_name" reads as a placeholder here, but an LLM that pulled
table_name values from the preceding query will pass the wrong column and the call will fail.
Fix: Select short_table_name in the discovery query, or rename the placeholder to
"<short_table_name>" so the distinction is explicit.
Finding: Several facts live in both SKILL.md and a reference file and will diverge on the next schema change:
- Inline Clinical Data Access (183–196) ↔
clinical_data_guide.md - "Available metadata fields" list (349–357) ↔ Key Columns table in
index_tables_guide.md(90–115) - Cloud storage / DICOMweb summaries ↔ their dedicated guides
Fix: Give each fact one home plus a pointer. Brief inline summaries are fine; full restatements are the drift hazard.
Finding: Quick Navigation uses bare filenames (use_cases.md, sql_patterns.md) while the body
always uses references/…. use_cases.md is referenced only in Quick Navigation and never
with a full path anywhere, so an instruction to "load use_cases.md" leaves the path ambiguous.
Fix: Normalize to references/<file>.md everywhere.
Finding: Core Capabilities is numbered 1–7, but two unnumbered H3s are interleaved: "Command-Line Download" (between 3 and 4) and "Generating Citations for Attribution" (between 5 and 6). The numbering implies a sequence/grouping that doesn't hold.
Fix: Fold the unnumbered sections under their neighbors, or drop the numbering entirely.
Finding: v24 is pinned in frontmatter (idc-data-version, line 9), in prose at SKILL.md:21,
and again at SKILL.md:656 — even though the skill correctly instructs Claude to verify at runtime via
get_idc_version(). Three edit points per release invites inconsistency.
Fix: State the concrete version once (frontmatter) and refer to "the pinned version" elsewhere; lean on the runtime check for the actual value.
Finding: Related to #1. The Data Discovery and Download sections carry several near-duplicate
examples (e.g., multiple sql_query() calls differing only in WHERE clause; ~6 download variants).
Claude can generalize from one well-chosen example per pattern.
Fix: Keep 1–2 representative examples per pattern inline; move the variation set to references.
Finding: The skill now has a solid metadata block, but two optional fields remain unused:
-
allowed-tools— could scope execution toBash(pip install + Python), reducing unintended tool use. -
argument-hint— could suggest expected input, e.g.[query or task description].
Low priority; consider once the higher-impact items land.
-
description(372 chars): third-person, says what and when, keyword-rich, and explicitly fires "even when the user doesn't mention IDC." This is the primary invocation trigger and it's strong — don't shorten it. - Quick Navigation load-on-demand table (84–95): textbook progressive disclosure.
-
Download method-signature comparison table (414–417) and the "
from_selectiondoes NOT take a DataFrame" warning (419): exactly the API-footgun content that belongs inline. Keep it. - Table-heavy layout: scannable and LLM-friendly.
Ten reference files, ~3,590 lines total, loaded lazily. Well-organized by topic; sizes reasonable for on-demand docs.
| File | Lines | Topic |
|---|---|---|
bigquery_guide.md |
858 | Advanced BigQuery queries |
digital_pathology_guide.md |
403 | Pathology (SM, ANN, SEG) |
dicomweb_guide.md |
399 | DICOMweb API access |
sql_patterns.md |
365 | Quick-reference SQL patterns |
cloud_storage_guide.md |
333 | S3/GCS bucket access |
clinical_data_guide.md |
329 | Clinical/tabular data |
use_cases.md |
278 | End-to-end workflows |
cli_guide.md |
272 | Command-line tools |
parquet_access_guide.md |
199 | Direct Parquet queries |
index_tables_guide.md |
154 | Table schemas, join columns |
| Priority | Finding | Type | Effort |
|---|---|---|---|
| 1 | #3 Clinical get_clinical_table() uses wrong column value |
Correctness footgun | Low |
| 2 | #1 Trim SKILL.md 750 → ~550 via extraction | LLM efficiency | Med |
| 3 | #2 Deduplicate indices_overview discovery loop |
Drift / tokens | Low |
| 4 | #4 Reconcile SKILL.md content that overlaps reference files | Drift | Low–Med |
| 5 | #5 Normalize references/ path style |
Consistency | Low |
| 6 | #6 Fix Core Capabilities numbering | Consistency | Low |
| 7 | #7 Collapse hardcoded v24 to a single source |
Maintenance | Low |
| 8 | #8 Consolidate redundant examples | LLM efficiency | Low–Med |
| 9 | #9 Consider allowed-tools / argument-hint frontmatter |
Hardening | Low |
Suggested order: start with #3 (a real copy-paste hazard), then the mechanical consistency fixes (#2, #5, #6, #7), then the focused size/example trim (#1, #4, #8) as one pass since it touches the most lines.
| Metric | Prior review | Current | Target |
|---|---|---|---|
| SKILL.md lines | 1,234 | 750 | ~500–550 |
| SKILL.md tokens (approx) | ~15–16K | ~9.1K | ~6–7K |
| % of lines in code blocks | — | ~39% | ~30% |
| Reference files | 5 (1,804 lines) | 10 (~3,590 lines) | 10 |
| Code blocks in SKILL.md | ~60 | 17 | ~12–15 |
| Missing references | 1 | 0 | 0 |
| Frontmatter fields | 4 | 4 (+4 metadata sub-fields) | +allowed-tools (optional) |