Commit 5a4e3f6
Refactor HiFi QC pipeline + add Dockstore manifest (#8)
* refactor(wdl): split hifi_qc_pipeline_3 into per-category modules
Reorganize monolithic wdl/hifi_qc_pipeline_3.wdl into LRP-style layout
(structs/, tasks/<Category>/, pipelines/<Tech>/<Category>/) per
docs/WDL_STYLE_RULES.md. Every task now carries the canonical
boilerplate: meta + outputs sub-block, tool citation where applicable,
parameter_meta for every input, extra_args passthrough, 5x input-based
disk sizing, resource-detection preamble, RuntimeAttr override hook,
and column-aligned runtime block. Workflow calls aliased t_01..t_07 in
lexical invocation order.
Also fix two shellcheck lints surfaced by miniwdl --strict (SC2002
useless-cat in the preamble, SC2086 unquoted \${NUM_CPUS}); update the
preamble in WDL_STYLE_RULES.md to match.
miniwdl check --strict wdl/pipelines/PacBio/QC/HifiPipeline.wdl passes
clean.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* refactor(wdl): take sample-sheet columns as direct workflow inputs
Replace the single File input_tsv with five parallel Array[String]
inputs (bams, genera, species, strains, barcodes). The caller now
hands every column in directly; the workflow no longer reads or
parses a TSV.
Consequences:
- Drop ParseSampleSheet task from tasks/Utility/SampleSheetUtils.wdl
(no longer used). MergeSampleStats stays.
- Renumber workflow call aliases from t_01..t_07 down to t_01..t_06
in lexical invocation order:
t_01_BatchGetTaxIdAndGenomeSize
t_02_BamToFastqAndStats (scatter)
t_03_HifiSeqkitStats
t_04_HifiKraken2
t_05_HifiReadStats
t_06_MergeSampleStats
- Barcodes were previously derived from BAM basenames inside
ParseSampleSheet; they are now explicit inputs.
miniwdl check --strict wdl/pipelines/PacBio/QC/HifiPipeline.wdl passes
clean.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* refactor(wdl): collapse HifiPipeline to a single-sample workflow
Strip the multi-sample scatter and replace its inputs (parallel arrays
of bams/genera/species/strains/barcodes) with scalar inputs for one
sample: input_bam, sample_name, genus, species.
- Drop tasks/Utility/SampleSheetUtils.wdl entirely; MergeSampleStats
no longer has a caller.
- Replace BatchGetTaxIdAndGenomeSize with GetTaxIdAndGenomeSize in
tasks/Utility/Taxonomy.wdl: scalar (genus, species) inputs, single
TSV plus tax_id (Int) and expected_genome_size (String) outputs.
- HifiPipeline now has 5 calls, no scatter, aliased t_01..t_05 in
lexical order:
t_01_GetTaxIdAndGenomeSize
t_02_BamToFastqAndStats
t_03_HifiSeqkitStats
t_04_HifiKraken2
t_05_HifiReadStats
- Workflow outputs renamed per_sample_* -> sample_* and now File
(not Array[File]); added sample_taxid_tsv and sample_bam_stats
pass-throughs.
- strain column dropped; its only consumer was the (removed) merge
task.
miniwdl check --strict wdl/pipelines/PacBio/QC/HifiPipeline.wdl passes
clean.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* refactor(wdl): expose per-metric scalars instead of stats TSV files
Replace the bundled-TSV outputs (sample_stats_tsv, sample_bam_stats,
sample_taxid_tsv, sample_stats_txt) with one workflow output per
metric, typed Int / Float / String as appropriate.
HifiReadStats:
- Add a small bash splitter after the awk that fans the single-row
hifi_read_stats.tsv out into stat.<key>.txt files (one per column),
then bind each file in the output block via read_int/read_float/
read_string.
- Drop the hifi_read_stats_tsv and hifi_read_stats_txt outputs;
the report (formatted ASCII table) is still exposed.
- Add the 27 new outputs to meta.outputs; update the stale
parameter_meta reference (BatchGetTaxIdAndGenomeSize -> the
new GetTaxIdAndGenomeSize).
HifiPipeline:
- Drop sample_stats_tsv, sample_stats_txt, sample_bam_stats,
sample_taxid_tsv.
- Keep sample_fastq, sample_kraken_report, sample_stats_report
as file outputs.
- Pass through tax_id (Int) and expected_genome_size (String) from
t_01, and the 27 metric scalars from t_05.
- Update meta.outputs to match.
miniwdl check --strict wdl/pipelines/PacBio/QC/HifiPipeline.wdl
passes clean.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* refactor(wdl): decouple BamToFastqAndStats from HifiReadStats
HifiReadStats no longer consumes bam_stats_tsv — the 3 bam-derived
metrics (mean_read_accuracy, mean_qual_score, mean_passes) were pure
passthrough with no computation. Drop the input, the awk column
lookups, the report/tsv/txt emissions, and the 3 output bindings.
HifiPipeline now exposes those scalars directly from t_02_BamToFastqAndStats
under the task's native names (mean_read_accuracy, mean_qual_score,
mean_passes), dropping the cosmetic HifiReadStats renames
(mean_read_base_qual, mean_num_passes).
BamToFastqAndStats drops the bam_stats_tsv File output — no remaining
consumer; the per-scalar txt files driving the typed outputs are kept.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* refactor(wdl): expose scalars from originating tasks; shrink HifiReadStats
HifiSeqkitStats and HifiKraken2 now emit per-metric scalar outputs
(read counts, length quantiles, Q20/Q30, GC%, bases over thresholds,
kraken classification percentages, top genus/species). HifiReadStats
no longer parses upstream TSVs — it takes the scalars directly,
computes the 3 coverage estimates (the only derived metric), and
writes the boxed ASCII report.
Workflow output names are unchanged; bindings now point at the
originating tasks (t_03_HifiSeqkitStats for length/quality/GC metrics,
t_04_HifiKraken2 for classification metrics, t_05_HifiReadStats for
the coverage estimates and the human-readable report). Dead avg_len
lookup is dropped.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* refactor(wdl): rename HifiReadStats -> HifiCoverageAndReport
Name now reflects what the task actually does: estimate coverage
(overall + reads >10kb / >20kb) and render the per-sample boxed
ASCII report. Artifact filename and output binding follow:
hifi_read_stats.report.txt -> hifi_stats.report.txt,
hifi_read_stats_report -> hifi_stats_report. Workflow call alias
updated to t_05_HifiCoverageAndReport.
Workflow output names unchanged.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* refactor(wdl): rename HifiCoverageAndReport -> CreateHifiQCReport
Task name and workflow call alias updated; artifact filename and
output binding (hifi_stats_report) unchanged. Workflow output names
unchanged.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* feat(wdl): port LRP SummarizePBI into HVP-LR HifiPipeline
Adds wdl/tasks/Utility/PacBioPbi.wdl with task SummarizeHifiPbi, a
verbatim port of broadinstitute/long-read-pipelines' SummarizePBI
(wdl/tasks/Utility/PBUtils.wdl) and its compute_pbi_stats.py
(docker/lr-pb/). Attribution preserved in task header; original
author Kiran V Garimella.
Rather than depending on the LRP-internal lr-pb container, the
Python script is embedded via heredoc and runs under python:3.11-slim
with numpy<2 and construct<2.10 pip-installed at task start.
HifiPipeline now takes File input_pbi and surfaces the 12 PBI-derived
metrics (pbi_reads, pbi_bases, pbi_mean_qual, pbi_median_qual,
polymerase_{mean,median,stdev,n50}, subread_{mean,median,stdev,n50})
plus the raw key/value map.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Revert "feat(wdl): port LRP SummarizePBI into HVP-LR HifiPipeline"
This reverts commit 9257209.
* chore(dockstore): add .dockstore.yml registering HifiPipeline
Initial Dockstore manifest (schema v1.2) listing the PacBio HiFi QC
workflow. Filters track main and jts_* branches plus all tags so
in-progress refactor branches publish alongside releases.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* chore(dockstore): add Terrance Shea as co-author; record ORCID + affiliation
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* refactor(wdl): rename HifiPipeline -> HifiReadQCPipeline
Workflow file and entry point renamed so the name itself signals
"QC pipeline for PacBio HiFi reads". Dockstore manifest updated to
match the new path and workflow name. No behavioral change; all
inputs, outputs, and call structure unchanged.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* chore(dockstore): add Terrance Shea ORCID
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* docs: link docs/WDL_STYLE_RULES.md from CLAUDE.md and AGENTS.md
The style rules already exist under docs/ but were not referenced
from the top-level agent guidance, so a fresh contributor or agent
had no way to discover them. Point at them from CLAUDE.md's preamble
and from AGENTS.md's WDL validation section plus the new-WDL checklist.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* docs(readme): refresh layout, paths, and current workflows
- Expand wdl/ subtree to show structs/tasks/pipelines split.
- Add docs/ and .dockstore.yml to the layout block.
- Replace the flat 'wdl/<workflow>.wdl' rows in the 'where to put new
things' table with the actual tasks/<Category>/ and
pipelines/<Tech>/<Category>/ paths used in this repo.
- Link docs/WDL_STYLE_RULES.md from 'Where to read next'.
- Add a 'Current workflows' table covering HifiReadQCPipeline and
point at the Dockstore manifest.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
---------
Co-authored-by: Jonn Smith <11667487+jonn-smith@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>1 parent 112b423 commit 5a4e3f6
11 files changed
Lines changed: 1054 additions & 839 deletions
File tree
- docs
- wdl
- pipelines/PacBio/QC
- structs
- tasks
- Preprocessing
- QC
- Utility
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
180 | 180 | | |
181 | 181 | | |
182 | 182 | | |
183 | | - | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
184 | 186 | | |
185 | 187 | | |
186 | 188 | | |
| |||
233 | 235 | | |
234 | 236 | | |
235 | 237 | | |
| 238 | + | |
236 | 239 | | |
237 | 240 | | |
238 | 241 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | | - | |
| 3 | + | |
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
10 | 10 | | |
11 | 11 | | |
12 | 12 | | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
13 | 17 | | |
14 | 18 | | |
15 | 19 | | |
| 20 | + | |
16 | 21 | | |
17 | 22 | | |
18 | 23 | | |
| |||
27 | 32 | | |
28 | 33 | | |
29 | 34 | | |
30 | | - | |
31 | | - | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
32 | 38 | | |
33 | 39 | | |
34 | 40 | | |
35 | 41 | | |
36 | 42 | | |
37 | 43 | | |
| 44 | + | |
38 | 45 | | |
39 | 46 | | |
40 | 47 | | |
| |||
47 | 54 | | |
48 | 55 | | |
49 | 56 | | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
50 | 65 | | |
51 | 66 | | |
52 | 67 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
81 | 81 | | |
82 | 82 | | |
83 | 83 | | |
84 | | - | |
| 84 | + | |
85 | 85 | | |
86 | 86 | | |
87 | 87 | | |
| |||
386 | 386 | | |
387 | 387 | | |
388 | 388 | | |
389 | | - | |
| 389 | + | |
390 | 390 | | |
391 | 391 | | |
392 | 392 | | |
| |||
0 commit comments