add noAddress: true to runtime blocks for airgap-safe tasks#653
Conversation
Adds noAddress: true to the runtime block of all WDL tasks that do not require internet access. Tasks that need network (NCBI downloads/uploads, Terra API calls, Nextstrain dataset fetches, GISAID uploads, pangolin, snpEff, kraken2 db builds) are left unchanged. This Cromwell/Terra runtime attribute prevents allocation of a public IP address to task VMs, improving security posture and reducing costs.
There was a problem hiding this comment.
Pull request overview
This PR updates the Terra/Cromwell runtime blocks across the pipeline WDL task library to mark airgap-safe tasks as not needing a public IP (noAddress: true), improving security posture and potentially reducing costs.
Changes:
- Added
noAddress: truetoruntimeblocks for many tasks that do not perform network operations. - Left tasks that explicitly download data (e.g., URL/GitHub fetches in
tasks_utils.wdl/tasks_nextstrain.wdl) withoutnoAddress: true.
Reviewed changes
Copilot reviewed 15 out of 15 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| pipes/WDL/tasks/tasks_utils.wdl | Adds noAddress: true to a large set of local utility tasks’ runtime blocks. |
| pipes/WDL/tasks/tasks_taxon_filter.wdl | Marks taxon filtering tasks as not requiring a public IP. |
| pipes/WDL/tasks/tasks_sarscov2.wdl | Marks report/ETL-style SARS-CoV-2 tasks as not requiring a public IP. |
| pipes/WDL/tasks/tasks_reports.wdl | Marks report generation/aggregation tasks as not requiring a public IP. |
| pipes/WDL/tasks/tasks_read_utils.wdl | Marks read/BAM utility tasks as not requiring a public IP. |
| pipes/WDL/tasks/tasks_nextstrain.wdl | Marks multiple local Nextstrain processing steps as not requiring a public IP. |
| pipes/WDL/tasks/tasks_ncbi.wdl | Marks various NCBI prep/formatting tasks as not requiring a public IP. |
| pipes/WDL/tasks/tasks_ncbi_tools.wdl | Marks local XML/TSV conversion and grouping tasks as not requiring a public IP. |
| pipes/WDL/tasks/tasks_metagenomics.wdl | Marks metagenomics classification/reporting tasks as not requiring a public IP. |
| pipes/WDL/tasks/tasks_megablast.wdl | Marks megablast-related compute tasks as not requiring a public IP. |
| pipes/WDL/tasks/tasks_intrahost.wdl | Marks intrahost analysis tasks as not requiring a public IP. |
| pipes/WDL/tasks/tasks_interhost.wdl | Marks interhost analysis tasks as not requiring a public IP. |
| pipes/WDL/tasks/tasks_demux.wdl | Marks demux and demux-metadata merge tasks as not requiring a public IP. |
| pipes/WDL/tasks/tasks_assembly.wdl | Marks assembly and assembly-adjacent tasks as not requiring a public IP. |
| pipes/WDL/tasks/tasks_16S_amplicon.wdl | Marks 16S amplicon processing tasks as not requiring a public IP. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
Closing this PR for now. Testing in Terra (workspace broad-viral-surveillance/measles-usa, submission efd2b746-9ac8-4b0b-9ee9-82775c79eb38) revealed that noAddress: true prevents VMs from pulling docker images hosted on non-Google registries (quay.io, Docker Hub, etc.). Private Google Access only provides routes to Google APIs -- reaching public endpoints like quay.io requires either Cloud NAT or a pull-through cache on Artifact Registry. Since nearly all of our workflow docker images are hosted on quay.io, this change is not viable until Terra implements docker pull-through registry caches (Artifact Registry remote repositories) for external registries. Once that is in place, noAddress: true should work seamlessly and we can revisit this. |
Summary
noAddress: trueto theruntimeblock of all WDL tasks that do not require internet accessTest plan
miniwdl checkpasses on all edited filesnoAddress: true🤖 Generated with Claude Code