Skip to content

add noAddress: true to runtime blocks for airgap-safe tasks#653

Closed
dpark01 wants to merge 1 commit into
masterfrom
dp-noAddress-airgap
Closed

add noAddress: true to runtime blocks for airgap-safe tasks#653
dpark01 wants to merge 1 commit into
masterfrom
dp-noAddress-airgap

Conversation

@dpark01

@dpark01 dpark01 commented Apr 1, 2026

Copy link
Copy Markdown
Member

Summary

  • Adds noAddress: true to the runtime block of all WDL tasks that do not require internet access
  • This Cromwell/Terra runtime attribute indicates the task VM does not need a public IP address, improving security and reducing costs
  • Tasks that require network access (NCBI downloads/uploads, Terra API calls, Nextstrain dataset fetches, GISAID uploads, etc.) are left unchanged

Test plan

  • Verify miniwdl check passes on all edited files
  • Verify tasks that need network do NOT have noAddress: true
  • Run CI validation (womtool + miniwdl check)

🤖 Generated with Claude Code

Adds noAddress: true to the runtime block of all WDL tasks that do not
require internet access. Tasks that need network (NCBI downloads/uploads,
Terra API calls, Nextstrain dataset fetches, GISAID uploads, pangolin,
snpEff, kraken2 db builds) are left unchanged. This Cromwell/Terra
runtime attribute prevents allocation of a public IP address to task VMs,
improving security posture and reducing costs.
Copilot AI review requested due to automatic review settings April 1, 2026 18:21

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the Terra/Cromwell runtime blocks across the pipeline WDL task library to mark airgap-safe tasks as not needing a public IP (noAddress: true), improving security posture and potentially reducing costs.

Changes:

  • Added noAddress: true to runtime blocks for many tasks that do not perform network operations.
  • Left tasks that explicitly download data (e.g., URL/GitHub fetches in tasks_utils.wdl / tasks_nextstrain.wdl) without noAddress: true.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated no comments.

Show a summary per file
File Description
pipes/WDL/tasks/tasks_utils.wdl Adds noAddress: true to a large set of local utility tasks’ runtime blocks.
pipes/WDL/tasks/tasks_taxon_filter.wdl Marks taxon filtering tasks as not requiring a public IP.
pipes/WDL/tasks/tasks_sarscov2.wdl Marks report/ETL-style SARS-CoV-2 tasks as not requiring a public IP.
pipes/WDL/tasks/tasks_reports.wdl Marks report generation/aggregation tasks as not requiring a public IP.
pipes/WDL/tasks/tasks_read_utils.wdl Marks read/BAM utility tasks as not requiring a public IP.
pipes/WDL/tasks/tasks_nextstrain.wdl Marks multiple local Nextstrain processing steps as not requiring a public IP.
pipes/WDL/tasks/tasks_ncbi.wdl Marks various NCBI prep/formatting tasks as not requiring a public IP.
pipes/WDL/tasks/tasks_ncbi_tools.wdl Marks local XML/TSV conversion and grouping tasks as not requiring a public IP.
pipes/WDL/tasks/tasks_metagenomics.wdl Marks metagenomics classification/reporting tasks as not requiring a public IP.
pipes/WDL/tasks/tasks_megablast.wdl Marks megablast-related compute tasks as not requiring a public IP.
pipes/WDL/tasks/tasks_intrahost.wdl Marks intrahost analysis tasks as not requiring a public IP.
pipes/WDL/tasks/tasks_interhost.wdl Marks interhost analysis tasks as not requiring a public IP.
pipes/WDL/tasks/tasks_demux.wdl Marks demux and demux-metadata merge tasks as not requiring a public IP.
pipes/WDL/tasks/tasks_assembly.wdl Marks assembly and assembly-adjacent tasks as not requiring a public IP.
pipes/WDL/tasks/tasks_16S_amplicon.wdl Marks 16S amplicon processing tasks as not requiring a public IP.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@dpark01

dpark01 commented Apr 1, 2026

Copy link
Copy Markdown
Member Author

Closing this PR for now. Testing in Terra (workspace broad-viral-surveillance/measles-usa, submission efd2b746-9ac8-4b0b-9ee9-82775c79eb38) revealed that noAddress: true prevents VMs from pulling docker images hosted on non-Google registries (quay.io, Docker Hub, etc.). Private Google Access only provides routes to Google APIs -- reaching public endpoints like quay.io requires either Cloud NAT or a pull-through cache on Artifact Registry.

Since nearly all of our workflow docker images are hosted on quay.io, this change is not viable until Terra implements docker pull-through registry caches (Artifact Registry remote repositories) for external registries. Once that is in place, noAddress: true should work seamlessly and we can revisit this.

@dpark01 dpark01 closed this Apr 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants