Adjust preemptible settings for assembly/metagenomics tasks#654
Merged
Conversation
…vial tasks Preemptible VMs no longer fall back to non-preemptible after exhausting attempts, so low values (1-2) just cause failures. Bump to 3 for expensive tasks (align_reads, kraken2, filter_bam_to_taxa, select_references) and remove preemptible entirely from cheap/fast tasks (run_discordance, report_primary_kraken_taxa, plot_coverage, tar_extract) where the cost savings are negligible.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adjusts WDL task runtime { preemptible: ... } settings to reduce hard failures due to VM preemption behavior changes on Terra/Cromwell, while preserving cost savings for more expensive tasks.
Changes:
- Increased
preemptibleattempts to3for expensive compute-heavy tasks (align_reads,kraken2,filter_bam_to_taxa,select_references). - Removed
preemptiblefrom several cheap/fast tasks to avoid avoidable preemption-related failures (tar_extract,plot_coverage,report_primary_kraken_taxa,run_discordance).
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| pipes/WDL/tasks/tasks_utils.wdl | Removes preemptible from tar_extract runtime to avoid preemption failures on a fast utility task. |
| pipes/WDL/tasks/tasks_reports.wdl | Removes preemptible from plot_coverage runtime to reduce failure risk on a reporting task. |
| pipes/WDL/tasks/tasks_metagenomics.wdl | Increases preemptible for kraken2 and filter_bam_to_taxa; removes it from report_primary_kraken_taxa. |
| pipes/WDL/tasks/tasks_assembly.wdl | Increases preemptible for select_references and align_reads; removes it from run_discordance. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Motivation
Preemptible VMs on Terra/Cromwell no longer automatically fall back to non-preemptible VMs after exhausting attempts -- they just fail. With values of 1-2, we were seeing hard failures on preemption. We cannot use maxRetries as a fallback mechanism either, since it disables Cromwell call caching entirely (see #641).
For expensive tasks, bumping to 3 gives more chances to land a preemptible VM while keeping cost savings. For trivial tasks, the cost savings from preemptible are negligible so we remove it to avoid unnecessary failure risk.
Demux tasks were already set to preemptible: 0 and are unchanged.
Test plan
Generated with Claude Code