wip: lexicmap and logan workflows w download

[d-callan]

wip as i try to figure how to make this happen well. mostly right now it represents what id like to be able to do for brc using kmindex/ logan as a guinea pig

FOR CONTRIBUTOR:

• I have read the Adding workflows guidelines
• License permits unrestricted use (educational + commercial)
• Please also take note of the reviewer guidelines below to facilitate a smooth review process.

FOR REVIEWERS:

• .dockstore.yml: file is present and aligned with creator metadata in workflow. ORCID identifiers are strongly encouraged in creator metadata. The .dockstore.yml file is required to run tests
• Workflow is sufficiently generic to be used with lab data and does not hardcode sample names, reference data and can be run without reading an accompanying tutorial.
• In workflow: annotation field contains short description of what the workflow does. Should start with This workflow does/runs/performs … xyz … to generate/analyze/etc …
• In workflow: workflow inputs and outputs have human readable names (spaces are fine, no underscore, dash only where spelling dictates it), no abbreviation unless it is generally understood. Altering input or output labels requires adjusting these labels in the the workflow-tests.yml file as well
• In workflow: name field should be human readable (spaces are fine, no underscore, dash only where spelling dictates it), no abbreviation unless generally understood
• Workflow folder: prefer dash (-) over underscore (_), prefer all lowercase. Folder becomes repository in iwc-workflows organization and is included in TRS id
• Readme explains what workflow does, what are valid inputs and what outputs users can expect. If a tutorial or other resources exist they can be linked. If a similar workflow exists in IWC readme should explain differences with existing workflow and when one might prefer one workflow over another
• Changelog contains appropriate entries
• Large files (> 100 KB) are uploaded to zenodo and location urls are used in test file

[github-actions]

Test Results (powered by Planemo)

Test Summary

Test State	Count
Total	1
Passed	0
Error	1
Failure	0
Skipped	0

Errored Tests

• ❌ kmindex-logan-sra-retrieval.ga_0

  Execution Problem:

  • Failed to run workflow, invocation ended in [failed] state.
    

  Workflow invocation details

  • Invocation Messages

    • Invocation scheduling failed because step 8 requires a dataset collection created by step 6, but dataset collection entered a failed state.
  • Steps

    • Step 1: Index To Query:

      • step_state: scheduled

    • Step 2: Query Sequence:

      • step_state: scheduled

    • Step 3: Z-value (k-mer extension factor):

      • step_state: scheduled

    • Step 4: Minimum shared k-mer proportion:

      • step_state: scheduled

    • Step 5: How many accessions to keep:

      • step_state: scheduled

    • Step 6: Unlabelled step (toolshed.g2.bx.psu.edu/repos/iuc/kmindex/kmindex_query/0.6.0+galaxy2):

      • step_state: scheduled

      • Jobs

        • Job 1:

          • Job state is error

          Traceback:

          Job Parameters:

          • Job parameter	Parameter value
            __input_ext	"input"
            __workflow_invocation_uuid__	"2b4c20183a6911f1898a7ced8dcd6908"
            chromInfo	"/tmp/tmp1fkqltoc/galaxy-dev/tool-data/shared/ucsc/chrom/?.len"
            db_opts	{"__current_case__": 1, "db_opts_selector": "db", "kmindex": "VIRALRNA_ROD"}
            dbkey	"?"
            fast	false
            format	"json"
            threshold	"0.5"
            verbose	"info"
            zvalue	"6"

    • Step 7: (Boolean) Subset Accessions (toolshed.g2.bx.psu.edu/repos/iuc/map_param_value/map_param_value/0.2.0):

      • step_state: scheduled

      • Jobs

        • Job 1:

          • Job state is running

          Command Line:

          • cd ../; python _evaluate_expression_.py

          Traceback:

          Job Parameters:

          • Job parameter	Parameter value
            __input_ext	"input"
            __workflow_invocation_uuid__	"2b4c20183a6911f1898a7ced8dcd6908"
            chromInfo	"/tmp/tmp1fkqltoc/galaxy-dev/tool-data/shared/ucsc/chrom/?.len"
            dbkey	"?"
            input_param_type	{"__current_case__": 1, "input_param": "5", "mappings": [{"__index__": 0, "from": "0", "to": "false"}], "type": "integer"}
            output_param_type	"boolean"
            unmapped	{"__current_case__": 0, "on_unmapped": "input"}

    • Step 8: Parse SRA Accessions:

      • step_state: new

    • Step 9: Unlabelled step:

      • step_state: new

    • Step 10: Sort Accessions By Best Match:

      • step_state: new

    • Step 11: Unlabelled step:

      • step_state: new

    • Step 12: Subset Accessions:

      • step_state: new

    • Step 13: Choose Subset or Full Set of Accessions:

      • step_state: new

    • Step 14: Unlabelled step:

      • step_state: new

      • Subworkflow Steps
  • Other invocation details

    • error_message

      • Failed to run workflow, invocation ended in [failed] state.

    • history_id

      • 2c89423437022e58

    • history_state

      • running

    • invocation_id

      • 2c89423437022e58

    • invocation_state

      • failed

    • messages

      • [{'dependent_workflow_step_id': 5, 'hdca_id': '2c89423437022e58', 'reason': 'collection_failed', 'workflow_step_id': 7}]

    • workflow_id

      • c4b5026932dea8b0

[github-actions]

Test Results (powered by Planemo)

Test Summary

Test State	Count
Total	2
Passed	0
Error	2
Failure	0
Skipped	0

Errored Tests

• ❌ kmindex-logan-sra-retrieval.ga_0

  Execution Problem:

  • Failed to run workflow, invocation ended in [failed] state.
    

  Workflow invocation details

  • Invocation Messages

    • Invocation scheduling failed because step 8 requires a dataset collection created by step 6, but dataset collection entered a failed state.
  • Steps

    • Step 1: Index To Query:

      • step_state: scheduled

    • Step 2: Query Sequence:

      • step_state: scheduled

    • Step 3: Z-value (k-mer extension factor):

      • step_state: scheduled

    • Step 4: Minimum shared k-mer proportion:

      • step_state: scheduled

    • Step 5: How many accessions to keep:

      • step_state: scheduled

    • Step 6: Unlabelled step (toolshed.g2.bx.psu.edu/repos/iuc/kmindex/kmindex_query/0.6.0+galaxy2):

      • step_state: scheduled

      • Jobs

        • Job 1:

          • Job state is error

          Traceback:

          Job Parameters:

          • Job parameter	Parameter value
            __input_ext	"input"
            __workflow_invocation_uuid__	"f55726b63b2011f1898a7ced8d636b86"
            chromInfo	"/tmp/tmpf04uph3m/galaxy-dev/tool-data/shared/ucsc/chrom/?.len"
            db_opts	{"__current_case__": 1, "db_opts_selector": "db", "kmindex": "VIRALRNA_ROD"}
            dbkey	"?"
            fast	false
            format	"json"
            threshold	"0.5"
            verbose	"info"
            zvalue	"6"

    • Step 7: (Boolean) Subset Accessions (toolshed.g2.bx.psu.edu/repos/iuc/map_param_value/map_param_value/0.2.0):

      • step_state: scheduled

      • Jobs

        • Job 1:

          • Job state is running

          Command Line:

          • cd ../; python _evaluate_expression_.py

          Traceback:

          Job Parameters:

          • Job parameter	Parameter value
            __input_ext	"input"
            __workflow_invocation_uuid__	"f55726b63b2011f1898a7ced8d636b86"
            chromInfo	"/tmp/tmpf04uph3m/galaxy-dev/tool-data/shared/ucsc/chrom/?.len"
            dbkey	"?"
            input_param_type	{"__current_case__": 1, "input_param": "5", "mappings": [{"__index__": 0, "from": "0", "to": "false"}], "type": "integer"}
            output_param_type	"boolean"
            unmapped	{"__current_case__": 0, "on_unmapped": "input"}

    • Step 8: Parse SRA Accessions:

      • step_state: new

    • Step 9: Unlabelled step:

      • step_state: new

    • Step 10: Sort Accessions By Best Match:

      • step_state: new

    • Step 11: Unlabelled step:

      • step_state: new

    • Step 12: Subset Accessions:

      • step_state: new

    • Step 13: Choose Subset or Full Set of Accessions:

      • step_state: new

    • Step 14: Unlabelled step:

      • step_state: new

      • Subworkflow Steps
  • Other invocation details

    • error_message

      • Failed to run workflow, invocation ended in [failed] state.

    • history_id

      • 465437169756c487

    • history_state

      • running

    • invocation_id

      • 465437169756c487

    • invocation_state

      • failed

    • messages

      • [{'dependent_workflow_step_id': 5, 'hdca_id': '465437169756c487', 'reason': 'collection_failed', 'workflow_step_id': 7}]

    • workflow_id

      • 18b8775abfc02bad
• ❌ lexicmap-sra-retrieval.ga_0

  Execution Problem:

  • Failed to run workflow, invocation ended in [failed] state.
    

  Workflow invocation details

  • Invocation Messages

    • Invocation scheduling failed because an unexpected failure occurred.
  • Steps

    • Step 1: Query Sequence:

      • step_state: scheduled

    • Step 2: LexicMap Index:

      • step_state: scheduled

    • Step 3: Minimum alignment length (bp):

      • step_state: scheduled

    • Step 4: Minimum percent identity (%):

      • step_state: scheduled

    • Step 5: Maximum E-value:

      • step_state: scheduled

    • Step 6: How many accessions to keep:

      • step_state: scheduled

    • Step 7: Minimum total query coverage per genome (%):

      • step_state: new
  • Other invocation details

    • error_message

      • Failed to run workflow, invocation ended in [failed] state.

    • history_id

      • 2bf28cee78a42a42

    • history_state

      • ok

    • invocation_id

      • 2bf28cee78a42a42

    • invocation_state

      • failed

    • messages

      • [{'details': None, 'reason': 'unexpected_failure', 'workflow_step_id': None}]

    • workflow_id

      • 58aafee06b486990

[d-callan]

so this is conceptually complete now i think @mvdbeek in terms of what im hoping to do for brc. i still dont know what to do for tests though

[mvdbeek]

We can mark certain workflows as requiring an external galaxy instance, that's probably what we'll need to do here. That work still needs finishing, I'll see where we are with this.

[github-actions]

Test Results (powered by Planemo)

Test Summary

Test State	Count
Total	2
Passed	0
Error	2
Failure	0
Skipped	0

Errored Tests

• ❌ kmindex-logan-sra-retrieval.ga_0

  Execution Problem:

  • Failed to run workflow, invocation ended in [failed] state.
    

  Workflow invocation details

  • Invocation Messages

    • Invocation scheduling failed because step 8 requires a dataset collection created by step 6, but dataset collection entered a failed state.
  • Steps

    • Step 1: Index To Query:

      • step_state: scheduled

    • Step 2: Query Sequence:

      • step_state: scheduled

    • Step 3: Z-value (k-mer extension factor):

      • step_state: scheduled

    • Step 4: Minimum shared k-mer proportion:

      • step_state: scheduled

    • Step 5: How many accessions to keep:

      • step_state: scheduled

    • Step 6: Unlabelled step (toolshed.g2.bx.psu.edu/repos/iuc/kmindex/kmindex_query/0.6.0+galaxy2):

      • step_state: scheduled

      • Jobs

        • Job 1:

          • Job state is error

          Traceback:

          Job Parameters:

          • Job parameter	Parameter value
            __input_ext	"input"
            __workflow_invocation_uuid__	"b6a145d83cc311f1898a7ced8d4c99b6"
            chromInfo	"/tmp/tmpafnerey0/galaxy-dev/tool-data/shared/ucsc/chrom/?.len"
            db_opts	{"__current_case__": 1, "db_opts_selector": "db", "kmindex": "VIRALRNA_ROD"}
            dbkey	"?"
            fast	false
            format	"json"
            threshold	"0.5"
            verbose	"info"
            zvalue	"6"

    • Step 7: (Boolean) Subset Accessions (toolshed.g2.bx.psu.edu/repos/iuc/map_param_value/map_param_value/0.2.0):

      • step_state: scheduled

      • Jobs

        • Job 1:

          • Job state is running

          Command Line:

          • cd ../; python _evaluate_expression_.py

          Traceback:

          Job Parameters:

          • Job parameter	Parameter value
            __input_ext	"input"
            __workflow_invocation_uuid__	"b6a145d83cc311f1898a7ced8d4c99b6"
            chromInfo	"/tmp/tmpafnerey0/galaxy-dev/tool-data/shared/ucsc/chrom/?.len"
            dbkey	"?"
            input_param_type	{"__current_case__": 1, "input_param": "5", "mappings": [{"__index__": 0, "from": "0", "to": "false"}], "type": "integer"}
            output_param_type	"boolean"
            unmapped	{"__current_case__": 0, "on_unmapped": "input"}

    • Step 8: Parse SRA Accessions:

      • step_state: new

    • Step 9: Unlabelled step:

      • step_state: new

    • Step 10: Sort Accessions By Best Match:

      • step_state: new

    • Step 11: Unlabelled step:

      • step_state: new

    • Step 12: Subset Accessions:

      • step_state: new

    • Step 13: Choose Subset or Full Set of Accessions:

      • step_state: new

    • Step 14: Unlabelled step:

      • step_state: new

      • Subworkflow Steps
  • Other invocation details

    • error_message

      • Failed to run workflow, invocation ended in [failed] state.

    • history_id

      • ea88001d610288e9

    • history_state

      • running

    • invocation_id

      • ea88001d610288e9

    • invocation_state

      • failed

    • messages

      • [{'dependent_workflow_step_id': 5, 'hdca_id': 'ea88001d610288e9', 'reason': 'collection_failed', 'workflow_step_id': 7}]

    • workflow_id

      • be2119f89e3a54d6
• ❌ lexicmap-sra-retrieval.ga_0

  Execution Problem:

  • Failed to run workflow, invocation ended in [failed] state.
    

  Workflow invocation details

  • Invocation Messages

    • Invocation scheduling failed because an unexpected failure occurred.
  • Steps

    • Step 1: Query Sequence:

      • step_state: scheduled

    • Step 2: LexicMap Index:

      • step_state: scheduled

    • Step 3: Minimum alignment length (bp):

      • step_state: scheduled

    • Step 4: Minimum percent identity (%):

      • step_state: scheduled

    • Step 5: Maximum E-value:

      • step_state: scheduled

    • Step 6: Minimum total query coverage per genome (%):

      • step_state: new
  • Other invocation details

    • error_message

      • Failed to run workflow, invocation ended in [failed] state.

    • history_id

      • fc7a1cb47820416b

    • history_state

      • ok

    • invocation_id

      • fc7a1cb47820416b

    • invocation_state

      • failed

    • messages

      • [{'details': None, 'reason': 'unexpected_failure', 'workflow_step_id': None}]

    • workflow_id

      • 108b7495c18dfa85

[mvdbeek]

If it helps we can test against usegalaxy.org now, see https://github.com/galaxyproject/iwc/tree/main/workflows#request-testing-against-an-external-galaxy-instance