short cluster jobs

[nitishnih]

Hi folks,

I am working with a biowulf user who had hundreds of short jobs running genome-seek. Some of the job names (or snakemake rules) we identified that ran only for a few seconds were:

    125 bcfstats
     69 ds_postprovars
     86 filtmutect2
    106 flagstat
    125 gather_recal
     44 merge_mutect2
     91 mutect2
     92 octofilter
     31 octosomatic
     63 samplesigpro
    387 somselect
     92 tumormerge

I believe these would perform better on our cluster if they were grouped. Would it be possible to do that? I can see it being grouped where the rules are defined or perhaps at UX level as an option to genome-seek. Let me know if you need more information.

Nitish

[skchronicles]

Hey Nitish,

It is good to hear from you! I hope all is well on your side. I followed up with a DM for some files I would like to review. If you could please send me those, that would be great. I also have quick question.

Are you more concerned about any discrepancies related to submitted wall times vs. actual observed run times (for these fast running jobs), or are you more concerned about the unnecessary overhead that is associated with submitting those fast running jobs to the job scheduler? Or maybe a combination of both?

Please let me know what you think, and I can take a closer look.

Best regards,
@skchronicles