Hi,
I'm trying to use Snippy in Galaxy. From 20 pairs of reads,2 of them worked and the other 18 did not. I can't seem to figure out why as I ran the command at the same time for all of them, so it can't be the parameters I set?
The standard error code read:
[13:58:48] This is snippy 4.6.0
[13:58:48] Written by Torsten Seemann
[13:58:48] Obtained from https://github.com/tseemann/snippy
[13:58:48] Detected operating system: linux
[13:58:48] Enabling bundled linux tools.
[13:58:48] Found bwa - /usr/local/bin/bwa
[13:58:48] Found bcftools - /usr/local/bin/bcftools
[13:58:48] Found samtools - /usr/local/bin/samtools
[13:58:48] Found java - /usr/local/bin/java
[13:58:48] Found snpEff - /usr/local/bin/snpEff
[13:58:48] Found samclip - /usr/local/bin/samclip
[13:58:48] Found seqtk - /usr/local/bin/seqtk
[13:58:48] Found parallel - /usr/local/bin/parallel
[13:58:48] Found freebayes - /usr/local/bin/freebayes
[13:58:48] Found freebayes-parallel - /usr/local/bin/freebayes-parallel
[13:58:48] Found fasta_generate_regions.py - /usr/local/bin/fasta_generate_regions.py
[13:58:48] Found vcfstreamsort - /usr/local/bin/vcfstreamsort
[13:58:48] Found vcfuniq - /usr/local/bin/vcfuniq
[13:58:48] Found vcffirstheader - /usr/local/bin/vcffirstheader
[13:58:48] Found gzip - /bin/gzip
[13:58:48] Found vt - /usr/local/bin/vt
[13:58:48] Found snippy-vcf_to_tab - /usr/local/bin/snippy-vcf_to_tab
[13:58:48] Found snippy-vcf_report - /usr/local/bin/snippy-vcf_report
[13:58:48] Checking version: samtools --version is >= 1.7 - ok, have 1.13
[13:58:48] Checking version: bcftools --version is >= 1.7 - ok, have 1.13
[13:58:48] Checking version: freebayes --version is >= 1.1 - ok, have 1.3.2
[13:58:49] Checking version: snpEff -version is >= 4.3 - ok, have 5.0
[13:58:49] Checking version: bwa is >= 0.7.12 - ok, have 0.7.17
[13:58:49] Using reference: /corral4/main/objects/a/8/7/dataset_a8733458-101a-499b-a009-a86e7e599625.dat
[13:58:49] Treating reference as 'fasta' format.
[13:58:49] Will use 1 CPU cores.
[13:58:49] Using read file: /corral4/main/objects/d/a/0/dataset_da05298d-0f1d-49ac-9387-e832f0446863.dat
[13:58:49] Using read file: /corral4/main/objects/b/f/f/dataset_bffd0e18-ccd6-4595-bbc3-2f1a1b3f9d09.dat
[13:58:49] Creating folder: ERR024388_forward
[13:58:49] Changing working directory: ERR024388_forward
[13:58:49] Creating reference folder: reference
[13:58:49] Extracting FASTA and GFF from reference.
[13:58:49] Wrote 1 sequences to ref.fa
[13:58:49] Wrote 0 features to ref.gff
[13:58:49] Freebayes will process 1 chunks of 4959417 bp, 1 chunks at a time.
[13:58:49] Using BAM RG (Read Group) ID: ERR024388_forward
[13:58:49] Running: samtools faidx reference/ref.fa 2>> snps.log
[13:58:49] Running: bwa index reference/ref.fa 2>> snps.log
[13:58:51] Running: mkdir -p reference/genomes && cp -f reference/ref.fa reference/genomes/ref.fa 2>> snps.log
[13:58:51] Running: ln -sf reference/ref.fa . 2>> snps.log
[13:58:51] Running: ln -sf reference/ref.fa.fai . 2>> snps.log
[13:58:51] Running: mkdir -p reference/ref && gzip -c reference/ref.gff > reference/ref/genes.gff.gz 2>> snps.log
[13:58:51] Running: bwa mem -Y -M -R '@rg\tID:ERR024388_forward\tSM:ERR024388_forward' -t 1 reference/ref.fa /corral4/main/objects/d/a/0/dataset_da05298d-0f1d-49ac-9387-e832f0446863.dat /corral4/main/objects/b/f/f/dataset_bffd0e18-ccd6-4595-bbc3-2f1a1b3f9d09.dat | samclip --max 10 --ref reference/ref.fa.fai | samtools sort -n -l 0 -T /corral4/main/jobs/077/057/77057553/tmp --threads 0 -m 0M | samtools fixmate -m --threads 0 - - | samtools sort -l 0 -T /corral4/main/jobs/077/057/77057553/tmp --threads 0 -m 0M | samtools markdup -T /corral4/main/jobs/077/057/77057553/tmp --threads 0 -r -s - - > snps.bam 2>> snps.log
[bam_sort] -m setting (0 bytes) is less than the minimum required (1M).
Trying to run with -m too small can lead to the creation of a very large number
of temporary files. This may make sort fail due to it exceeding limits on the
number of files it can have open at the same time.
Please check your -m parameter. It should be an integer followed by one of the
letters K (for kilobytes), M (megabytes) or G (gigabytes). You should ensure it
is at least the minimum above, and much higher if you are sorting a large file.
[bam_sort] -m setting (0 bytes) is less than the minimum required (1M).
Trying to run with -m too small can lead to the creation of a very large number
of temporary files. This may make sort fail due to it exceeding limits on the
number of files it can have open at the same time.
Please check your -m parameter. It should be an integer followed by one of the
letters K (for kilobytes), M (megabytes) or G (gigabytes). You should ensure it
is at least the minimum above, and much higher if you are sorting a large file.
[bam_mating_core] ERROR: Couldn't read header
[M::bwa_idx_load_from_disk] read 0 ALT contigs
[samclip] samclip 0.4.0 by Torsten Seemann (@torstenseemann)
[samclip] Loading: reference/ref.fa.fai
[samclip] Found 1 sequences in reference/ref.fa.fai
[M::process] read 125000 sequences (10000000 bp)...
[M::process] read 125000 sequences (10000000 bp)...
[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (3, 55101, 0, 18)
[M::mem_pestat] skip orientation FF as there are not enough pairs
[M::mem_pestat] analyzing insert size distribution for orientation FR...
[M::mem_pestat] (25, 50, 75) percentile: (209, 253, 310)
[M::mem_pestat] low and high boundaries for computing mean and std.dev: (7, 512)
[M::mem_pestat] mean and std.dev: (264.19, 74.02)
[M::mem_pestat] low and high boundaries for proper pairs: (1, 613)
[M::mem_pestat] skip orientation RF as there are not enough pairs
[M::mem_pestat] analyzing insert size distribution for orientation RR...
[M::mem_pestat] (25, 50, 75) percentile: (209, 280, 322)
[M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 548)
[M::mem_pestat] mean and std.dev: (248.38, 79.44)
[M::mem_pestat] low and high boundaries for proper pairs: (1, 661)
[M::mem_pestat] skip orientation RR
[M::mem_process_seqs] Processed 125000 reads in 4.356 CPU sec, 4.361 real sec
[13:58:56] Error running command, check ERR024388_forward/snps.log
Though the Galaxy troubleshooting page suggests that there is not a memory allocation issue but that the tool was used wrong (but 2 datasets worked?) or the data was unreadable (again, I don't see hoe 2 would work and the others wouldn't as they were all from the same fastq dump originally). The server says I have only used 12% of 250GB so I doubt the command would take up too much space?
Hi,
I'm trying to use Snippy in Galaxy. From 20 pairs of reads,2 of them worked and the other 18 did not. I can't seem to figure out why as I ran the command at the same time for all of them, so it can't be the parameters I set?
The standard error code read:
[13:58:48] This is snippy 4.6.0
[13:58:48] Written by Torsten Seemann
[13:58:48] Obtained from https://github.com/tseemann/snippy
[13:58:48] Detected operating system: linux
[13:58:48] Enabling bundled linux tools.
[13:58:48] Found bwa - /usr/local/bin/bwa
[13:58:48] Found bcftools - /usr/local/bin/bcftools
[13:58:48] Found samtools - /usr/local/bin/samtools
[13:58:48] Found java - /usr/local/bin/java
[13:58:48] Found snpEff - /usr/local/bin/snpEff
[13:58:48] Found samclip - /usr/local/bin/samclip
[13:58:48] Found seqtk - /usr/local/bin/seqtk
[13:58:48] Found parallel - /usr/local/bin/parallel
[13:58:48] Found freebayes - /usr/local/bin/freebayes
[13:58:48] Found freebayes-parallel - /usr/local/bin/freebayes-parallel
[13:58:48] Found fasta_generate_regions.py - /usr/local/bin/fasta_generate_regions.py
[13:58:48] Found vcfstreamsort - /usr/local/bin/vcfstreamsort
[13:58:48] Found vcfuniq - /usr/local/bin/vcfuniq
[13:58:48] Found vcffirstheader - /usr/local/bin/vcffirstheader
[13:58:48] Found gzip - /bin/gzip
[13:58:48] Found vt - /usr/local/bin/vt
[13:58:48] Found snippy-vcf_to_tab - /usr/local/bin/snippy-vcf_to_tab
[13:58:48] Found snippy-vcf_report - /usr/local/bin/snippy-vcf_report
[13:58:48] Checking version: samtools --version is >= 1.7 - ok, have 1.13
[13:58:48] Checking version: bcftools --version is >= 1.7 - ok, have 1.13
[13:58:48] Checking version: freebayes --version is >= 1.1 - ok, have 1.3.2
[13:58:49] Checking version: snpEff -version is >= 4.3 - ok, have 5.0
[13:58:49] Checking version: bwa is >= 0.7.12 - ok, have 0.7.17
[13:58:49] Using reference: /corral4/main/objects/a/8/7/dataset_a8733458-101a-499b-a009-a86e7e599625.dat
[13:58:49] Treating reference as 'fasta' format.
[13:58:49] Will use 1 CPU cores.
[13:58:49] Using read file: /corral4/main/objects/d/a/0/dataset_da05298d-0f1d-49ac-9387-e832f0446863.dat
[13:58:49] Using read file: /corral4/main/objects/b/f/f/dataset_bffd0e18-ccd6-4595-bbc3-2f1a1b3f9d09.dat
[13:58:49] Creating folder: ERR024388_forward
[13:58:49] Changing working directory: ERR024388_forward
[13:58:49] Creating reference folder: reference
[13:58:49] Extracting FASTA and GFF from reference.
[13:58:49] Wrote 1 sequences to ref.fa
[13:58:49] Wrote 0 features to ref.gff
[13:58:49] Freebayes will process 1 chunks of 4959417 bp, 1 chunks at a time.
[13:58:49] Using BAM RG (Read Group) ID: ERR024388_forward
[13:58:49] Running: samtools faidx reference/ref.fa 2>> snps.log
[13:58:49] Running: bwa index reference/ref.fa 2>> snps.log
[13:58:51] Running: mkdir -p reference/genomes && cp -f reference/ref.fa reference/genomes/ref.fa 2>> snps.log
[13:58:51] Running: ln -sf reference/ref.fa . 2>> snps.log
[13:58:51] Running: ln -sf reference/ref.fa.fai . 2>> snps.log
[13:58:51] Running: mkdir -p reference/ref && gzip -c reference/ref.gff > reference/ref/genes.gff.gz 2>> snps.log
[13:58:51] Running: bwa mem -Y -M -R '@rg\tID:ERR024388_forward\tSM:ERR024388_forward' -t 1 reference/ref.fa /corral4/main/objects/d/a/0/dataset_da05298d-0f1d-49ac-9387-e832f0446863.dat /corral4/main/objects/b/f/f/dataset_bffd0e18-ccd6-4595-bbc3-2f1a1b3f9d09.dat | samclip --max 10 --ref reference/ref.fa.fai | samtools sort -n -l 0 -T /corral4/main/jobs/077/057/77057553/tmp --threads 0 -m 0M | samtools fixmate -m --threads 0 - - | samtools sort -l 0 -T /corral4/main/jobs/077/057/77057553/tmp --threads 0 -m 0M | samtools markdup -T /corral4/main/jobs/077/057/77057553/tmp --threads 0 -r -s - - > snps.bam 2>> snps.log
[bam_sort] -m setting (0 bytes) is less than the minimum required (1M).
Trying to run with -m too small can lead to the creation of a very large number
of temporary files. This may make sort fail due to it exceeding limits on the
number of files it can have open at the same time.
Please check your -m parameter. It should be an integer followed by one of the
letters K (for kilobytes), M (megabytes) or G (gigabytes). You should ensure it
is at least the minimum above, and much higher if you are sorting a large file.
[bam_sort] -m setting (0 bytes) is less than the minimum required (1M).
Trying to run with -m too small can lead to the creation of a very large number
of temporary files. This may make sort fail due to it exceeding limits on the
number of files it can have open at the same time.
Please check your -m parameter. It should be an integer followed by one of the
letters K (for kilobytes), M (megabytes) or G (gigabytes). You should ensure it
is at least the minimum above, and much higher if you are sorting a large file.
[bam_mating_core] ERROR: Couldn't read header
[M::bwa_idx_load_from_disk] read 0 ALT contigs
[samclip] samclip 0.4.0 by Torsten Seemann (@torstenseemann)
[samclip] Loading: reference/ref.fa.fai
[samclip] Found 1 sequences in reference/ref.fa.fai
[M::process] read 125000 sequences (10000000 bp)...
[M::process] read 125000 sequences (10000000 bp)...
[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (3, 55101, 0, 18)
[M::mem_pestat] skip orientation FF as there are not enough pairs
[M::mem_pestat] analyzing insert size distribution for orientation FR...
[M::mem_pestat] (25, 50, 75) percentile: (209, 253, 310)
[M::mem_pestat] low and high boundaries for computing mean and std.dev: (7, 512)
[M::mem_pestat] mean and std.dev: (264.19, 74.02)
[M::mem_pestat] low and high boundaries for proper pairs: (1, 613)
[M::mem_pestat] skip orientation RF as there are not enough pairs
[M::mem_pestat] analyzing insert size distribution for orientation RR...
[M::mem_pestat] (25, 50, 75) percentile: (209, 280, 322)
[M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 548)
[M::mem_pestat] mean and std.dev: (248.38, 79.44)
[M::mem_pestat] low and high boundaries for proper pairs: (1, 661)
[M::mem_pestat] skip orientation RR
[M::mem_process_seqs] Processed 125000 reads in 4.356 CPU sec, 4.361 real sec
[13:58:56] Error running command, check ERR024388_forward/snps.log
Though the Galaxy troubleshooting page suggests that there is not a memory allocation issue but that the tool was used wrong (but 2 datasets worked?) or the data was unreadable (again, I don't see hoe 2 would work and the others wouldn't as they were all from the same fastq dump originally). The server says I have only used 12% of 250GB so I doubt the command would take up too much space?