THRESHER - Full Pipeline: Run the complete analysis pipeline from scratch.

thresher full -h

options:
  -h, --help            show this help message and exit
  --metadata METADATA   Path to the input metadata file.
                        The input metadata file should be a tab-delimited file with 3 or 5 columns.
                        3 columns: genome_name, genome_accession, genome_path
                        5 columns: genome_name, genome_accession, genome_path, patient_id, collection_date
                        At least 4 genomes should be provided to perform the analysis.
  -o OUTPUT, --output OUTPUT
                        Path to output directory. If not provided, defaults to thresher_full_output_<YYYY_MM_DD_HHMMSS> under the current working directory.
  --species {sau,sepi,cdiff,kp}
                        Bacteria species.
                        Available options: [sau, sepi, cdiff, kp]
                        sau: Staphylococcus aureus
                        sepi: Staphylococcus epidermidis
                        cdiff: Clostridium difficile
                        kp: Klebsiella pneumoniae
  --epi_mode [True, False]
                        Whether to perform epidemiological analysis.
                        Available Options: [True, False]
                        True: Determine strains, clusters and make plots.
                        False: Determine strains without determing clusters and making plots.
                        Default is True
                        
  --whatsgnu_db_path WHATSGNU_DB_PATH
                        The path to the existing WhatsGNU database.
                        If provided, the WhatsGNU database will not be downloaded.
                        If not provided, defaults to <OUTPUT>/whatsgnu/db
  --bakta_db_type BAKTA_DB_TYPE
                        The type of Bakta database.
                        Available options: [full, light]
                        Default is full
  --bakta_db_path BAKTA_DB_PATH
                        The path of the directory where the existing Bakta database locates.
                        If provided, the Bakta database will not be downloaded.
                        If not provided, defaults to <OUTPUT>/bakta/db
  --snp_coverage_threshold SNP_COVERAGE_THRESHOLD
                        Minimum alignment coverage (0-100) required for pairwise SNP distances to be included in analysis.
                        Low-coverage alignments can yield unreliable SNP count.
                        SNP distance below this threshold are excluded. Default: 80.
  --core_threshold CORE_THRESHOLD
                        Panaroo Core-genome sample threshold. The frequency of a gene in your sample required to classify it as 'core'.
                        Range is 0.0 to 1.0. Default is 0.95.
  --core_bootstrap_method {ultrafast,nonparametric}
                        The bootstrap method for core genome phylogeny used for hierarchical clustering.
                        Available options: [ultrafast, nonparametric]
                        ultrafast: Use ultrafast bootstrap method.
                        nonparametric: Use standard nonparametric bootstrap method.
                        Default is ultrafast.
  --core_bootstrap_number CORE_BOOTSTRAP_NUMBER
                        The number of bootstrap replicates for core genome phylogeny.
                        If method is ultrafast, default is 1000.
                        If method is nonparametric, default is 100.
  --group_bootstrap_method {ultrafast,nonparametric}
                        The bootstrap method for phylogeny of each hierarchical group.
                        Available options: [ultrafast, nonparametric]
                        ultrafast: Use ultrafast bootstrap method.
                        nonparametric: Use standard nonparametric bootstrap method.
                        Default is ultrafast.
  --group_bootstrap_number GROUP_BOOTSTRAP_NUMBER
                        The number of bootstrap replicates for phylogeny of each hierarchical group.
                        If method is ultrafast, default is 1000.
                        If method is nonparametric, default is 100.
  --correction_bootstrap CORRECTION_BOOTSTRAP
                        Minimum bootstrap support threshold for applying phylogenetic corrections to strain composition (default: 0).
  --use_cladebreaker USE_CLADEBREAKER
                        Use CladeBreaker to restrain the strain composition.
                        Options are [True, False].
                        Default is True.
  --threshold_floor THRESHOLD_FLOOR
                        The floor of the range of SNP distances to search for the optimal phylothreshold.
                        Default is 5.
  --threshold_ceiling THRESHOLD_CEILING
                        The ceiling of the range of SNP distances to search for the optimal phylothreshold.
                        Default is 500.
  --singleton_threshold SINGLETON_THRESHOLD
                        The SNP distance threshold above which every genome in the group is considered a singleton.
                        Default is 100.
  --endpoint ENDPOINT   The endpoint method to use for determing clusters and making plots.
                        Available Options: [plateau, peak, discrepancy, public]
                        plateau : Phylothreshold set at a plateau where further increases no longer change the number or composition of strains within the group
                        peak: Phylothreshold set at the peak number of clones defined within the group.
                        discrepancy: Phylothreshold set at the point where the discrepancy is minimized within the group.
                        public: Phylothreshold set at the first time a public genome is included in any strain within the group.
                        Default is plateau.
  --plateau_length PLATEAU_LENGTH
                        The plateau length for the plateau endpoint method. Default is 15.
                        Only used when endpoint method is 'plateau'.
  -t THREADS, --threads THREADS
                        Thread number. Default is 1.
  --prefix PREFIX       Prefix for config file, output files, and analysis naming. If not provided, defaults to timestamp: YYYY_MM_DD_HHMMSS
  --conda_prefix CONDA_PREFIX
                        Directory for conda environments needed for this analysis. If not provided, defaults to <OUTPUT>/conda_envs_<YYYY_MM_DD_HHMMSS>
  --force               Bypass system compatibility checks (operating system and available RAM) and force execution of the pipeline.
                        This may cause instability or failures.

Required Input

Note: At least 4 genomes are required to perform the analysis.

Genome Assemblies:
- Ensure genome assemblies have undergone quality control prior to use as input.
- All genome assemblies must belong to the same species; otherwise, Panaroo will fail to generate a core genome alignment due to insufficient shared core genes, and the pipeline will terminate at this step.
Input Metadata(--metadata):
- Path to a tab-delimited text file containing at least three columns (for lite mode) with no header. For full mode, additional columns are required.
- Columns:
  - Column 1: Genome name (required for both lite and full modes).
  - Column 2: GenBank accession number (required for both lite and full modes). If unavailable, use "new" (all lowercase).
    
    How do I know the Genbank accession for my genomes? See the Genbank Accession page.
  - Column 3: Path to the genome (required for both lite and full modes).
  - Column 4: Patient ID (required for full mode).
  - Column 5: Collection date in the format yyyy-mm-dd (required for full mode).
- Example metadata file for full mode:
  
  Example Input Metadata File
Species(--species):
- Indicate the species being analyzed. Currently supported species include:
- sau (Staphylococcus aureus)
- sepi (Staphylococcus epidermidis)
- cdiff (Clostridium difficile)
- kp (Klebsiella pneumoniae)

Optional Input

Output Directory(--output):
- Path to the output directory. If not provided, defaults to thresher_full_output_<YYYY_MM_DD_HHMMSS> under the current working directory.
Epidemiological Mode(--epi_mode):
- Choose between True (default) and False modes.
- True: Determines strains, clusters, and generates plots.
- False: Determines strains without determining clusters or generating plots.
This mode is only enabled when the input metadata file contains patient ID and collection date columns. If these columns are missing, the pipeline will automatically switch to False mode, regardless of the user's selection.
WhatsGNU Database Path(--whatsgnu_db_path):
- Path to an existing WhatsGNU database. If not provided, the database will be downloaded to <OUTPUT>/whatsgnu/db.
Bakta Database Type(--bakta_db_type) and Path(--bakta_db_path):
- Specify the type of Bakta database (full or light, default is full).
- Path to an existing Bakta database. If not provided, the database will be downloaded to <OUTPUT>/bakta/db.
SNP Coverage Threshold(--snp_coverage_threshold):
- Minimum alignment coverage percentage (0-100) required for pairwise SNP distances to be included in analysis (default: 80). Genome pairs with alignment coverage below this threshold are excluded from downstream cladebreaker analysis.
- Low-coverage alignments can produce artificially low SNP counts, as unaligned regions are not compared and potential variants in those regions go undetected. This can lead to falsely inflated genomic similarity between genomically distantly related genomes. The default threshold of 80% balances sensitivity with reliability.
Core Threshold(--core_threshold):
- Panaroo Core-genome sample threshold. The frequency of a gene in your sample required to classify it as 'core'.
- Range is 0.0 to 1.0. Default is 0.95.
Bootstrap Methods and Numbers(--core/group_bootstrap_method, --core/group_bootstrap_number):
- Specify bootstrap methods (ultrafast or nonparametric) and the number of replicates for both core genome phylogeny and group phylogeny.
- Default methods are ultrafast, with default replicate numbers of 1000 for ultrafast and 100 for nonparametric.
CladeBreaker(--use_cladebreaker):
- Whether or not to use CladeBreaker to restrain the strain composition using the closely related genomes in the WhatsGNU database.
- Enable when investigating putative novel or locally-restricted strains that should be genomically distinct from globally circulating strains.
- True or False, default is True.
Threshold Floor(--threshold_floor):
- The floor of the range tested to search for the optimal phylothreshold (default: 5).
- This parameter sets the lower limit of SNP phylothresholds considered when determining the optimal phylothreshold for defining strains within hierarchical clustering groups.
Threshold Ceiling(--threshold_ceiling):
- The ceiling of the range tested to search for the optimal phylothreshold (default: 500).
- This parameter sets the upper limit of SNP phylothresholds considered when determining the optimal phylothreshold for defining strains within hierarchical clustering groups.
- Adjust this ceiling based on the expected genomic diversity within your dataset and the species being analyzed. A higher ceiling allows for the inclusion of more distantly related genomes in the strain identification process, which may be appropriate for highly diverse species.
Singleton Threshold(--singleton_threshold):
- The SNP distance threshold above which every genome in the hierarchical clustering group is considered a singleton (default: 100).
- If the minimal SNP distance between any two genomes in a hierarchical clustering group is equal to or greater than this threshold, all genomes in that group will be classified as singletons, meaning each genome forms its own unique strain.
- This parameter helps to avoid defining strains in groups where genomes are too distantly related to form meaningful strain. Adjust this threshold based on the expected genomic diversity within your dataset and the species being analyzed.
Correction Bootstrap Threshold(--correction_bootstrap):
- Minimum bootstrap support required to apply phylogenetic corrections to SNP strain composition (default: 0). Nodes with bootstrap values below this threshold are excluded from correction, except for the root node which is always retained.
- Higher thresholds enforce stricter corrections but may reduce the number of corrections applied, particularly in trees with lower overall bootstrap support. The default of 0 applies all possible corrections regardless of bootstrap support, which may be appropriate for small genome groups where high bootstrap values are difficult to achieve. Adjust based on your bootstrap method and resampling depth.
Endpoint Method(--endpoint):
- Choose the endpoint method for determining clusters and generating plots. Options include plateau (default), peak, discrepancy, and public.
- plateau: Phylothreshold set at a plateau where further increases no longer change the number or composition of strains within the group.
- peak: Phylothreshold set at the peak number of clones defined within the group.
- discrepancy: Phylothreshold set at the point where the discrepancy is minimized within the group.
- public: Phylothreshold set at the first time a public genome is included in any strain within the group.
Plateau Length(--plateau_length):
- Specify the plateau length for the plateau endpoint method (default is 15). Only used when the endpoint method is plateau.
Threads(--threads / -t):
- Number of threads to use (default is the maximum available).
- The default thread count is 1, which may result in lengthy runtimes. It is highly recommended to increase the thread count to improve performance, regardless of dataset size.
- Bakta genome annotation runs {threads} parallel jobs, each requiring approximately 10 GB of RAM. Ensure your system has sufficient memory to support the requested thread count.
Prefix(--prefix):
- Prefix for config file, output files, and analysis naming. If not provided, defaults to a timestamp in the format YYYY_MM_DD_HHMMSS.
Conda Prefix(--conda_prefix):
- Directory for conda environments needed for this analysis. If not provided, defaults to <OUTPUT>/conda_envs_<YYYY_MM_DD_HHMMSS>.
Force Execution(--force):
- Bypass system compatibility checks (operating system and available RAM) and force execution of the pipeline. This may cause instability or failures.

Output

Config Files:

Config file used for the analysis: config/config_{prefix}.yaml

MLST Analysis:

Summary:
- mlst/summary/mlst_summary.csv
- Columns included in the MLST summary csv file:
  - genome: Genome name
  - ST: Sequence Type
  - MLST: Broader MLST grouping when available.
    - e.g., for Staphylococcus aureus, the Clonal Complex (CC).
Raw MLST results:
- mlst/raw/{genome_file_name}_mlst.csv

BlastX results for MRSA: (species: sau):

Summary:
- blastx/mrsa/output/summary/blastx_MRSA_strains.csv Columns included in the MRSA blastx strains summary csv files:
  - strain: Strain ID identified by Thresher
  - MRSA: Whether the strain is identified as MRSA (MSSA or MRSA)
- blastx/mrsa/output/summary/blastx_MRSA_genomes.csv Columns included in the MRSA blastx genomes summary csv files:
  - genome: Genome name
  - MRSA: Whether the genome is identified as MRSA (MSSA or MRSA)
Raw blastx results:
- blastx/mrsa/output/raw/{genome_name}_blastx_mrsa.tsv

Bakta Annotation:

bakta_annotation/{genome_name}/

Assembly Scan:

assembly_scan/{genome_name}_assembly_scan.txt

Pair-wise comparison using Mummer4:

Study concatenated reports: mummer4_study/{genome_name}_concatenated.report
public concatenated reports: mummer4_public/{genome_name}_concatenated.report
public SNP matrix: mummer4_public/public_snp_matrix.RDS
Study SNP matrix: mummer4_study/study_snp_matrix.RDS

WhatsGNU:

whatsgnu/{genome_name}/{genome_name}_WhatsGNU_topgenomes.txt

Public genomes downloaded using Datasets:

datasets_topgenomes/

Pan-genome analysis using Panaroo:

panaroo/

Snippy:

Snippy analysis of groups: snippy/output/
Snippy core-genome alignments: snippy/output/cleaned_aln/

IQTree:

Core gene tree:
- iqtree/core_gene_tree/
Group trees:
- iqtree/group_tree/

Hierarchical Clustering:

Hierarchical clustering groups(RDS): thresher/input/hierarchical_clustering_groups.RDS
Hierarchical clustering groups(csv): thresher/input/hierarchical_clustering_groups_simplified.csv Columns included in the hierarchical clustering groups simplified csv file:
- genome: Genome name
- group: Hierarchical clustering group ID
- overlimit: Whether the minimal SNP distance of the genome to any other genome in the same group exceeds 1000 SNPs, If TRUE, the genome will be considered an outlier and excluded from group analysis even though it is assigned to a hierarchical clustering group.
Thresher input: thresher/input/thresher_input.RDS

Strain Composition:

The strain composition files generated by all 4 endpoint methods contain the following columns:

strain_id : Strain ID assigned by Thresher using the specified endpoint method.
genome : Genome name.

Plateau strains:
- thresher/output/plateau_strains.RDS
- thresher/output/plateau_strains.csv
Peak strains:
- thresher/output/peak_strains.RDS
- thresher/output/peak_strains.csv
Discrepancy strains:
- thresher/output/discrepancy_strains.RDS
- thresher/output/discrepancy_strains.csv
Public strains:
- thresher/output/public_strains.RDS
- thresher/output/public_strains.csv
Group-specific thresholds:
- Plateau thresholds: thresher/output/group_plateau.csv
  - Columns included in the group-specific thresholds csv files:
    - group: Hierarchical clustering group ID
    - plateau: Phylothreshold determined for the group using the plateau endpoint method.
    - plateau_length: Plateau length used for determining the phylothreshold.
- Peak thresholds: thresher/output/group_peak.csv
  - Columns included in the group-specific thresholds csv files:
    - group: Hierarchical clustering group ID
    - peak: Phylothreshold determined for the group using the peak endpoint method.
- Discrepancy thresholds: thresher/output/group_discrepancy.csv
  - Columns included in the group-specific thresholds csv files:
    - group: Hierarchical clustering group ID
    - discrepancy: Phylothreshold determined for the group using the discrepancy endpoint method.
- Public thresholds: thresher/output/group_public.csv
  - Columns included in the group-specific thresholds csv files:
    - group: Hierarchical clustering group ID
    - public: Phylothreshold determined for the group using the public endpoint method.

Sanity Check Plots:

Plateau: thresher/output/QC/plateau_qc_plot.pdf
Peak: thresher/output/QC/peak_qc_plot.pdf
Public: thresher/output/QC/public_qc_plot.pdf
Discrepancy: thresher/output/QC/discrepancy_qc_plot.pdf
Figure descriptions:
- X-axis: Phylogenetic Average Distance
  
  The average phylogenetic distance between two strains, calculated by averaging the pairwise branch-length distances between all genomes in strain A and all genomes in strain B on the core genome phylogeny (core_gene tree).
- Y-axis: gSNP Average Distance
  
  The average SNP distance between two strains, calculated by averaging the pairwise SNP distances between all genomes in strain A and all genomes in strain B. Pairwise SNP distances are computed using MUMmer4.
- Colored dots: Each dot represents a comparison between two strains. Blue dots indicate strain pairs within the same hierarchical clustering group. Red dots indicate strain pairs from different hierarchical clustering groups.
- Doted line: A vertical dotted line at gSNP distance of 100.

Sanity Check Tables:

All csv files contain the following columns:
1. subject: subject strain ID
2. query: query strain ID
3. snp_average_distance: average SNP distance between the subject and query strains
4. phylogeny_average_distance: average phylogenetic distance between the subject and query strains
5. same_group: whether the subject and query strains belong to the same hierarchical clustering group

Plateau: thresher/output/QC/plateau_qc_table.csv
Peak: thresher/output/QC/peak_qc_table.csv
Public: thresher/output/QC/public_qc_table.csv
Discrepancy: thresher/output/QC/discrepancy_qc_table.csv

Multi-SNP-Threshold Plot:

thresher/output/MSTP/{Group_ID}_MSTP.pdf
Figure descriptions:
- Shared X-axis: SNP Threshold tested
  
  The SNP thresholds tested to define strains.
- Upper panel: The median and mean bootstrap values of the clades containing the strains defined at the SNP threshold.
- Lower panel:
  - Lines: The number of singletons, clones, and the discrepant genomes when the strain compositions determined by pairwise SNP distances mapped to the corresponding reference-based phylogenetic tree of the group.
  - Annotations: The final phylothresholds determined by each endpoint method (plateau, peak, discrepancy, public) are indicated on the plot.

Visualization:

Core-gene tree annotated with hierarchical clustering groups:
- plots/core_gene_tree_group.pdf
- plots/core_gene_tree_group.RDS(R object)
Core-gene tree annotated with MLST:
- plots/core_gene_tree_mlst.pdf
- plots/core_gene_tree_mlst.RDS(R object)
SNP distance: plots/SNP_Distance.pdf
Strain compositions visualized with ML Phylogeny and SNP distance Heatmap of each hirarchical group for four endpoint methods:
- RDS files:
  - Plateau: plots/strain_compositions/plateau/plateau_strain_tree_snp.RDS
  - Peak: plots/strain_compositions/peak/peak_strain_tree_snp.RDS
  - Discrepancy: plots/strain_compositions/discrepancy/discrepancy_strain_tree_snp.RDS
  - Public: plots/strain_compositions/public/public_strain_tree_snp.RDS
- PDF files:
  - Plateau: plots/strain_compositions/plateau/Group{Group_ID}_plateau_strain_composition.pdf
  - Peak: plots/strain_compositions/peak/Group{Group_ID}_peak_strain_composition.pdf
  - Discrepancy: plots/strain_compositions/discrepancy/Group{Group_ID}_discrepancy_strain_composition.pdf
  - Public: plots/strain_compositions/public/Group{Group_ID}_public_strain_composition.pdf
- Figure descriptions:
  - ML Phylogeny (left panel)
    - Midpoint-rooted ML tree of the group
    - Tip points: Blue circles = study genomes, Brown circles = public genomes (CladeBreaker)
    - Highlighted clades: Genomes assigned to the same clone (strain with ≥2 genomes) are highlighted with a green rectangle and labeled with their strain ID (e.g., 1_1)
    - Scale bar: Branch length in substitutions per site
    - Optimal Phylothreshold: The optimal phylothreshold determined by the selected endpoint method displayed at the top
  - SNP Distance Heatmap (right panel)
    - Rows aligned to phylogeny tip labels; columns show study genomes only
    - White to blue gradient: Pairwise SNP distances between study genomes
    - Grey cells: Comparisons of study genomes to corresponding top public genomes. The color of these cells are greyed out. The cells with no values indicate this study genome is not compared to the public genome(s) as the public genome(s) is/are not the top hit for this study genome in WhatsGNU analysis.
    - Text color indicates SNP quality:
      - Green = good quality (alignments meet coverage threshold)
      - Red = poor quality (low-coverage alignments, interpret with caution)

Cluster Plots: (if full analysis mode is enabled)

Cluster plots: plots/ClusterPlots/
Persistence plot: plots/PersistencePlot.pdf
Clusters summary:
- RDS: thresher/output/clusters_summary.RDS
- CSV: thresher/output/clusters_summary.csv Columns included in the clusters summary csv file:
  - cluster: Cluster ID assigned by Thresher.
  - strain: Strain ID assigned by Thresher.
  - MLST: Broader MLST grouping when available.
    - e.g., for Staphylococcus aureus, the Clonal Complex (CC).
  - AMR: If species is sau, whether the strain is identified as MRSA (MSSA or MRSA).
  - genomes: genome names belonging to the cluster, separated by "|".
  - patients: patient IDs involved in the cluster, separated by "|".
  - first_seen: The earliest collection date among genomes in the cluster.
  - last_seen: The latest collection date among genomes in the cluster.
  - persistence: The persistence of the cluster in days (last_seen - first_seen).

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

THRESHER - Full Pipeline: Run the complete analysis pipeline from scratch.

Required Input

Optional Input

Output

Uh oh!

FilesExpand file tree

usage_full_pipeline.md

Latest commit

History

usage_full_pipeline.md

File metadata and controls

THRESHER - Full Pipeline: Run the complete analysis pipeline from scratch.

Required Input

Optional Input

Output