graph LR
Data_I_O_Handler["Data I/O Handler"]
Data_Parsing["Data Parsing"]
Data_Curation["Data Curation"]
Data_Merging["Data Merging"]
Data_Validation["Data Validation"]
Data_Filtering_Subsampling["Data Filtering & Subsampling"]
Sequence_Preprocessing["Sequence Preprocessing"]
External_Data_Importer["External Data Importer"]
Measurement_Data_Handler["Measurement Data Handler"]
Core_Data_Structures_Utilities["Core Data Structures & Utilities"]
Data_I_O_Handler -- "provides data to" --> Data_Parsing
Data_I_O_Handler -- "provides data to" --> Data_Curation
Data_I_O_Handler -- "provides data to" --> Data_Validation
Data_I_O_Handler -- "provides data to" --> Data_Merging
Data_I_O_Handler -- "provides data to" --> Data_Filtering_Subsampling
Data_I_O_Handler -- "provides data to" --> Sequence_Preprocessing
Data_I_O_Handler -- "provides data to" --> External_Data_Importer
Data_I_O_Handler -- "provides data to" --> Measurement_Data_Handler
Data_I_O_Handler -- "uses" --> Core_Data_Structures_Utilities
Data_Parsing -- "provides data to" --> Data_Curation
Data_Parsing -- "provides data to" --> Data_Validation
Data_Parsing -- "provides data to" --> Data_Merging
Data_Parsing -- "provides data to" --> Data_Filtering_Subsampling
Data_Parsing -- "provides data to" --> Sequence_Preprocessing
Data_Parsing -- "provides data to" --> Measurement_Data_Handler
Data_Curation -- "provides data to" --> Data_Validation
Data_Curation -- "provides data to" --> Data_Filtering_Subsampling
Data_Curation -- "provides data to" --> Measurement_Data_Handler
Data_Curation -- "uses" --> Core_Data_Structures_Utilities
Data_Merging -- "provides data to" --> Data_Curation
Data_Merging -- "provides data to" --> Data_Validation
Data_Merging -- "provides data to" --> Data_Filtering_Subsampling
Data_Merging -- "provides data to" --> Measurement_Data_Handler
Data_Validation -- "uses" --> Core_Data_Structures_Utilities
Data_Filtering_Subsampling -- "provides data to" --> Data_Validation
Data_Filtering_Subsampling -- "provides data to" --> Measurement_Data_Handler
Data_Filtering_Subsampling -- "uses" --> Core_Data_Structures_Utilities
Sequence_Preprocessing -- "provides data to" --> Data_Validation
External_Data_Importer -- "provides data to" --> Data_Validation
External_Data_Importer -- "provides data to" --> Data_Curation
External_Data_Importer -- "provides data to" --> Data_Parsing
External_Data_Importer -- "uses" --> Core_Data_Structures_Utilities
Measurement_Data_Handler -- "provides data to" --> Data_I_O_Handler
Measurement_Data_Handler -- "uses" --> Core_Data_Structures_Utilities
The Data I/O & Preprocessing component in augur is a critical subsystem responsible for managing the entire lifecycle of data from ingestion to initial transformation, ensuring data quality and readiness for downstream bioinformatics analyses. It adheres to Bioinformatics Pipeline Toolkit patterns by providing modular, data-centric functionalities.
This is the foundational component for all data ingress and egress. It manages reading from and writing to diverse bioinformatics file formats such as FASTA, VCF, JSON, and Newick trees.
Related Classes/Methods:
augur.io(1:1)augur.io.file(1:1)augur.io.metadata(1:1)augur.io.sequences(1:1)augur.io.strains(1:1)augur.io.json(1:1)augur.io.print(1:1)augur.io.shell_command_runner(1:1)
Responsible for interpreting raw input data from various file formats and transforming it into structured Python objects or data frames that are suitable for further processing within the pipeline.
Related Classes/Methods:
Focuses on cleaning, standardizing, and enriching metadata and other data fields. This includes tasks like formatting dates, abbreviating author names, and applying geolocation rules to ensure data consistency.
Related Classes/Methods:
augur.curate(1:1)augur.curate.abbreviate_authors(1:1)augur.curate.apply_geolocation_rules(1:1)augur.curate.format_dates(1:1)
Handles the combination of multiple datasets or metadata files into a single, coherent structure, typically based on common identifiers or keys.
Related Classes/Methods:
Ensures the integrity and correctness of data by validating it against predefined schemas and rules. It identifies and handles inconsistencies or errors in the input data.
Related Classes/Methods:
Implements logic to select specific subsets of data based on various criteria (filtering) or to reduce the overall dataset size (subsampling) for performance or specific analytical needs.
Related Classes/Methods:
augur.filter(1:1)augur.filter._run(1:1)augur.filter.io(1:1)augur.filter.subsample(1:1)augur.filter.weights_file(1:1)
Performs specialized operations on sequence data, including creating indices for efficient access (e.g., augur.index) and masking problematic or ambiguous regions (augur.mask) to improve downstream analysis accuracy.
Related Classes/Methods:
Facilitates the ingestion of data from specific external software formats, such as BEAST output files, converting them into augur-compatible structures for integration into the pipeline.
Related Classes/Methods:
augur.import_(1:1)augur.import_.beast(1:1)
Manages the concatenation and export of measurement-specific data, preparing it for analysis or output. This component ensures that quantitative data is correctly aggregated and formatted.
Related Classes/Methods:
augur.measurements(1:1)augur.measurements.concat(1:1)augur.measurements.export(1:1)
Provides fundamental data types, enumerations (e.g., for validation modes), and a centralized error handling mechanism (AugurError and its subclasses) that are utilized across the entire Data I/O & Preprocessing component. It acts as a foundational support layer.
Related Classes/Methods: