graph LR
Main_Orchestrators["Main Orchestrators"]
Distributed_Executors["Distributed Executors"]
Training_Engines["Training Engines"]
Main_Orchestrators -- "initiates and controls" --> Distributed_Executors
Main_Orchestrators -- "invokes" --> Training_Engines
Distributed_Executors -- "launches and manages" --> Training_Engines
The Experiment Orchestration subsystem is the central control unit for managing the entire lifecycle of machine learning experiments, encompassing both pretraining and finetuning. It orchestrates data flow, model interaction, and optimization, acting as the primary pipeline manager.
These components serve as the primary entry points for experiments. They are responsible for parsing command-line arguments, setting up the execution environment (e.g., distributed mode), initializing the model, data loaders, and optimizers. They orchestrate the overall training and evaluation pipeline, acting as the top-level coordinators.
Related Classes/Methods:
These components facilitate the execution of experiments in distributed environments, primarily leveraging tools like Submitit. They manage the distributed setup, job submission, and encapsulate the training logic to be executed across multiple nodes or GPUs, abstracting away the complexities of distributed training.
Related Classes/Methods:
These components implement the core training and evaluation logic for a single epoch. This includes iterating over data, performing forward and backward passes, updating model weights, and accumulating performance metrics. They are the workhorses that execute the actual learning process.
Related Classes/Methods: