graph LR
fairseq_trainer["fairseq.trainer"]
fairseq_models_fairseq_model["fairseq.models.fairseq_model"]
fairseq_criterions_fairseq_criterion["fairseq.criterions.fairseq_criterion"]
fairseq_optim_["fairseq.optim.*"]
fairseq_optim_lr_scheduler_["fairseq.optim.lr_scheduler.*"]
fairseq_data_iterators["fairseq.data.iterators"]
fairseq_checkpoint_utils["fairseq.checkpoint_utils"]
fairseq_distributed_utils["fairseq.distributed.utils"]
fairseq_trainer -- "calls for forward passes on" --> fairseq_models_fairseq_model
fairseq_trainer -- "passes outputs and targets to" --> fairseq_criterions_fairseq_criterion
fairseq_trainer -- "uses" --> fairseq_optim_
fairseq_trainer -- "interacts with" --> fairseq_optim_lr_scheduler_
fairseq_trainer -- "consumes data from" --> fairseq_data_iterators
fairseq_trainer -- "leverages" --> fairseq_checkpoint_utils
fairseq_trainer -- "utilizes" --> fairseq_distributed_utils
fairseq_models_fairseq_model -- "performs forward pass for" --> fairseq_trainer
fairseq_models_fairseq_model -- "parameters are updated by" --> fairseq_optim_
fairseq_models_fairseq_model -- "state is saved/loaded by" --> fairseq_checkpoint_utils
fairseq_models_fairseq_model -- "supported by" --> fairseq_distributed_utils
fairseq_criterions_fairseq_criterion -- "computes loss for" --> fairseq_trainer
fairseq_optim_ -- "updates parameters of" --> fairseq_models_fairseq_model
fairseq_optim_ -- "receives adjustments from" --> fairseq_optim_lr_scheduler_
fairseq_optim_ -- "state is saved/loaded by" --> fairseq_checkpoint_utils
fairseq_optim_lr_scheduler_ -- "provides adjustments to" --> fairseq_optim_
fairseq_optim_lr_scheduler_ -- "state is saved/loaded by" --> fairseq_checkpoint_utils
fairseq_data_iterators -- "provides data to" --> fairseq_trainer
fairseq_checkpoint_utils -- "saves/loads state of" --> fairseq_trainer
fairseq_checkpoint_utils -- "saves/loads state of" --> fairseq_models_fairseq_model
fairseq_checkpoint_utils -- "saves/loads state of" --> fairseq_optim_
fairseq_checkpoint_utils -- "saves/loads state of" --> fairseq_optim_lr_scheduler_
fairseq_distributed_utils -- "provides support for" --> fairseq_trainer
fairseq_distributed_utils -- "provides support for" --> fairseq_models_fairseq_model
The Training & Optimization Engine subsystem in Fairseq is centered around the fairseq.trainer component, which orchestrates the entire training process. This subsystem embodies the core machine learning loop, managing data flow, model execution, loss computation, parameter updates, and distributed training aspects. Its design emphasizes modularity, allowing various components like optimizers, learning rate schedulers, and models to be plugged in.
The central orchestrator of the training process. It manages the training loop, handles forward and backward passes, coordinates gradient computation, parameter updates, learning rate adjustments, and facilitates distributed training. It also manages checkpointing and logging.
Related Classes/Methods:
Represents the neural network architecture being trained. It defines the model's layers and its forward pass logic, processing input data to produce predictions.
Related Classes/Methods:
Computes the loss value, quantifying the discrepancy between model predictions and target values. This loss guides the optimization process by providing a signal for gradient computation.
Related Classes/Methods:
Implements various optimization algorithms (e.g., Adam, SGD) responsible for updating model parameters based on computed gradients to minimize the loss function.
Related Classes/Methods:
Defines strategies for dynamically adjusting the learning rate during training, such as step-wise decay, warm-up, or cosine annealing, to improve convergence and performance.
Related Classes/Methods:
Manages the iteration over datasets, providing training and validation batches to the trainer. It handles aspects like batching, shuffling, and data loading.
Related Classes/Methods:
Provides functionalities to save and load the complete training state, including model parameters, optimizer state, and learning rate scheduler state, enabling training resumption and model persistence.
Related Classes/Methods:
Offers helper functions and abstractions to manage and facilitate distributed training environments, including gradient synchronization, data parallelism, and communication across multiple devices or nodes.
Related Classes/Methods: