graph LR
RL_Agent["RL Agent"]
RL_Policy["RL Policy"]
Model["Model"]
MCTS["MCTS"]
Experience_Replay_Buffer["Experience Replay Buffer"]
Environment["Environment"]
RL_Agent -- "manages" --> Environment
RL_Agent -- "coordinates" --> RL_Policy
RL_Agent -- "utilizes" --> Experience_Replay_Buffer
RL_Policy -- "queries" --> Model
RL_Policy -- "updates" --> Model
RL_Policy -- "utilizes" --> MCTS
RL_Policy -- "interacts with" --> Environment
RL_Policy -- "stores experience in" --> Experience_Replay_Buffer
Model -- "provides predictions to" --> RL_Policy
Model -- "provides predictions to" --> MCTS
MCTS -- "queries" --> Model
MCTS -- "informs" --> RL_Policy
Experience_Replay_Buffer -- "receives data from" --> RL_Policy
Experience_Replay_Buffer -- "provides data to" --> RL_Policy
Environment -- "provides observations to" --> RL_Policy
Environment -- "receives actions from" --> RL_Policy
Environment -- "returns rewards and next states to" --> RL_Policy
click Model href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/LightZero/Model.md" "Details"
click MCTS href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/LightZero/MCTS.md" "Details"
The lzero project implements a modular architecture for Zero-style reinforcement learning algorithms. At its core, the RL Agent orchestrates the entire learning pipeline, managing the interaction between the RL Policy, Model, MCTS, Experience Replay Buffer, and Environment. The RL Policy is responsible for the algorithm's logic, utilizing the Model for predictions and MCTS for informed action selection. Experiences gathered from the Environment are stored in the Experience Replay Buffer, which then feeds data back to the RL Policy for model updates. This cyclical interaction allows the system to continuously improve its performance through self-play and learning from accumulated experience.
Manages the overall training and evaluation workflow. It initializes and coordinates interactions between the Policy, Environment, Learner, Collector, Evaluator, and Replay Buffer components. It handles experiment setup, logging, and checkpointing.
Related Classes/Methods:
Implements the core logic of the reinforcement learning algorithm (e.g., AlphaZero, MuZero, EfficientZero). This includes defining the neural network architecture (Model), performing Monte Carlo Tree Search (MCTS) for action selection, calculating losses, and updating the model parameters during learning. It has distinct modes for collection, learning, and evaluation.
Related Classes/Methods:
Model [Expand]
Represents the neural network architecture used by the RL Policy. It typically includes a representation network (to encode observations into latent states), a dynamics network (to predict next states and rewards), and a prediction network (to output policy logits and value estimates).
Related Classes/Methods:
MCTS [Expand]
Executes the tree search algorithm to determine optimal actions. It simulates future trajectories, expands the search tree, and calculates visit counts and Q-values to guide action selection.
Related Classes/Methods:
Stores collected experience data (game segments or transitions) from the Environment. It provides mechanisms for sampling batches of data for policy learning, often incorporating prioritization schemes.
Related Classes/Methods:
Represents the simulation environment with which the RL Agent and Policy interact. It provides observations, processes actions, and returns rewards and next states.
Related Classes/Methods: