AReaL-lite

Introducing AReaL-lite

Our new release AReaL-lite is a light-weight and algorithm-first codebase that prioritizes better development experiences for AI researchers. As a result, AReaL-lite delivers most AReaL functionalities while maintains its high performance with much fewer lines of code. This allows users to build their own agentic training workflows with minimal efforts.

With 80% fewer lines of code, AReaL-lite maintains 90% of AReaL's high performance and core functionality. Check out our AReaL-lite design doc and the quickstart guide to begin your journey with AReaL-lite!

Future Works

AReaL-lite serves as the first phase in AReaL's broader refactoring initiative. It functions both as a standalone training library with intuitive interfaces and as the foundation for AReaL's future core API definitions. The plan is to transform AReaL's current worker-based architecture into an algorithm-first architecture similar to AReaL-lite, where AReaL will extend AReaL-lite's APIs and implementations to support additional backends for efficient large-scale training.

v0.3.0

Milestone Release v0.3.0

• Support asynchronous RL training with decoupled PPO loss, rollout interuption, and staleness control.
• Support Qwen3 training.
• Refactor and simplify Ray-based launching.
• Add github page documentation and tutorials.

What's Changed

• Delete .github/PULL_REQUEST_TEMPLATE directory by @garrett4wade in #46
• Support asynchronous RL training, Qwen3, and the latest SGLang by @garrett4wade in #47
• Update issue templates by @garrett4wade in #48
• [Feature & Doc & Bug Fix] Add docs, simplified ray-based scripts, and fix issues to stablize asynchronous experiments by @garrett4wade in #52
• [Feature] Amend yaml configurations for Ray experiments by @garrett4wade in #53
• [CI] Add github page CI by @garrett4wade in #54
• [Doc] Fix doc by @garrett4wade in #55
• [CI] Fix doc CI by @garrett4wade in #56
• [CI] Fix doc CI again by @garrett4wade in #57
• [CI] Try to fix doc CI by @garrett4wade in #58
• [Feature] Support behavior importance weight capping and update evaluation scripts by @garrett4wade in #59
• [Feature] Create docs and examples for multi-turn agent RL by @garrett4wade in #60
• [Feature] Add link to documentation in README by @garrett4wade in #61
• [Doc & Fix] Simplify the environment setup procedure by @garrett4wade in #62
• [Fix] Fix tutorial async_ppo script and doc structure by @garrett4wade in #63
• [Doc] Fix documentation for using Docker containers and customized agents by @garrett4wade in #64
• [Doc] Add doc for reproducing benchmark results by @garrett4wade in #65
• [Doc] add dataset customization docs by @GurrenLagann97 in #66
• [Doc] Add doc for rollout workers by @garrett4wade in #67
• [Doc] Add customization guide for new algorithms. by @garrett4wade in #68
• [Doc] Mark the equivalent between zero-staleness and synchronous PPO. by @garrett4wade in #69
• [Feature] Update Dockerfile to v0.3.0 by @garrett4wade in #70
• [Doc] Add verl benchmark scripts by @garrett4wade in #71
• [Doc] Add doc for reproducing released models by @garrett4wade in #73
• [Doc] Update README. by @garrett4wade in #74
• add hf links in readme and blog by @GurrenLagann97 in #75
• [Doc] Fix README links. by @garrett4wade in #77
• update blog to discuss staleness by @samjia2000 in #76
• [Doc] Fix README typos and move multi-turn training curve to doc by @garrett4wade in #78
• Switch dataset path / model path to HF location to ease community usage by @Lyken17 in #82
• Fix formatting. by @garrett4wade in #90
• [Feature] Add multi-turn agent example by @garrett4wade in #72
• [Bug] Suppress the SGLang patch error in Ray workers by @garrett4wade in #93
• [Bug] Fix the dependency of a virtual environment for sympy==1.12 by @garrett4wade in #92
• [Doc] Add WeChat QR code by @garrett4wade in #99
• [Doc] fix wandb mode typo by @garrett4wade in #100

New Contributors

• @xssstory made their first contribution in #16
• @Ximingwang-09 made their first contribution in #32
• @HollowMan6 made their first contribution in #38
• @GurrenLagann97 made their first contribution in #66
• @samjia2000 made their first contribution in #76
• @Lyken17 made their first contribution in #82

Full Changelog: v0.2.0...v0.3.0

v0.2.0

Our milestone release, AReaL-boba 🎉

Features

• Quickstart by default yaml config and commandline overrides. Check our updated tutorial!
• Full SGLang support and other system optimizations for 1.5x faster RL training.
• SOTA 7B math reasoning: 61.9 AIME24 & 48.3 AIME25
• 200-sample 32B tuning match QwQ on AIME24

We fully open-source all code, model, and data. Check our technical blog for more details!

Release v0.1.2

Release Notes

Features

• Optimized Data Transfer: Change broadcast-based data transfer into gather-scatter for better performance.
• Refactored Master Worker: Provide better code readability and support asyncio package with uvloop.
• Support Tensorboard Logging: Support CLI options to enable Tensorboard logging on the master worker.

Documentation

• Fix Estimated Training Time: Fixed the estimated training time of 7B experiments in README.

Release v0.1.1

Feature

• User-friendly Launch Tutorials: Updated tutorials and scripts to enable one-click startup of training workflows for faster setup and experimentation.
• Loss Scale Normalization: Normalized loss scaling by token count across micro-batches to stabilize training.
• Configurable Loss Scaling: Added CLI options to customize loss scale window size and initial scaling values.
• Micro-Batch Splitting Optimization: Improved micro-batch splitting logic to ensure balanced workload distribution and enhance training efficiency.

Bug Fixes

• Dataloader Seed Reproducibility: Fixed an issue where dataloaders reused identical random seeds across epochs, ensuring proper shuffling and reproducibility.
• Math Verification Stability: Resolved timeout errors in mathematical verification steps during training.

Documentation

• Update README: Updated 7B-zero model performance figures.