alibaba · PanAndy · Oct 29, 2025 · Oct 27, 2025
diff --git a/.gitignore b/.gitignore
@@ -1,3 +1,4 @@
+*.png
 *.pyc
 */checkpoint_dir
 */dataset

diff --git a/README.md b/README.md
@@ -21,7 +21,7 @@
   <a href="./assets/roll_wechat.png" target="_blank">
     <img src="https://img.shields.io/badge/WeChat-green?logo=wechat" alt="WeChat QR">
   </a>
-    <a href="https://deepwiki.com/alibaba/ROLL" target="_blank">
+  <a href="https://deepwiki.com/alibaba/ROLL" target="_blank">
     <img src="https://deepwiki.com/badge.svg" alt="Ask DeepWiki">
   </a>
 </p>
@@ -40,7 +40,7 @@ Leveraging a multi-role distributed architecture with Ray for flexible resource
 
 | 📣   Updates                                                                                                                                                                                                                                                                                                                            |
 |:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| **[10/23/2025]** 🎉 Our Papers released, see [Asymmetric Proximal Policy Optimization: mini-critics boost LLM reasoning](https://arxiv.org/abs/2510.01656) and [Attention Illuminates LLM Reasoning: The Preplan-and-Anchor Rhythm Enables Fine-Grained Policy Optimization](https://arxiv.org/abs/2510.13554).                                                                                                                      |
+| **[10/23/2025]** 🎉 Our Papers released, see [Asymmetric Proximal Policy Optimization: mini-critics boost LLM reasoning](https://arxiv.org/abs/2510.01656) and [Attention Illuminates LLM Reasoning: The Preplan-and-Anchor Rhythm Enables Fine-Grained Policy Optimization](https://arxiv.org/abs/2510.13554).                         |
 | **[10/14/2025]** 🎉 Our Paper released, see [Part II: ROLL Flash -- Accelerating RLVR and Agentic Training with Asynchrony](https://arxiv.org/abs/2510.11345), the code will be released soon.                                                                                                                                          |
 | **[09/28/2025]** 🎉 Ascend NPU support — see [usage guide](https://alibaba.github.io/ROLL/docs/English/UserGuide/ascend/ascend_usage).                                                                                                                                                                                                  |
 | **[09/25/2025]** 🎉 Our Paper released, see [RollPacker: Mitigating Long-Tail Rollouts for Fast, Synchronous RL Post-Training](https://arxiv.org/abs/2509.21009)                                                                                                                                                                        |
@@ -105,8 +105,8 @@ Leveraging a multi-role distributed architecture with Ray for flexible resource
 [SGLang](https://alibaba.github.io/ROLL/docs/English/UserGuide/backend/sglang)
 
 #### Advanced Features
-[Agentic Asynchronous Parallel Rollout](https://alibaba.github.io/ROLL/docs/English/UserGuide/agentic_async_parallel_rollout)  
-[Agentic Asynchronous Training Feature](https://alibaba.github.io/ROLL/docs/English/UserGuide/async_training_agentic)  
+[Asynchronous Parallel Rollout](https://alibaba.github.io/ROLL/docs/English/UserGuide/async_parallel_rollout)  
+[Asynchronous Training Feature](https://alibaba.github.io/ROLL/docs/English/UserGuide/async_training)  
 
 #### Performance Optimization & Resource Management 
 [Resource Config](https://alibaba.github.io/ROLL/docs/English/UserGuide/device_mapping)   
@@ -162,7 +162,7 @@ We are continuously working to expand ROLL's capabilities:
 - [IPRO](https://arxiv.org/abs/2510.14255): A novel video diffusion framework using reinforcement learning to enhance identity preservation in human-centric I2V generation, optimizing diffusion models with face identity scorer and KL-divergence regularization.
 - [TaoSR-SHE](https://arxiv.org/abs/2510.07972): Stepwise Hybrid Examination Reinforcement Learning Framework for Taobao Search Relevance, with SRPO (hybrid reward model + offline verifier), diversified data filtering, and multi-stage curriculum learning.
 - [EARL](https://arxiv.org/abs/2510.05943): Efficient Agentic RL Systems for LLMs, introducing a dynamic parallelism selector and a layout-aware data dispatcher to boost throughput, reduce memory and data movement bottlenecks, enabling stable large-scale agentic RL without hard context-length limits.
-- [LiveThinking](https://arxiv.org/abs/2510.07685): Real-time reasoning for AI-powered livestreaming by distilling a 670B teacher LLM to a 30B MoE (3B active) via Rejection Sampling Fine-Tuning, then compressing reasoning with GRPO; delivers sub-second latency and ~30x compute reduction, with gains in response correctness (3.3%), helpfulness (21.8%), and GMV in Taobao Live Digital Live Service.
+- [LiveThinking](https://arxiv.org/abs/2510.07685): Real-time reasoning for AI-powered livestreaming by distilling a 670B teacher LLM to a 30B MoE (3B active) via Rejection Sampling Fine-Tuning, then compressing reasoning with GRPO; delivers sub-second latency and ~30x compute reduction, with gains in response correctness (3.3%), helpfulness (21.8%), and GMV in Taobao Live.
 - [TaoSR-AGRL](https://www.arxiv.org/abs/2510.08048): Adaptive Guided Reinforcement Learning for LLM-based e-commerce relevance, introducing Rule-aware Reward Shaping and Adaptive Guided Replay to improve long-horizon reasoning, rule adherence, and training stability in Taobao Search; deployed in main search handling hundreds of millions of users.
 - [RecGPT](https://www.arxiv.org/abs/2507.22879): a next-generation, LLM-driven framework that places user intent at the core of recommender systems, fostering a more sustainable and mutually beneficial ecosystem.
 - [TaoSR1](https://arxiv.org/abs/2508.12365): A novel LLM framework directly deploying Chain-of-Thought (CoT) reasoning for e-commerce query-product relevance prediction, overcoming deployment challenges for superior performance.

diff --git a/docs_roll/docs/English/QuickStart/image_address.md b/docs_roll/docs/English/QuickStart/image_address.md
@@ -3,6 +3,7 @@ We provide pre-built Docker images for a quick start (Links will be updated):
 
 * `torch2.6.0 + SGlang0.4.6`: roll-registry.cn-hangzhou.cr.aliyuncs.com/roll/pytorch:nvcr-24.05-py3-torch260-sglang046
 * `torch2.6.0 + vLLM0.8.4`: roll-registry.cn-hangzhou.cr.aliyuncs.com/roll/pytorch:nvcr-24.05-py3-torch260-vllm084
+* `torch2.8.0 + vLLM0.10.2`: roll-registry-vpc.cn-hangzhou.cr.aliyuncs.com/roll/pytorch:nvcr-25.06-py3-torch280-vllm0102
 
 For AMD GPU users, We provided pre-built Docker images for a quick start as well:
 * `torch2.8.0 + vLLM0.10.0`: hub.docker.com/r/rlsys/roll_opensource

diff --git a/docs_roll/docs/English/UserGuide/agentic/agentic_engineer_practice.md b/docs_roll/docs/English/UserGuide/agentic/agentic_engineer_practice.md
@@ -0,0 +1,301 @@
+# Agentic Engineering Practice Documentation
+
+This document introduces the development practices of the Agentic component in the ROLL framework, including environment manager development protocols, GlobalDataset usage, validation mode configuration, and trajectory synthesis functionality.
+
+## 1. EnvManager Development Protocol
+
+EnvManager is the core component of the Agentic framework, responsible for environment management and trajectory generation. Developing new EnvManagers requires following the following protocol:
+
+### 1.1 Core Loop Mechanism
+
+EnvManager must implement the `run_rollout_loop` method, which follows the following protocol:
+
+```python
+def run_rollout_loop(self, data: DataProto):
+    """
+    1. Each time run_rollout_loop is called, it continuously executes episodes 
+       until receiving a command that data collection is complete
+    2. Need to reset seed to ensure consistency across all groups
+    3. episode_id is obtained from the scheduler
+
+    Seed update logic:
+       group_seed = base_seed + group_id
+       episode_seed = group_seed + episode_id
+
+    trajectory_id: f"{group_id}_{episode_id}_{episode_seed}"
+    """
+
+    # Minimal call example
+    while self.running:
+        # Get episode_id from scheduler
+        self.episode_id = ray.get(self.output_queue.get_episode_id.remote(self.env_config["group_id"]))
+        if self.episode_id is None:
+            break
+
+        # Reset environment
+        rollout_cache = self.reset()
+
+        while rollout_cache is not None and not rollout_cache.terminated and not rollout_cache.truncated:
+            # Make decision
+            lm_output = self.make_decision(rollout_cache)
+            # Execute environment step
+            rollout_cache = self.step(lm_output)
+
+        # Submit trajectory
+        rollout = self.formulate_rollouts(rollout_cache)
+        ray.get(self.output_queue.put.remote(self.env_config['group_id'], self.episode_id, start_step, rollout))
+```
+
+### 1.2 EnvManager Development Constraints
+
+- **While loop infinite loop**: EnvManager continuously executes episodes through a while loop
+- **Exit only when dataset traversal is complete**: When dataset traversal is complete, the `reset()` method returns None, triggering loop exit
+- **Each episode must have corresponding trajectory put**: Each completed episode must submit trajectory data through `output_queue.put`
+
+## 2. GlobalDataset Usage
+
+### 2.1 Design Purpose
+
+To avoid memory access/memory bottlenecks caused by each env reading data independently, the framework provides the GlobalDataset component at the framework level to implement unified management and distribution of datasets.
+
+### 2.2 Class Definition and Location
+
+```python
+# Location: roll.datasets.global_dataset.GlobalDataset
+@ray.remote
+class GlobalDataset:
+    def __init__(self, dataset_name, split: str = "train", mode="sample", dataset_kwargs: Dict = None):
+        # mode: "sample" or "traversal"
+```
+
+### 2.3 Two Working Modes
+
+#### Sample Mode
+- **Purpose**: Random sampling of datasets in training mode
+- **Features**: Randomly select data items each time
+- **Configuration**: `mode="sample"`
+
+#### Traversal Mode
+- **Purpose**: Need to traverse the entire dataset in validation mode
+- **Features**: Traverse dataset sequentially, ensuring each data item is accessed
+- **Configuration**: `mode="traversal"`
+
+### 2.4 Core Features
+
+- **Deterministic sampling**: The `get_data_item` method ensures the same seed returns the same data
+- **State management**: Internally maintains index state, supporting dataset reset and traversal
+
+### 2.5 Usage Example
+
+Refer to the implementation of MathEnv, SWEEnv, TerminalBenchEnv:
+
+```python
+class MathEnv(GEMMathEnv):
+    def __init__(self, dataset_name: str = "", mode: str = "train", **kwargs):
+        # Convert train/val mode to sample/traversal
+        global_dataset_mode = "sample" if self.mode == "train" else "traversal"
+
+        self.dataset = GlobalDataset.options(
+            name=f"{self.mode}_{dataset_name}",
+            get_if_exists=True,
+            namespace=RAY_NAMESPACE
+        ).remote(
+            dataset_name=dataset_name,
+            split=split,
+            mode=global_dataset_mode
+        )
+
+        # Create and register dataset_manager, this is necessary for implementing multiple val
+        self.dataset_manager = GlobalDatasetManager.options(
+            name=f"{self.mode}_dataset_manager",
+            get_if_exists=True,
+            namespace=RAY_NAMESPACE
+        ).remote()
+        ray.get(self.dataset_manager.register.remote(
+            dataset_name=dataset_name, 
+            dataset_ref=self.dataset
+        ))
+```
+
+## 3. Validation Dataset Traversal Configuration During Training
+
+### 3.1 Configuration Principles
+
+For scenarios that require dataset traversal (such as math/code/swe validation scenarios), special configuration is required:
+
+- **Set mode parameter when defining env**: In val mode, need to set `mode=val`
+- **Set val_batch_size=-1**: This allows traversal of the entire val dataset
+- **Exit when env.reset returns None**: When dataset traversal is complete, env.reset will return None
+
+### 3.2 MathEnv Implementation Reference
+
+Location: `roll.pipeline.agentic.env.gem.math_env.MathEnv`
+
+```python
+class MathEnv(GEMMathEnv):
+    def __init__(self, mode: str = "train", **kwargs):
+        # Convert mode
+        global_dataset_mode = "sample" if self.mode == "train" else "traversal"
+        self.dataset = GlobalDataset.remote(
+            dataset_name=dataset_name,
+            split=split,
+            mode=global_dataset_mode
+        )
+
+    def reset(self, seed: Optional[None] = None) -> Tuple[str, dict[str, Any]]:
+        data = ray.get(self.dataset.get_data_item.remote(seed=seed))
+        if data is None:
+            return None, None  # Dataset traversal complete
+        # Process data...
+```
+
+### 3.3 YAML Configuration Example
+
+```yaml
+rollout_batch_size: 128
+val_batch_size: -1  # Traverse entire dataset
+
+deep_math:
+    env_type: "roll_math"
+    env_config:
+      mode: val  # Set to validation mode
+      dataset_name: data/math_deepmath_deal.jsonl
+      split: train
+      question_key: prompt
+      answer_key: ground_truth
+```
+
+### 3.4 Random Sampling Evaluation Scenarios
+
+For random sampling evaluation scenarios such as games, simply configure in the conventional way, ensuring the same seed returns the same data. Random sampling is the default implementation, no special configuration required.
+
+## 4. Trajectory Synthesis Dataset Traversal Configuration
+
+### 4.1 AgenticRolloutPipeline Implementation
+
+Location: `roll/pipeline/agentic/agentic_rollout_pipeline.py`
+
+### 4.2 Startup Method
+
+```shell
+python examples/start_agentic_rollout_pipeline.py --config_path $CONFIG_PATH --config_name $CONFIG_NAME
+```
+
+### 4.3 Core Configuration Reference
+
+```yaml
+# Trajectory storage directory
+rollout_dump_dir: /data/oss_bucket_0/lixing/log/swe/${model_name}/rollout_trajectories
+
+# Support ODPS storage
+# rollout_dump_dir: odps://odps_project/tables/table_name/ds=${model_name}
+
+# Environment manager configuration
+train_env_manager:
+  max_env_num_per_worker: 16
+  num_env_groups: 32
+  group_size: 1  # Support multiple trajectories for the same prompt rollout
+  tags: [SWEEnvVal]
+  num_groups_partition: [32]
+
+# Custom environment configuration
+custom_envs:
+  SWEEnvVal:
+    env_type: "swe_env"
+    env_config:
+      mode: val  # Validation mode
+```
+
+### 4.4 Trajectory Dump Configuration
+
+In the `formulate_rollouts` method of EnvManager, need to register dump fields and types:
+
+```python
+def formulate_rollouts(self, rollout_cache: RolloutCache):
+    # Prepare data
+    save = {
+        "task_idx": task_idx,
+        "episode_score": episode_score,
+        "traj_messages": traj_messages,
+        "metrics": metrics,
+        # ... other fields
+    }
+
+    # Register dump fields
+    lm_input.non_tensor_batch["model_name"] = np.array(
+        [os.path.basename(self.pipeline_config.base_dir)], dtype=object
+    )
+    lm_input.non_tensor_batch["save_content"] = np.array([json.dumps(save)], dtype=object)
+    lm_input.non_tensor_batch["step"] = np.array([self.current_step], dtype=object)
+    lm_input.non_tensor_batch["task_idx"] = np.array([task_idx], dtype=object)
+    lm_input.non_tensor_batch["stop_reason"] = np.array([stop_reason], dtype=object)
+    lm_input.non_tensor_batch["mode"] = np.array([self.mode], dtype=object)
+    lm_input.non_tensor_batch["episode_score"] = np.array([episode_score], dtype=object)
+
+    # Configure database field types
+    colummns_config = [
+        ["task_idx", "bigint"],
+        ["model_name", "string"],
+        ["stop_reason", "string"],
+        ["episode_score", "double"],
+        ["mode", "string"],
+        ["save_content", "string"],
+    ]
+    lm_input.meta_info["COLUMMNS_CONFIG"] = colummns_config
+
+    return lm_input
+```
+
+### 4.5 Important Notes
+
+- **Keys in columns_config will be removed from data_proto after dump**
+- **save_content field contains complete trajectory information, stored in JSON format**
+- **Support local file system and ODPS table storage**
+- **Each trajectory has a unique trajectory_id for tracking**
+
+## 5. Trajectory Filtering
+
+### 5.1 Usage Method
+
+The trajectory filtering function is implemented by configuring the filter class through `roll.pipeline.agentic.agentic_config.EnvManagerConfig.group_filter_cls`. `roll.pipeline.agentic.agentic_pipeline.GroupFilter` is the default implementation.
+
+### 5.2 Custom Filtering Logic
+
+Custom complex trajectory filtering logic can be implemented, for example:
+
+```python
+class GroupFilter:
+    def __init__(self, config: AgenticConfig, env_manager_config: EnvManagerConfig, mode: str):
+        pass
+
+    def filter(self, group_id: int, episode_id: int, group: list[DataProto]):
+        for data in group:
+            if data.meta_info["drop_flag"]:
+                return True
+```
+
+Through custom filter functions, flexible filtering strategies can be implemented based on various trajectory attributes (such as score, length, stop reason, etc.).
+
+## 6. Frequently Asked Questions
+
+### Q1: How to handle dataset traversal completion?
+
+A: Check the `get_data_item` return value in the `reset` method. If it returns None, it means dataset traversal is complete, and you should return None to exit the loop.
+
+### Q2: How to ensure experiment reproducibility?
+
+A: Through a unified seed management mechanism, ensure the same seed returns the same data. The `get_data_item` method of GlobalDataset guarantees this.
+
+### Q3: How to handle large-scale trajectory data storage?
+
+A: You can use ODPS table storage by configuring `rollout_dump_dir` as an `odps://` format URL. For example:
+
+```yaml
+rollout_dump_dir: odps://odps_project/tables/table_name/ds=${model_name}
+```
+
+### Q4: How to debug the trajectory generation process?
+
+A: You can debug the trajectory generation process by configuring log levels and adding custom logs. Trajectory data will be completely saved in JSON format for easy analysis.
+
+For multi-round interaction local debugging, refer to the documentation: [Debug Guide](../../../简体中文/快速开始/debug_guide.md)
diff --git a/docs_roll/docs/English/UserGuide/algorithms/GSPO.md b/docs_roll/docs/English/UserGuide/algorithms/GSPO.md
@@ -45,7 +45,7 @@ add_token_level_kl: false
 
 ### Core Parameter Descriptions
 
-- `adv_estimator`: Advantage estimator type, set to "reinforce"
+- `adv_estimator`: Advantage estimator type, set to "grpo"
 - `importance_sampling`: Importance sampling method, set to "seq" for sequence-level sampling
 - `rollout_batch_size`: Number of prompts per rollout_batch_size
 - `num_return_sequences_in_group`: Number of responses generated per prompt (group size), the total number of samples trained per pipeline step is (rollout_batch_size * num_return_sequences_in_group)

diff --git a/docs_roll/docs/English/UserGuide/algorithms/RAFT_Plus_Plus.md b/docs_roll/docs/English/UserGuide/algorithms/RAFT_Plus_Plus.md
@@ -45,7 +45,7 @@ add_token_level_kl: false
 
 ### Core Parameter Descriptions
 
-- `adv_estimator`: Advantage estimator type, set to "reinforce", which is the core configuration of RAFT++ algorithm
+- `adv_estimator`: Advantage estimator type, set to "grpo", which is the core configuration of RAFT++ algorithm
 - `norm_mean_type`: Mean type for reward normalization: the options are "batch", "group", "running", or None; the default is None
 - `norm_std_type`: Std type for reward normalization: the options are "batch", "group", "running", or None; the default is None
 - `whiten_advantages`: Whether to whiten advantage values, default value is false