[Hardware] Provide dockerfile and Pre-built docker images for AMD GPUs. Update docs for AMD users.

Author: aaab8b

README.md

@@ -37,6 +37,7 @@ Leveraging a multi-role distributed architecture with Ray for flexible resource
 
 | 📣   Updates                                                                                                                                                                                                                                                                                                                                                    |
 |:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| **[08/13/2025]** 🎉 ROLL supports AMD GPUs with out-of-box image docker and Dockerfile and specific yamls under `examples/` directory. Please refer to [Installation](https://alibaba.github.io/ROLL/docs/English/QuickStart/installation).                                                    |
 | **[08/10/2025]** 🎉 Agentic RL supports [stepwise learning](examples/qwen2.5-0.5B-agentic/agent_val_frozen_lake_gigpo.yaml), like [GiGPO](https://arxiv.org/abs/2505.10978); Distill supports [multi-model model](examples/qwen2.5-vl-7B-distill/distill_vl_megatron.yaml). Explore the new capabilities!                                                       |
 | **[07/31/2025]** 🎉 Refactor agentic rl design. Support agentic rl [async training](examples/qwen2.5-0.5B-agentic/agent_val_frozen_lake_async.yaml). Explore the new capabilities!                                                                                                                                                                              |
 | **[07/31/2025]** 🎉 Support [DistillPipeline](examples/qwen2.5-7B-distill_megatron/run_distill_pipeline.sh)/[DpoPipeline](examples/qwen2.5-3B-dpo_megatron/run_dpo_pipeline.sh). Support [lora](examples/qwen2.5-7B-rlvr_megatron/rlvr_lora_zero3.yaml). Support [GSPO](https://arxiv.org/abs/2507.18071)                                                       |

docker/Dockerfile.torch280.vllm.AMD

@@ -0,0 +1,30 @@
+FROM rocm/pytorch-training:v25.6
+RUN pip3 install ray==2.48.0 \
+&& pip3 install pydantic==2.11.7 \
+&& pip3 install huggingface-hub==0.34.0 \
+&& pip3 install https://github.com/aaab8b/roll_amd_dependencies/releases/download/v0.1/megatron_core-0.12.2-cp310-cp310-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl \
+&& pip3 install https://github.com/aaab8b/roll_amd_dependencies/releases/download/v0.1/vllm-0.10.0+rocm634-cp310-cp310-linux_x86_64.whl \
+&& pip3 install tensordict \
+&&pip3 install modelscope \
+&& pip3 install loralib \
+&& pip3 install isort \
+&& pip3 install jsonlines \
+&& pip3 install pyext \
+&& pip3 install dacite \
+&& pip3 install codetiming \
+&& pip3 install swanlab \
+&& pip3 install math-verify \
+&& pip3 install gym \
+&& pip3 install gymnasium[toy-text] \
+&& pip3 install gym_sokoban \
+&& pip3 install hydra-core \
+&& pip3 install latex2sympy2==1.5.4 \
+&& pip3 install pycocotools
+RUN git clone https://github.com/alibaba/ROLL.git /app/ROLL
+
+WORKDIR /app/ROLL
+RUN pip3 install ./mcore_adapter \
+&& export PYTHONPATH=/app/ROLL:$PYTHONPATH
+
+RUN sed -i 's/_flash_attn_max_version = PkgVersion("2\.7\.3")/_flash_attn_max_version = PkgVersion("3.0.0.post1")/' /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformer_engine/pytorch/attention.py
+

docs_roll/docs/English/QuickStart/image_address.md

@@ -6,4 +6,7 @@ We provide pre-built Docker images for a quick start (Links will be updated):
 * `torch2.5.1 + SGlang0.4.3`: roll-registry.cn-hangzhou.cr.aliyuncs.com/roll/pytorch:nvcr-24.05-py3-torch251-sglang043
 * `torch2.5.1 + vLLM0.7.3`: roll-registry.cn-hangzhou.cr.aliyuncs.com/roll/pytorch:nvcr-24.05-py3-torch251-vllm073
 
+For AMD GPU users, We provided pre-built Docker images for a quick start as well:
+* `torch2.8.0 + vLLM0.10.0`: hub.docker.com/r/rlsys/roll_opensource
+
 You can also find [Dockerfiles](https://github.com/StephenRi/ROLL/tree/feature/fix-ref-for-docs/docker) under the `docker/` directory to build your own images.

docs_roll/docs/English/QuickStart/installation.md

@@ -2,7 +2,7 @@
 
 ## 🐳 Install from Docker
 
-We provide pre-built Docker images for a quick start. Choose your desired image from the [Image Addresses](https://alibaba.github.io/ROLL/docs/English/QuickStart/image_address).
+We provide pre-built Docker images both on CUDA and ROCm for a quick start. Choose your desired image from the [Image Addresses](https://alibaba.github.io/ROLL/docs/English/QuickStart/image_address).
 
 ## 🛠️ Install from Custom Environment
 
@@ -22,6 +22,21 @@ cd ROLL
 pip install -r requirements.txt # Or follow your specific installation steps
 # For development, consider: pip install -e .
 ```
+
+For AMD users, please ensure you meet the following prerequisites:
+
+```bash
+# Prerequisites
+ROCm Version >= 6.3.4
+PyTorch >= 2.6.0
+vLLM >= 0.8.4
+# Clone the repository and install
+git clone https://github.com/alibaba/ROLL.git
+cd ROLL
+pip install -r requirements.txt # Or follow your specific installation steps
+```
+We highly suggest to use pre-built Docker images from [Image Addresses](https://alibaba.github.io/ROLL/docs/English/QuickStart/image_address) instead of installation from Custom Environment for ROCm users.
+
 ## 🔄 About Model Checkpoint Format
 
 For `MegatronStrategy`, model checkpoints are saved in Megatron format by default. To convert them back to HuggingFace format, please use the following command:

examples/qwen2.5-0.5B-agentic/agent_val_frozen_lake_amd.yaml

@@ -1,5 +1,5 @@
 defaults:
-  - ../config/envs@_here_
+  - ../config/traj_envs@_here_
   - ../config/deepspeed_zero@_here_
   - ../config/deepspeed_zero2@_here_
   - ../config/deepspeed_zero3@_here_
@@ -107,7 +107,7 @@ actor_infer:
   strategy_args:
     strategy_name: vllm
     strategy_config:
-      gpu_memory_utilization: 0.6
+      gpu_memory_utilization: 0.4
       block_size: 16
       load_format: auto
   device_mapping: list(range(0,8))
@@ -126,17 +126,6 @@ reference:
   device_mapping: list(range(0,8))
   infer_batch_size: 2
 
-
-action_pattern: <answer>(.*?)</answer>
-think_action_pattern: <think>(.*?)</think>\s*<answer>(.*?)</answer>
-user_prompt_no_think_format: <answer> [your answer] </answer>
-user_prompt_think_format: <think> [Your thoughts] </think> <answer> [your answer] </answer>
-added_text_no_think: <answer>
-added_text_think: <think>
-
-max_tokens_per_step: 128
-max_actions_per_traj: 10
-
 reward_normalization:
   grouping: traj_group_id # 可以tags(env_type)/traj_group_id(group)/batch(rollout_batch)... group_by计算reward/adv
   method: mean_std # asym_clip / identity / mean_std
@@ -158,71 +147,25 @@ val_env_manager:
   num_groups_partition: [256, 256, 256, 256] # TODO: If not set, all env names divide nums equally. Under the same group, the env config and env seed (prompt) are equal in each generation
 
 
+# Here, you can override variables defined in the imported envs. max_tokens_per_step: 128 in custom_env.SimpleSokoban, here replaced by 64
+max_tokens_per_step: 64
+
 custom_envs:
   SimpleSokoban:
-    env_type: sokoban
-    max_tokens_per_step: ${max_tokens_per_step}
-    user_prompt_format: ${user_prompt_no_think_format}
-    added_text: ${added_text_no_think}
-    env_manager_cls: roll.pipeline.agentic.env_manager.traj_env_manager.TrajEnvManager
-    use_thread_lock: true
-    env_config: # keys should be a subset of SokobanConfig
-      env_instruction: "You are solving the Sokoban puzzle. You are the player and you need to push all boxes to targets. When you are right next to a box, you can push it by moving in the same direction. You cannot push a box through a wall, and you cannot pull a box. The answer must be one of action in a turn, format is <answer>Right</answer>"
-      action_pattern: ${action_pattern}
-      max_steps: ${max_actions_per_traj}
-      dim_x: 6
-      dim_y: 6
-      num_boxes: 1
+    ${custom_env.SimpleSokoban}
   LargerSokoban:
-    env_type: sokoban
-    max_tokens_per_step: ${max_tokens_per_step}
-    user_prompt_format: ${user_prompt_no_think_format}
-    added_text: ${added_text_no_think}
-    env_manager_cls: roll.pipeline.agentic.env_manager.traj_env_manager.TrajEnvManager
-    use_thread_lock: true
-    env_config:
-      env_instruction: "You are solving the Sokoban puzzle. You are the player and you need to push all boxes to targets. When you are right next to a box, you can push it by moving in the same direction. You cannot push a box through a wall, and you cannot pull a box. The answer must be one of action in a turn, format is <answer>Right</answer>"
-      action_pattern: ${action_pattern}
-      max_steps: ${max_actions_per_traj}
-      dim_x: 8
-      dim_y: 8
-      num_boxes: 2
-      search_depth: 10
+    ${custom_env.LargerSokoban}
   SokobanDifferentGridVocab:
-    env_type: sokoban
-    max_tokens_per_step: ${max_tokens_per_step}
-    user_prompt_format: ${user_prompt_no_think_format}
-    added_text: ${added_text_no_think}
-    env_manager_cls: roll.pipeline.agentic.env_manager.traj_env_manager.TrajEnvManager
-    use_thread_lock: true
-    env_config: # keys should be a subset of SokobanConfig
-      env_instruction: "You are solving the Sokoban puzzle. You are the player and you need to push all boxes to targets. When you are right next to a box, you can push it by moving in the same direction. You cannot push a box through a wall, and you cannot pull a box. The answer must be one of action in a turn, format is <answer>Right</answer>"
-      action_pattern: ${action_pattern}
-      max_steps: ${max_actions_per_traj}
-      search_depth: 30
-      dim_x: 6
-      dim_y: 6
-      num_boxes: 1
-      grid_lookup: { 0: "W", 1: ".", 2: "G", 3: "C", 4: "B", 5: "A", 6: "@" }
-      grid_vocab: { "W": "wall", ".": "empty", "G": "target", "C": "box on target", "B": "box", "A": "player", "@": "player on target" }
+    ${custom_env.SokobanDifferentGridVocab}
   FrozenLake:
-    env_type: frozen_lake
-    max_tokens_per_step: ${max_tokens_per_step}
-    user_prompt_format: ${user_prompt_no_think_format}
-    added_text: ${added_text_no_think}
-    env_manager_cls: roll.pipeline.agentic.env_manager.traj_env_manager.TrajEnvManager
-    use_thread_lock: true
-    env_config:
-      env_instruction: "You are solving the FrozenLake puzzle. Forbid the whole and go to the target. You may move to the unintended direction due to the slippery ice. The answer must be one of action in a turn, format is <answer>Right</answer>"
-      action_pattern: ${action_pattern}
-      max_steps: ${max_actions_per_traj}
-      is_slippery: false
+    ${custom_env.FrozenLake}
   FrozenLakeThink:
+    ${custom_env.FrozenLakeThink}
+  FrozenLakeLocallyDefineExamples:  # Can import from unified envs config or define dict locally
     env_type: frozen_lake
     max_tokens_per_step: ${max_tokens_per_step}
     user_prompt_format: ${user_prompt_think_format}
-    added_text: ${added_text_think}
-    env_manager_cls: roll.pipeline.agentic.env_manager.traj_env_manager.TrajEnvManager
+    env_manager_cls: ${env_manager_cls}
     use_thread_lock: true
     env_config:
       env_instruction: "You are solving the FrozenLake puzzle. Forbid the whole and go to the target. You may move to the unintended direction due to the slippery ice. The answer must be one of action in a turn, format is <answer>Right</answer>"

roll/distributed/scheduler/log_monitor.py

@@ -188,12 +188,20 @@ def __init__(self):
         self.node_ip_address = ray.util.get_node_ip_address()
         self.rank = get_driver_rank()
         self.world_size = get_driver_world_size()
-        self.log_monitor = LogMonitor(
-            node_ip_address=self.node_ip_address,
-            logs_dir=self.log_dir,
-            gcs_publisher=StdPublisher(),
-            is_proc_alive_fn=is_proc_alive,
-        )
+        if ray.__version__ <"2.47.0":
+            self.log_monitor = LogMonitor(
+                node_ip_address=self.node_ip_address,
+                logs_dir=self.log_dir,
+                gcs_publisher=StdPublisher(),
+                is_proc_alive_fn=is_proc_alive,
+            )
+        else:
+            self.log_monitor = LogMonitor(
+                node_ip_address=self.node_ip_address,
+                logs_dir=self.log_dir,
+                gcs_client=StdPublisher(),
+                is_proc_alive_fn=is_proc_alive,
+            )
         monitor_logger.setLevel(logging.CRITICAL)
 
         self.exception_monitor = None

roll/third_party/vllm/vllm_0_10_0/llm.py

@@ -187,7 +187,6 @@ def add_requests(
                 params=sampling_params,
                 arrival_time=time.time(),
                 lora_request=lora_request,
-                prompt_adapter_request=None,
             )
 
     def abort_request(self, request_id: Union[str, Iterable[str]]) -> None: