You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Expand all lines: docs_roll/docs/English/UserGuide/trackers_and_metrics.md
+27-27Lines changed: 27 additions & 27 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -155,7 +155,7 @@ The ROLL framework automatically records the following types of metrics:
155
155
- time/adv: Duration of the advantages calculation phase.
156
156
157
157
### Execution Phases
158
-
In the following time and memory metrics, {metric_infix} will be replaced with specific execution phase identifiers, such as:
158
+
In the following time and memory metrics, `{metric_infix}` will be replaced with specific execution phase identifiers, such as:
159
159
- train_step: Training phase
160
160
- generate: Text generation/inference phase
161
161
- model_update: Model parameter update/synchronization phase
@@ -165,37 +165,37 @@ In the following time and memory metrics, {metric_infix} will be replaced with s
165
165
- compute_rewards: Reward computation phase
166
166
167
167
#### Time Metrics
168
-
- time/{metric_infix}/total: Total execution time for the entire operation (from entering state_offload_manager to exiting).
169
-
- time/{metric_infix}/execute: Execution time for actual business logic (i.e., the yield part, such as model training, generation, etc.).
170
-
- time/{metric_infix}/onload: Time to load model state (strategy.load_states()) to GPU or memory.
171
-
- time/{metric_infix}/offload: Time to offload model state (strategy.offload_states()) from GPU or memory.
168
+
- time/`{metric_infix}`/total: Total execution time for the entire operation (from entering state_offload_manager to exiting).
169
+
- time/`{metric_infix}`/execute: Execution time for actual business logic (i.e., the yield part, such as model training, generation, etc.).
170
+
- time/`{metric_infix}`/onload: Time to load model state (strategy.load_states()) to GPU or memory.
171
+
- time/`{metric_infix}`/offload: Time to offload model state (strategy.offload_states()) from GPU or memory.
172
172
173
173
#### GPU Memory Metrics
174
174
- Memory snapshot at the beginning (after model state offloading)
175
-
- memory/{metric_infix}/**start/offload**/allocated/{device_id}: Currently allocated GPU memory on a specific device_id.
176
-
- memory/{metric_infix}/**start/offload**/reserved/{device_id}: Currently reserved GPU memory on a specific device_id.
177
-
- memory/{metric_infix}/**start/offload**/max_allocated/{device_id}: Peak allocated GPU memory from the start of this operation to the current moment on a specific device_id.
178
-
- memory/{metric_infix}/**start/offload**/max_reserved/{device_id}: Peak reserved GPU memory from the start of this operation to the current moment on a specific device_id.
175
+
- memory/`{metric_infix}`/**start/offload**/allocated/`{device_id}`: Currently allocated GPU memory on a specific device_id.
176
+
- memory/`{metric_infix}`/**start/offload**/reserved/`{device_id}`: Currently reserved GPU memory on a specific device_id.
177
+
- memory/`{metric_infix}`/**start/offload**/max_allocated/`{device_id}`: Peak allocated GPU memory from the start of this operation to the current moment on a specific device_id.
178
+
- memory/`{metric_infix}`/**start/offload**/max_reserved/`{device_id}`: Peak reserved GPU memory from the start of this operation to the current moment on a specific device_id.
179
179
- Memory snapshot after loading model state (before executing business logic)
180
-
- memory/{metric_infix}/**start/onload**/allocated/{device_id}: Currently allocated GPU memory on a specific device_id.
181
-
- memory/{metric_infix}/**start/onload**/reserved/{device_id}: Currently reserved GPU memory on a specific device_id.
182
-
- memory/{metric_infix}/**start/onload**/max_allocated/{device_id}: Peak allocated GPU memory from the start of this operation to the current moment on a specific device_id.
183
-
- memory/{metric_infix}/**start/onload**/max_reserved/{device_id}: Peak reserved GPU memory from the start of this operation to the current moment on a specific device_id.
180
+
- memory/`{metric_infix}`/**start/onload**/allocated/`{device_id}`: Currently allocated GPU memory on a specific device_id.
181
+
- memory/`{metric_infix}`/**start/onload**/reserved/`{device_id}`: Currently reserved GPU memory on a specific device_id.
182
+
- memory/`{metric_infix}`/**start/onload**/max_allocated/`{device_id}`: Peak allocated GPU memory from the start of this operation to the current moment on a specific device_id.
183
+
- memory/`{metric_infix}`/**start/onload**/max_reserved/`{device_id}`: Peak reserved GPU memory from the start of this operation to the current moment on a specific device_id.
184
184
- Memory snapshot after executing business logic (before offloading model state)
185
-
- memory/{metric_infix}/**end/onload**/allocated/{device_id}: Currently allocated GPU memory on a specific device_id.
186
-
- memory/{metric_infix}/**end/onload**/reserved/{device_id}: Currently reserved GPU memory on a specific device_id.
187
-
- memory/{metric_infix}/**end/onload**/max_allocated/{device_id}: Peak allocated GPU memory from the start of this operation to the current moment on a specific device_id.
188
-
- memory/{metric_infix}/**end/onload**/max_reserved/{device_id}: Peak reserved GPU memory from the start of this operation to the current moment on a specific device_id.
189
-
- memory/{metric_infix}/**end/onload**/max_allocated_frac/{device_id}: Fraction of peak allocated GPU memory relative to total GPU memory on a specific device_id.
190
-
- memory/{metric_infix}/**end/onload**/max_reserved_frac/{device_id}: Fraction of peak reserved GPU memory relative to total GPU memory on a specific device_id.
185
+
- memory/`{metric_infix}`/**end/onload**/allocated/`{device_id}`: Currently allocated GPU memory on a specific device_id.
186
+
- memory/`{metric_infix}`/**end/onload**/reserved/`{device_id}`: Currently reserved GPU memory on a specific device_id.
187
+
- memory/`{metric_infix}`/**end/onload**/max_allocated/`{device_id}`: Peak allocated GPU memory from the start of this operation to the current moment on a specific device_id.
188
+
- memory/`{metric_infix}`/**end/onload**/max_reserved/`{device_id}`: Peak reserved GPU memory from the start of this operation to the current moment on a specific device_id.
189
+
- memory/`{metric_infix}`/**end/onload**/max_allocated_frac/`{device_id}`: Fraction of peak allocated GPU memory relative to total GPU memory on a specific device_id.
190
+
- memory/`{metric_infix}`/**end/onload**/max_reserved_frac/`{device_id}`: Fraction of peak reserved GPU memory relative to total GPU memory on a specific device_id.
191
191
- Memory snapshot after offloading model state (at operation end)
192
-
- memory/{metric_infix}/**end/offload**/allocated/{device_id}: Currently allocated GPU memory on a specific device_id.
193
-
- memory/{metric_infix}/**end/offload**/reserved/{device_id}: Currently reserved GPU memory on a specific device_id.
194
-
- memory/{metric_infix}/**end/offload**/max_allocated/{device_id}: Peak allocated GPU memory from the start of this operation to the current moment on a specific device_id.
195
-
- memory/{metric_infix}/**end/offload**/max_reserved/{device_id}: Peak reserved GPU memory from the start of this operation to the current moment on a specific device_id.
192
+
- memory/`{metric_infix}`/**end/offload**/allocated/`{device_id}`: Currently allocated GPU memory on a specific device_id.
193
+
- memory/`{metric_infix}`/**end/offload**/reserved/`{device_id}`: Currently reserved GPU memory on a specific device_id.
194
+
- memory/`{metric_infix}`/**end/offload**/max_allocated/`{device_id}`: Peak allocated GPU memory from the start of this operation to the current moment on a specific device_id.
195
+
- memory/`{metric_infix}`/**end/offload**/max_reserved/`{device_id}`: Peak reserved GPU memory from the start of this operation to the current moment on a specific device_id.
196
196
197
197
#### CPU Memory Metrics
198
-
- memory/cpu/{metric_infix}/start/rss: Actual physical memory (Resident Set Size) occupied by the process at the start of the operation.
199
-
- memory/cpu/{metric_infix}/start/vms: Virtual memory (Virtual Memory Size) occupied by the process at the start of the operation.
200
-
- memory/cpu/{metric_infix}/end/rss: Actual physical memory occupied by the process at the end of the operation.
201
-
- memory/cpu/{metric_infix}/end/vms: Virtual memory occupied by the process at the end of the operation.
198
+
- memory/cpu/`{metric_infix}`/start/rss: Actual physical memory (Resident Set Size) occupied by the process at the start of the operation.
199
+
- memory/cpu/`{metric_infix}`/start/vms: Virtual memory (Virtual Memory Size) occupied by the process at the start of the operation.
200
+
- memory/cpu/`{metric_infix}`/end/rss: Actual physical memory occupied by the process at the end of the operation.
201
+
- memory/cpu/`{metric_infix}`/end/vms: Virtual memory occupied by the process at the end of the operation.
0 commit comments