|
17 | 17 |
|
18 | 18 | This is a comprehensive library to monitor, debug and profile the jobs running on Cloud TPU. |
19 | 19 | To learn about Cloud TPU, refer to the [full documentation](https://cloud.google.com/tpu/docs/intro-to-tpu). |
| 20 | + |
| 21 | +## Features |
| 22 | +### 1. Debugging |
| 23 | +#### 1.1 Collect Stack Traces |
| 24 | +This module will dump the python traces when a fault such as Segmentation fault, Floating-point exception, Illegal operation exception occurs in the program. Additionally, it will also periodically collect stack traces to help debug when a program running on Cloud TPU is stuck or hung somewhere. |
| 25 | + |
| 26 | +## Installation |
| 27 | +To install the package, run the following command on TPU VM: |
| 28 | + |
| 29 | +``` |
| 30 | +pip install cloud-tpu-diagnostics |
| 31 | +``` |
| 32 | + |
| 33 | +## Usage |
| 34 | +To use this package, first import the module: |
| 35 | + |
| 36 | +``` |
| 37 | +from cloud_tpu_diagnostics import diagnostic |
| 38 | +from cloud_tpu_diagnostics.configuration import debug_configuration |
| 39 | +from cloud_tpu_diagnostics.configuration import diagnostic_configuration |
| 40 | +from cloud_tpu_diagnostics.configuration import stack_trace_configuration |
| 41 | +``` |
| 42 | + |
| 43 | +Then, create configuration object for stack traces. The module will only collect stack traces when `collect_stack_trace` parameter is set to `True`. There are following scenarios supported currently: |
| 44 | + |
| 45 | +##### Scenario 1: Do not collect stack traces on faults |
| 46 | + |
| 47 | +``` |
| 48 | +stack_trace_config = stack_trace_configuration.StackTraceConfig( |
| 49 | + collect_stack_trace=False) |
| 50 | +``` |
| 51 | +This configuration will prevent you from collecting stack traces in the event of a fault or process hang. |
| 52 | + |
| 53 | +##### Scenario 2: Collect stack traces on faults and display on console |
| 54 | + |
| 55 | +``` |
| 56 | +stack_trace_config = stack_trace_configuration.StackTraceConfig( |
| 57 | + collect_stack_trace=True, |
| 58 | + stack_trace_to_cloud=False) |
| 59 | +``` |
| 60 | +If there is a fault or process hang, this configuration will show the stack traces on the console (stderr). |
| 61 | + |
| 62 | +##### Scenario 3: Collect stack traces on faults and upload on cloud |
| 63 | + |
| 64 | +``` |
| 65 | +stack_trace_config = stack_trace_configuration.StackTraceConfig( |
| 66 | + collect_stack_trace=True, |
| 67 | + stack_trace_to_cloud=True) |
| 68 | +``` |
| 69 | +This configuration will temporary collect stack traces inside `/tmp/debugging` directory on TPU host if there is a fault or process hang. Additionally, the traces collected in TPU host memory will be uploaded to Google Cloud Logging, which will make it easier to troubleshoot and fix the problems. |
| 70 | + |
| 71 | +By default, stack traces will be collected every 10 minutes. In order to change the duration between two stack trace collection events, add the following configuration: |
| 72 | + |
| 73 | +``` |
| 74 | +stack_trace_config = stack_trace_configuration.StackTraceConfig( |
| 75 | + collect_stack_trace=True, |
| 76 | + stack_trace_to_cloud=True, |
| 77 | + stack_trace_interval_seconds=300) |
| 78 | +``` |
| 79 | +This configuration will collect the stack traces on cloud after every 5 minutes. |
| 80 | + |
| 81 | +Then, create configuration object for debug. |
| 82 | + |
| 83 | +``` |
| 84 | +debug_config = debug_configuration.DebugConfig( |
| 85 | + stack_trace_config=stack_trace_config) |
| 86 | +``` |
| 87 | + |
| 88 | +Then, create configuration object for diagnostic. |
| 89 | + |
| 90 | +``` |
| 91 | +diagnostic_config = diagnostic_configuration.DiagnosticConfig( |
| 92 | + debug_config=debug_config) |
| 93 | +``` |
| 94 | + |
| 95 | +Finally, call the `diagnose()` method using `with` and wrap the statements inside the context manager for which you want to collect the stack traces. |
| 96 | + |
| 97 | +``` |
| 98 | +with diagnostic.diagnose(diagnostic_config): |
| 99 | + run_job(...) |
| 100 | +``` |
0 commit comments