Skip to content

Commit 053e6b0

Browse files
SurbhiJainUSCcopybara-github
authored andcommitted
Update README for the initial release of cloud-tpu-diagnostics package to production
PiperOrigin-RevId: 538581626
1 parent d1fe368 commit 053e6b0

2 files changed

Lines changed: 84 additions & 2 deletions

File tree

pip_package/CHANGELOG.md

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -38,9 +38,10 @@ To release a new version (e.g. from `1.0.0` -> `2.0.0`):
3838

3939
## [Unreleased]
4040

41-
## [0.1.0] - YYYY-MM-DD
41+
## [0.1.0] - 2023-06-08
4242

43-
* Initial release
43+
* Initial release of cloud-tpu-diagnostics PyPI package
44+
* FEATURE: Contains debug module to collect stack traces on faults
4445

4546
[Unreleased]: https://github.com/google/cloud-tpu-monitoring-debugging/compare/v0.1.0...HEAD
4647
[0.1.0]: https://github.com/google/cloud-tpu-monitoring-debugging/releases/tag/v0.1.0

pip_package/README.md

Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,3 +17,84 @@
1717

1818
This is a comprehensive library to monitor, debug and profile the jobs running on Cloud TPU.
1919
To learn about Cloud TPU, refer to the [full documentation](https://cloud.google.com/tpu/docs/intro-to-tpu).
20+
21+
## Features
22+
### 1. Debugging
23+
#### 1.1 Collect Stack Traces
24+
This module will dump the python traces when a fault such as Segmentation fault, Floating-point exception, Illegal operation exception occurs in the program. Additionally, it will also periodically collect stack traces to help debug when a program running on Cloud TPU is stuck or hung somewhere.
25+
26+
## Installation
27+
To install the package, run the following command on TPU VM:
28+
29+
```
30+
pip install cloud-tpu-diagnostics
31+
```
32+
33+
## Usage
34+
To use this package, first import the module:
35+
36+
```
37+
from cloud_tpu_diagnostics import diagnostic
38+
from cloud_tpu_diagnostics.configuration import debug_configuration
39+
from cloud_tpu_diagnostics.configuration import diagnostic_configuration
40+
from cloud_tpu_diagnostics.configuration import stack_trace_configuration
41+
```
42+
43+
Then, create configuration object for stack traces. The module will only collect stack traces when `collect_stack_trace` parameter is set to `True`. There are following scenarios supported currently:
44+
45+
##### Scenario 1: Do not collect stack traces on faults
46+
47+
```
48+
stack_trace_config = stack_trace_configuration.StackTraceConfig(
49+
collect_stack_trace=False)
50+
```
51+
This configuration will prevent you from collecting stack traces in the event of a fault or process hang.
52+
53+
##### Scenario 2: Collect stack traces on faults and display on console
54+
55+
```
56+
stack_trace_config = stack_trace_configuration.StackTraceConfig(
57+
collect_stack_trace=True,
58+
stack_trace_to_cloud=False)
59+
```
60+
If there is a fault or process hang, this configuration will show the stack traces on the console (stderr).
61+
62+
##### Scenario 3: Collect stack traces on faults and upload on cloud
63+
64+
```
65+
stack_trace_config = stack_trace_configuration.StackTraceConfig(
66+
collect_stack_trace=True,
67+
stack_trace_to_cloud=True)
68+
```
69+
This configuration will temporary collect stack traces inside `/tmp/debugging` directory on TPU host if there is a fault or process hang. Additionally, the traces collected in TPU host memory will be uploaded to Google Cloud Logging, which will make it easier to troubleshoot and fix the problems.
70+
71+
By default, stack traces will be collected every 10 minutes. In order to change the duration between two stack trace collection events, add the following configuration:
72+
73+
```
74+
stack_trace_config = stack_trace_configuration.StackTraceConfig(
75+
collect_stack_trace=True,
76+
stack_trace_to_cloud=True,
77+
stack_trace_interval_seconds=300)
78+
```
79+
This configuration will collect the stack traces on cloud after every 5 minutes.
80+
81+
Then, create configuration object for debug.
82+
83+
```
84+
debug_config = debug_configuration.DebugConfig(
85+
stack_trace_config=stack_trace_config)
86+
```
87+
88+
Then, create configuration object for diagnostic.
89+
90+
```
91+
diagnostic_config = diagnostic_configuration.DiagnosticConfig(
92+
debug_config=debug_config)
93+
```
94+
95+
Finally, call the `diagnose()` method using `with` and wrap the statements inside the context manager for which you want to collect the stack traces.
96+
97+
```
98+
with diagnostic.diagnose(diagnostic_config):
99+
run_job(...)
100+
```

0 commit comments

Comments
 (0)