Skip to content

Commit 141c19e

Browse files
committed
docs: source the GCS snippet from Snipsync and clarify credentials
temporalio/features#866 merged, so the go-gcs-driver-create region now exists upstream. Replaces the hand-written GCS code block with a Snipsync wrapper, which adds the source link and keeps the sample tied to code the features repo builds. Also names the processes that need bucket credentials, rather than saying "components": the Client and the Workers each reach the bucket directly. Points at the Amazon S3 and Cloud Storage access control documentation for granting that access.
1 parent c2faa17 commit 141c19e

1 file changed

Lines changed: 32 additions & 23 deletions

File tree

docs/develop/go/best-practices/data-handling/external-storage.mdx

Lines changed: 32 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -35,9 +35,12 @@ follow. Only the driver setup differs between the two. Everything after that is
3535
- A bucket that you have read and write access to. Refer to [lifecycle management](/external-storage#lifecycle) to
3636
ensure that your payloads remain available for the entire lifetime of the Workflow. For multi-region durability, see
3737
[Durable External Storage](/external-storage#durable-external-storage).
38-
- Credentials with permission to write objects on components that store payloads, and to read objects on components that
39-
retrieve them. Because the drivers check whether an object already exists before uploading, storing also requires read
40-
permission. Components that only retrieve payloads do not need write permission.
38+
- Bucket credentials on both your Temporal Client and your Workers, since each reaches the bucket directly. Refer to
39+
[Amazon S3 access control](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-access-control.html) or
40+
[Cloud Storage IAM permissions](https://cloud.google.com/storage/docs/access-control/iam-permissions) for how to grant
41+
access.
42+
- Permission to write objects on processes that store payloads, and to read objects on processes that retrieve them.
43+
Storing also requires read permission, because the drivers check whether an object already exists before uploading.
4144
- Install the driver module, the client adapter for your cloud provider's SDK, and that SDK:
4245

4346
<Tabs groupId="external-storage-backend" queryString>
@@ -78,26 +81,23 @@ follow. Only the driver setup differs between the two. Everything after that is
7881
[environment variables, an IAM role, or your AWS config file](https://docs.aws.amazon.com/sdk-for-go/v2/developer-guide/configure-gosdk.html).
7982

8083
<!--SNIPSTART go-s3-driver-create-->
81-
82-
[features/snippets/external_storage/s3_setup/s3_driver_create.go](https://github.com/temporalio/features/blob/main/features/snippets/external_storage/s3_setup/s3_driver_create.go)
83-
84-
```go
85-
cfg, err := config.LoadDefaultConfig(context.Background(),
86-
config.WithRegion("us-east-2"),
87-
)
88-
if err != nil {
89-
log.Fatalf("load AWS config: %v", err)
90-
}
91-
92-
driver, err := s3driver.NewDriver(s3driver.Options{
93-
Client: awssdkv2.NewClient(s3.NewFromConfig(cfg)),
94-
Bucket: s3driver.StaticBucket("my-temporal-payloads"),
95-
})
96-
if err != nil {
97-
log.Fatalf("create S3 driver: %v", err)
98-
}
99-
```
100-
84+
[features/snippets/external_storage/s3_setup/s3_driver_create.go](https://github.com/temporalio/features/blob/main/features/snippets/external_storage/s3_setup/s3_driver_create.go)
85+
```go
86+
cfg, err := config.LoadDefaultConfig(context.Background(),
87+
config.WithRegion("us-east-2"),
88+
)
89+
if err != nil {
90+
log.Fatalf("load AWS config: %v", err)
91+
}
92+
93+
driver, err := s3driver.NewDriver(s3driver.Options{
94+
Client: awssdkv2.NewClient(s3.NewFromConfig(cfg)),
95+
Bucket: s3driver.StaticBucket("my-temporal-payloads"),
96+
})
97+
if err != nil {
98+
log.Fatalf("create S3 driver: %v", err)
99+
}
100+
```
101101
<!--SNIPEND-->
102102

103103
</TabItem>
@@ -107,6 +107,10 @@ follow. Only the driver setup differs between the two. Everything after that is
107107
The Google Cloud SDK reads
108108
[Application Default Credentials](https://cloud.google.com/docs/authentication/application-default-credentials).
109109

110+
<!--SNIPSTART go-gcs-driver-create-->
111+
112+
[features/snippets/external_storage/gcs_setup/gcs_driver_create.go](https://github.com/temporalio/features/blob/main/features/snippets/external_storage/gcs_setup/gcs_driver_create.go)
113+
110114
```go
111115
gcsClient, err := storage.NewClient(context.Background())
112116
if err != nil {
@@ -122,6 +126,8 @@ follow. Only the driver setup differs between the two. Everything after that is
122126
}
123127
```
124128

129+
<!--SNIPEND-->
130+
125131
</TabItem>
126132

127133
</Tabs>
@@ -153,6 +159,9 @@ follow. Only the driver setup differs between the two. Everything after that is
153159

154160
<!--SNIPEND-->
155161

162+
A Worker inherits this configuration from the Client it is created with. When your Workers run in their own process,
163+
repeat this setup there.
164+
156165
By default, payloads of 256 KiB or larger are offloaded to external storage. You can adjust this with the
157166
`PayloadSizeThreshold` option, even setting it to 1 to externalize all payloads regardless of size. Refer to
158167
[Configure payload size threshold](#configure-payload-size-threshold) for more information.

0 commit comments

Comments
 (0)