@@ -154,180 +154,38 @@ If you need a storage backend other than what the built-in drivers allow, you ca
154154Refer to [Choose a storage system](/external-storage#choose-storage) for guidance on selecting a backing store and
155155[Lifecycle management](/external-storage#lifecycle) for retention requirements.
156156
157- The following driver keeps payloads on the filesystem. It comes from the
158- [external-storage sample](https://github.com/temporalio/samples-typescript/tree/main/external-storage), which runs it
159- end to end and tests it. A shared filesystem works for local development and for Workers that mount the same volume. For
160- anything else, use a storage system that every Client and Worker can reach.
157+ The [external-storage sample](https://github.com/temporalio/samples-typescript/tree/main/external-storage) implements a
158+ complete driver that keeps payloads on the filesystem, with tests that exercise it end to end. Its `store` and
159+ `retrieve` methods show the shape every driver has. Each one fans out over the batch it was handed, returns results in
160+ the order it received them, and passes the SDK's abort signal down so that one failure cancels its siblings:
161161
162162<!--SNIPSTART typescript-custom-storage-driver -->
163163[external-storage/src/filesystem-storage-driver.ts](https://github.com/temporalio/samples-typescript/blob/main/external-storage/src/filesystem-storage-driver.ts)
164164```ts
165- export class FileSystemStorageDriver implements StorageDriver {
166- readonly name : string ;
167-
168- /**
169- * Stable identifier for this *implementation*, shared by every instance and reported
170- * to the server via Worker heartbeat for observability. Distinct from `name`, which
171- * identifies this particular configured instance. The SDK's own drivers use values
172- * like `aws.s3driver` and `gcp.gcsdriver`.
173- */
174- readonly type = ' sample.filesystemdriver' ;
175-
176- private readonly rootDir : string ;
177- private readonly maxPayloadSize : number ;
178-
179- constructor ({ rootDir , driverName = ' sample.filesystemdriver' , maxPayloadSize }: FileSystemStorageDriverOptions ) {
180- this.rootDir = path .resolve (rootDir );
181- this.name = driverName ;
182- this.maxPayloadSize = maxPayloadSize ?? DEFAULT_MAX_PAYLOAD_SIZE ;
183- }
184-
185- /**
186- * Called with every payload the SDK decided to offload, batched per driver. Must
187- * return one claim per payload, in the same order.
188- *
189- * Throwing here fails the enclosing Workflow or Activity Task *retryably*, so a
190- * transient I/O error is retried rather than killing the Execution. That makes it
191- * safe to let errors propagate instead of, say, silently falling back to inline
192- * payloads, which would defeat the point of the size threshold.
193- */
194- async store (context : StorageDriverStoreContext , payloads : Payload []): Promise < StorageDriverClaim []> {
195- const keyPrefix = buildKeyPrefix (context .target );
196- return runAllAbortingOnFirstError(context.abortSignal, (signal ) =>
197- payloads .map ((payload ) => this .storePayload (payload , keyPrefix , signal )),
198- );
199- }
200-
201- /** Inverse of {@link store } : one payload per claim, in the same order. */
202- async retrieve (context : StorageDriverRetrieveContext , claims : StorageDriverClaim []): Promise < Payload []> {
203- return runAllAbortingOnFirstError(context.abortSignal, (signal ) =>
204- claims .map ((claim ) => this .retrievePayload (claim , signal )),
205- );
206- }
207-
208- private async storePayload (
209- payload : Payload ,
210- keyPrefix : string ,
211- abortSignal : AbortSignal ,
212- ): Promise < StorageDriverClaim > {
213- // Store the encoded Payload proto, not just `payload.data`. A Payload also carries
214- // metadata (the `encoding` key, protobuf message names, anything a custom
215- // PayloadConverter added), and metadata values are arbitrary bytes that would not
216- // survive a round trip through the string-valued claim map. Encoding the whole
217- // message keeps the payload byte-for-byte identical end to end.
218- const payloadBytes = PayloadProto .encode (payload ).finish ();
219- if (payloadBytes.length > this.maxPayloadSize) {
220- throw new Error (
221- ` Payload of ${payloadBytes .length } bytes exceeds the configured maxPayloadSize of ${this .maxPayloadSize } bytes ` ,
222- );
223- }
224-
225- const hashValue = createHash (HASH_ALGORITHM ).update (payloadBytes ).digest (' hex' );
226- const key = ` ${keyPrefix }/${HASH_ALGORITHM }/${hashValue } ` ;
227-
228- try {
229- await this .writeIfAbsent (key , payloadBytes , abortSignal );
230- } catch (err) {
231- // Wrapping adds the context that makes a retry loop diagnosable: a bare ENOENT
232- // says nothing about which key or which storage root. On ES2022 and above, prefer
233- // `new Error(message, { cause: err })` to keep the original stack attached; these
234- // samples target ES2021, so the message is folded in instead.
235- throw new Error(
236- ` FileSystemStorageDriver failed to store [rootDir=${this .rootDir }, key=${key }]: ${describe (err )} ` ,
237- );
238- }
239-
240- // The claim is the only thing that reaches Temporal Server, embedded in the
241- // reference payload that replaces the real one. Keep it small and keep it free of
242- // anything sensitive: it is visible in Workflow History and in the Web UI.
243- //
244- // `rootDir` is deliberately absent. It is deployment configuration, and a second
245- // Worker may well mount the same storage at a different path; putting it in the
246- // claim would pin every historical reference to one machine's filesystem layout.
247- return new StorageDriverClaim ({ key , hashAlgorithm: HASH_ALGORITHM , hashValue });
248- }
249-
250- /**
251- * Writes the blob unless it is already there. The write goes to a temporary file and
252- * is then renamed, because `rename` is atomic: a reader can only ever observe the
253- * complete blob, never a half-written one. Without that, a crash mid-write would
254- * leave a file whose name promises content it does not contain, and content
255- * addressing would hand it to a reader as valid.
256- */
257- private async writeIfAbsent(key: string, payloadBytes: Uint8Array, abortSignal: AbortSignal): Promise<void> {
258- const filePath = this .resolveKey (key );
259- if (await exists (filePath )) return ;
260-
261- await mkdir (path .dirname (filePath ), { recursive: true });
262- const tempPath = ` ${filePath }.${randomUUID ()}.tmp ` ;
263- try {
264- await writeFile(tempPath , payloadBytes , { signal : abortSignal });
265- await rename(tempPath , filePath );
266- } catch (err ) {
267- await unlink(tempPath ).catch(() => undefined );
268- // Two Workers can race to store identical bytes. On POSIX the loser's rename
269- // silently replaces an identical file; on Windows it fails. Either way the blob
270- // is present and correct, so treat that as success.
271- if (await exists (filePath )) return;
272- throw err;
273- }
274- }
275-
276- private async retrievePayload(claim: StorageDriverClaim, abortSignal: AbortSignal): Promise<Payload> {
277- const { key , hashAlgorithm , hashValue: expectedHash } = claim .claimData ;
278- // Claims come off the wire, so validate rather than assume. A missing field means a
279- // claim written by a different driver, or by an older version of this one.
280- if (! key ) {
281- throw new Error(" FileSystemStorageDriver claim is missing required field 'key'" );
282- }
283- if (hashAlgorithm !== HASH_ALGORITHM || ! expectedHash ) {
284- throw new Error(
285- ` FileSystemStorageDriver claim [key=${key }] must carry hashAlgorithm='${HASH_ALGORITHM }' and a hashValue, ` +
286- ` got hashAlgorithm='${hashAlgorithm ?? ' ' }' ` ,
287- );
288- }
289-
290- const filePath = this .resolveKey (key );
291- let payloadBytes : Uint8Array ;
292- try {
293- payloadBytes = await readFile (filePath , { signal: abortSignal });
294- } catch (err ) {
295- throw new Error(
296- ` FileSystemStorageDriver failed to retrieve [rootDir=${this .rootDir }, key=${key }]: ${describe (err )} ` ,
297- );
298- }
299-
300- // Verifying is cheap next to the read and catches truncation, corruption, and a
301- // claim pointing at the wrong blob. Failing loudly here is much better than
302- // handing malformed bytes to the PayloadConverter, where the error would surface
303- // as a confusing deserialization failure far from its cause.
304- const actualHash = createHash (HASH_ALGORITHM ).update (payloadBytes ).digest (' hex' );
305- if (actualHash !== expectedHash ) {
306- throw new Error(
307- ` FileSystemStorageDriver integrity check failed [key=${key }]: ` +
308- ` expected ${HASH_ALGORITHM }:${expectedHash }, got ${HASH_ALGORITHM }:${actualHash } ` ,
309- );
310- }
311-
312- return PayloadProto .decode (payloadBytes );
313- }
314-
315- /**
316- * Maps a key to a path under `rootDir`, refusing anything that escapes it. Keys
317- * arrive from Workflow History, which we should not treat as trusted input: a claim
318- * containing `../../etc/passwd` must not turn into a read outside the blob store.
319- */
320- private resolveKey(key: string): string {
321- const filePath = path .resolve (this .rootDir , key );
322- if (filePath !== this .rootDir && ! filePath .startsWith (this .rootDir + path .sep )) {
323- throw new Error(` FileSystemStorageDriver refused a key that resolves outside rootDir [key=${key }] ` );
324- }
325- return filePath ;
326- }
165+ async store(context: StorageDriverStoreContext, payloads: Payload[]): Promise<StorageDriverClaim[]> {
166+ const keyPrefix = buildKeyPrefix (context .target );
167+ return runAllAbortingOnFirstError (context .abortSignal , (signal ) =>
168+ payloads .map ((payload ) => this .storePayload (payload , keyPrefix , signal )),
169+ );
170+ }
171+
172+ /** Inverse of { @link store } : one payload per claim, in the same order. */
173+ async retrieve(context: StorageDriverRetrieveContext, claims: StorageDriverClaim[]): Promise<Payload[]> {
174+ return runAllAbortingOnFirstError (context .abortSignal , (signal ) =>
175+ claims .map ((claim ) => this .retrievePayload (claim , signal )),
176+ );
327177}
328178```
329179<!--SNIPEND-->
330180
181+ The per-payload work happens in the private `storePayload` and `retrievePayload` methods. Read
182+ [the full driver](https://github.com/temporalio/samples-typescript/blob/main/external-storage/src/filesystem-storage-driver.ts)
183+ for the parts this page only describes: content-addressed keys, an atomic write, a hash check on read, and a guard that
184+ rejects claims resolving outside the storage root.
185+
186+ A shared filesystem works for local development and for Workers that mount the same volume. For anything else, use a
187+ storage system that every Client and Worker can reach.
188+
331189The following sections walk through the key parts of the driver implementation.
332190
333191### 1. Implement the StorageDriver interface
@@ -340,8 +198,7 @@ A custom driver implements the `StorageDriver` interface, which has two readonly
340198- `type` is a string that identifies the driver implementation, and the Worker reports it in its heartbeat. Unlike
341199 `name`, `type` must be the same across all instances of the same driver type regardless of configuration. Two S3
342200 drivers named `"s3-primary"` and `"s3-archive"` would both report `"aws.s3driver"` as their type, while the built-in
343- GCS driver reports `"gcp.gcsdriver"`. The filesystem driver in the preceding code sample reports
344- `"sample.filesystemdriver"`.
201+ GCS driver reports `"gcp.gcsdriver"`. The filesystem driver in the sample reports `"sample.filesystemdriver"`.
345202- `store()` receives an array of payloads and returns one `StorageDriverClaim` per payload. A claim wraps a set of
346203 string key-value pairs that the driver uses to locate the payload later.
347204- `retrieve()` receives the claims that `store()` produced and returns the original payloads.
0 commit comments