Skip to content

Commit 3e48a86

Browse files
committed
docs(external-storage): shorten the custom-driver excerpt
The whole FileSystemStorageDriver class was too long to read inline. Show just store and retrieve, which is what the surrounding prose walks through, and link the sample for the per-payload work in the private helpers: content-addressed keys, the atomic write, the hash check on read, and the guard against claims that resolve outside the storage root. Markers narrowed in samples-typescript to match.
1 parent f94ad79 commit 3e48a86

1 file changed

Lines changed: 25 additions & 168 deletions

File tree

docs/develop/typescript/best-practices/data-handling/external-storage.mdx

Lines changed: 25 additions & 168 deletions
Original file line numberDiff line numberDiff line change
@@ -154,180 +154,38 @@ If you need a storage backend other than what the built-in drivers allow, you ca
154154
Refer to [Choose a storage system](/external-storage#choose-storage) for guidance on selecting a backing store and
155155
[Lifecycle management](/external-storage#lifecycle) for retention requirements.
156156

157-
The following driver keeps payloads on the filesystem. It comes from the
158-
[external-storage sample](https://github.com/temporalio/samples-typescript/tree/main/external-storage), which runs it
159-
end to end and tests it. A shared filesystem works for local development and for Workers that mount the same volume. For
160-
anything else, use a storage system that every Client and Worker can reach.
157+
The [external-storage sample](https://github.com/temporalio/samples-typescript/tree/main/external-storage) implements a
158+
complete driver that keeps payloads on the filesystem, with tests that exercise it end to end. Its `store` and
159+
`retrieve` methods show the shape every driver has. Each one fans out over the batch it was handed, returns results in
160+
the order it received them, and passes the SDK's abort signal down so that one failure cancels its siblings:
161161

162162
<!--SNIPSTART typescript-custom-storage-driver -->
163163
[external-storage/src/filesystem-storage-driver.ts](https://github.com/temporalio/samples-typescript/blob/main/external-storage/src/filesystem-storage-driver.ts)
164164
```ts
165-
export class FileSystemStorageDriver implements StorageDriver {
166-
readonly name: string;
167-
168-
/**
169-
* Stable identifier for this *implementation*, shared by every instance and reported
170-
* to the server via Worker heartbeat for observability. Distinct from `name`, which
171-
* identifies this particular configured instance. The SDK's own drivers use values
172-
* like `aws.s3driver` and `gcp.gcsdriver`.
173-
*/
174-
readonly type = 'sample.filesystemdriver';
175-
176-
private readonly rootDir: string;
177-
private readonly maxPayloadSize: number;
178-
179-
constructor({ rootDir, driverName = 'sample.filesystemdriver', maxPayloadSize }: FileSystemStorageDriverOptions) {
180-
this.rootDir = path.resolve(rootDir);
181-
this.name = driverName;
182-
this.maxPayloadSize = maxPayloadSize ?? DEFAULT_MAX_PAYLOAD_SIZE;
183-
}
184-
185-
/**
186-
* Called with every payload the SDK decided to offload, batched per driver. Must
187-
* return one claim per payload, in the same order.
188-
*
189-
* Throwing here fails the enclosing Workflow or Activity Task *retryably*, so a
190-
* transient I/O error is retried rather than killing the Execution. That makes it
191-
* safe to let errors propagate instead of, say, silently falling back to inline
192-
* payloads, which would defeat the point of the size threshold.
193-
*/
194-
async store(context: StorageDriverStoreContext, payloads: Payload[]): Promise<StorageDriverClaim[]> {
195-
const keyPrefix = buildKeyPrefix(context.target);
196-
return runAllAbortingOnFirstError(context.abortSignal, (signal) =>
197-
payloads.map((payload) => this.storePayload(payload, keyPrefix, signal)),
198-
);
199-
}
200-
201-
/** Inverse of {@link store}: one payload per claim, in the same order. */
202-
async retrieve(context: StorageDriverRetrieveContext, claims: StorageDriverClaim[]): Promise<Payload[]> {
203-
return runAllAbortingOnFirstError(context.abortSignal, (signal) =>
204-
claims.map((claim) => this.retrievePayload(claim, signal)),
205-
);
206-
}
207-
208-
private async storePayload(
209-
payload: Payload,
210-
keyPrefix: string,
211-
abortSignal: AbortSignal,
212-
): Promise<StorageDriverClaim> {
213-
// Store the encoded Payload proto, not just `payload.data`. A Payload also carries
214-
// metadata (the `encoding` key, protobuf message names, anything a custom
215-
// PayloadConverter added), and metadata values are arbitrary bytes that would not
216-
// survive a round trip through the string-valued claim map. Encoding the whole
217-
// message keeps the payload byte-for-byte identical end to end.
218-
const payloadBytes = PayloadProto.encode(payload).finish();
219-
if (payloadBytes.length > this.maxPayloadSize) {
220-
throw new Error(
221-
`Payload of ${payloadBytes.length} bytes exceeds the configured maxPayloadSize of ${this.maxPayloadSize} bytes`,
222-
);
223-
}
224-
225-
const hashValue = createHash(HASH_ALGORITHM).update(payloadBytes).digest('hex');
226-
const key = `${keyPrefix}/${HASH_ALGORITHM}/${hashValue}`;
227-
228-
try {
229-
await this.writeIfAbsent(key, payloadBytes, abortSignal);
230-
} catch (err) {
231-
// Wrapping adds the context that makes a retry loop diagnosable: a bare ENOENT
232-
// says nothing about which key or which storage root. On ES2022 and above, prefer
233-
// `new Error(message, { cause: err })` to keep the original stack attached; these
234-
// samples target ES2021, so the message is folded in instead.
235-
throw new Error(
236-
`FileSystemStorageDriver failed to store [rootDir=${this.rootDir}, key=${key}]: ${describe(err)}`,
237-
);
238-
}
239-
240-
// The claim is the only thing that reaches Temporal Server, embedded in the
241-
// reference payload that replaces the real one. Keep it small and keep it free of
242-
// anything sensitive: it is visible in Workflow History and in the Web UI.
243-
//
244-
// `rootDir` is deliberately absent. It is deployment configuration, and a second
245-
// Worker may well mount the same storage at a different path; putting it in the
246-
// claim would pin every historical reference to one machine's filesystem layout.
247-
return new StorageDriverClaim({ key, hashAlgorithm: HASH_ALGORITHM, hashValue });
248-
}
249-
250-
/**
251-
* Writes the blob unless it is already there. The write goes to a temporary file and
252-
* is then renamed, because `rename` is atomic: a reader can only ever observe the
253-
* complete blob, never a half-written one. Without that, a crash mid-write would
254-
* leave a file whose name promises content it does not contain, and content
255-
* addressing would hand it to a reader as valid.
256-
*/
257-
private async writeIfAbsent(key: string, payloadBytes: Uint8Array, abortSignal: AbortSignal): Promise<void> {
258-
const filePath = this.resolveKey(key);
259-
if (await exists(filePath)) return;
260-
261-
await mkdir(path.dirname(filePath), { recursive: true });
262-
const tempPath = `${filePath}.${randomUUID()}.tmp`;
263-
try {
264-
await writeFile(tempPath, payloadBytes, { signal: abortSignal });
265-
await rename(tempPath, filePath);
266-
} catch (err) {
267-
await unlink(tempPath).catch(() => undefined);
268-
// Two Workers can race to store identical bytes. On POSIX the loser's rename
269-
// silently replaces an identical file; on Windows it fails. Either way the blob
270-
// is present and correct, so treat that as success.
271-
if (await exists(filePath)) return;
272-
throw err;
273-
}
274-
}
275-
276-
private async retrievePayload(claim: StorageDriverClaim, abortSignal: AbortSignal): Promise<Payload> {
277-
const { key, hashAlgorithm, hashValue: expectedHash } = claim.claimData;
278-
// Claims come off the wire, so validate rather than assume. A missing field means a
279-
// claim written by a different driver, or by an older version of this one.
280-
if (!key) {
281-
throw new Error("FileSystemStorageDriver claim is missing required field 'key'");
282-
}
283-
if (hashAlgorithm !== HASH_ALGORITHM || !expectedHash) {
284-
throw new Error(
285-
`FileSystemStorageDriver claim [key=${key}] must carry hashAlgorithm='${HASH_ALGORITHM}' and a hashValue, ` +
286-
`got hashAlgorithm='${hashAlgorithm ?? ''}'`,
287-
);
288-
}
289-
290-
const filePath = this.resolveKey(key);
291-
let payloadBytes: Uint8Array;
292-
try {
293-
payloadBytes = await readFile(filePath, { signal: abortSignal });
294-
} catch (err) {
295-
throw new Error(
296-
`FileSystemStorageDriver failed to retrieve [rootDir=${this.rootDir}, key=${key}]: ${describe(err)}`,
297-
);
298-
}
299-
300-
// Verifying is cheap next to the read and catches truncation, corruption, and a
301-
// claim pointing at the wrong blob. Failing loudly here is much better than
302-
// handing malformed bytes to the PayloadConverter, where the error would surface
303-
// as a confusing deserialization failure far from its cause.
304-
const actualHash = createHash(HASH_ALGORITHM).update(payloadBytes).digest('hex');
305-
if (actualHash !== expectedHash) {
306-
throw new Error(
307-
`FileSystemStorageDriver integrity check failed [key=${key}]: ` +
308-
`expected ${HASH_ALGORITHM}:${expectedHash}, got ${HASH_ALGORITHM}:${actualHash}`,
309-
);
310-
}
311-
312-
return PayloadProto.decode(payloadBytes);
313-
}
314-
315-
/**
316-
* Maps a key to a path under `rootDir`, refusing anything that escapes it. Keys
317-
* arrive from Workflow History, which we should not treat as trusted input: a claim
318-
* containing `../../etc/passwd` must not turn into a read outside the blob store.
319-
*/
320-
private resolveKey(key: string): string {
321-
const filePath = path.resolve(this.rootDir, key);
322-
if (filePath !== this.rootDir && !filePath.startsWith(this.rootDir + path.sep)) {
323-
throw new Error(`FileSystemStorageDriver refused a key that resolves outside rootDir [key=${key}]`);
324-
}
325-
return filePath;
326-
}
165+
async store(context: StorageDriverStoreContext, payloads: Payload[]): Promise<StorageDriverClaim[]> {
166+
const keyPrefix = buildKeyPrefix(context.target);
167+
return runAllAbortingOnFirstError(context.abortSignal, (signal) =>
168+
payloads.map((payload) => this.storePayload(payload, keyPrefix, signal)),
169+
);
170+
}
171+
172+
/** Inverse of {@link store}: one payload per claim, in the same order. */
173+
async retrieve(context: StorageDriverRetrieveContext, claims: StorageDriverClaim[]): Promise<Payload[]> {
174+
return runAllAbortingOnFirstError(context.abortSignal, (signal) =>
175+
claims.map((claim) => this.retrievePayload(claim, signal)),
176+
);
327177
}
328178
```
329179
<!--SNIPEND-->
330180

181+
The per-payload work happens in the private `storePayload` and `retrievePayload` methods. Read
182+
[the full driver](https://github.com/temporalio/samples-typescript/blob/main/external-storage/src/filesystem-storage-driver.ts)
183+
for the parts this page only describes: content-addressed keys, an atomic write, a hash check on read, and a guard that
184+
rejects claims resolving outside the storage root.
185+
186+
A shared filesystem works for local development and for Workers that mount the same volume. For anything else, use a
187+
storage system that every Client and Worker can reach.
188+
331189
The following sections walk through the key parts of the driver implementation.
332190

333191
### 1. Implement the StorageDriver interface
@@ -340,8 +198,7 @@ A custom driver implements the `StorageDriver` interface, which has two readonly
340198
- `type` is a string that identifies the driver implementation, and the Worker reports it in its heartbeat. Unlike
341199
`name`, `type` must be the same across all instances of the same driver type regardless of configuration. Two S3
342200
drivers named `"s3-primary"` and `"s3-archive"` would both report `"aws.s3driver"` as their type, while the built-in
343-
GCS driver reports `"gcp.gcsdriver"`. The filesystem driver in the preceding code sample reports
344-
`"sample.filesystemdriver"`.
201+
GCS driver reports `"gcp.gcsdriver"`. The filesystem driver in the sample reports `"sample.filesystemdriver"`.
345202
- `store()` receives an array of payloads and returns one `StorageDriverClaim` per payload. A claim wraps a set of
346203
string key-value pairs that the driver uses to locate the payload later.
347204
- `retrieve()` receives the claims that `store()` produced and returns the original payloads.

0 commit comments

Comments
 (0)