|
| 1 | +--- |
| 2 | +id: batch-iterator |
| 3 | +title: "Batch Iterator" |
| 4 | +sidebar_label: "Batch Iterator" |
| 5 | +description: "Pages through unbounded datasets using Continue-As-New to prevent history overflow while maintaining exactly-once processing guarantees." |
| 6 | +--- |
| 7 | + |
| 8 | +import Tabs from '@theme/Tabs'; |
| 9 | +import TabItem from '@theme/TabItem'; |
| 10 | + |
| 11 | +:::info[TLDR] |
| 12 | +**Process one page at a time** and call Continue-as-New with the next offset after each page so the Workflow's event history never grows without bound. With this method you can process infinite pages. Use this when your record set is arbitrarily large, you need a durable checkpoint after every page, and sequential page-by-page throughput is acceptable. |
| 13 | +::: |
| 14 | + |
| 15 | +## Overview |
| 16 | + |
| 17 | +The Batch Iterator pattern processes a large record set one page at a time. Each Workflow run processes a single page and then calls Continue-as-New with the next offset, producing a chain of short-lived runs that together cover the entire record set without accumulating unbounded event history. |
| 18 | + |
| 19 | +## Problem |
| 20 | + |
| 21 | +A single Workflow run is limited to 50,000 history events (aim for 2,000) and 2,000 in-flight Activities. Processing millions of records in one run is not possible within these bounds. |
| 22 | + |
| 23 | +You need a way to process an arbitrarily large record set reliably, with the ability to resume from a checkpoint if the Workflow is interrupted, and without overwhelming downstream systems with a burst of concurrent requests. |
| 24 | + |
| 25 | +## Solution |
| 26 | + |
| 27 | +Each Workflow run fetches one page of records using a persistent `offset` parameter, processes each record sequentially, and then calls `continueAsNew` with the incremented offset. The next run picks up exactly where the previous one left off. |
| 28 | + |
| 29 | +Because each run processes only a bounded number of records, history stays well within limits. The offset acts as a durable checkpoint: if the Workflow is interrupted mid-page, the next run replays only from the start of the current page. |
| 30 | + |
| 31 | +```mermaid |
| 32 | +flowchart TD |
| 33 | + DB[("Data Source\n(paginated)")] |
| 34 | + WF1["Workflow Run 1\n(offset=0)"] |
| 35 | + WF2["Workflow Run 2\n(offset=PAGE_SIZE)"] |
| 36 | + WF3["Workflow Run N\n(offset=N×PAGE_SIZE)"] |
| 37 | + Done(["Complete"]) |
| 38 | +
|
| 39 | + DB -->|"fetch page 1"| WF1 |
| 40 | + WF1 -->|"processRecord ×PAGE_SIZE"| Acts1["Activities"] |
| 41 | + WF1 -->|"continueAsNew\n(offset=PAGE_SIZE)"| WF2 |
| 42 | +
|
| 43 | + DB -->|"fetch page 2"| WF2 |
| 44 | + WF2 -->|"processRecord ×PAGE_SIZE"| Acts2["Activities"] |
| 45 | + WF2 -->|"continueAsNew\n(offset=N×PAGE_SIZE)"| WF3 |
| 46 | +
|
| 47 | + DB -->|"fetch page N"| WF3 |
| 48 | + WF3 -->|"processRecord ×PAGE_SIZE"| Acts3["Activities"] |
| 49 | + WF3 -->|"last page → return"| Done |
| 50 | +``` |
| 51 | + |
| 52 | +The following describes each step in the diagram: |
| 53 | + |
| 54 | +1. The Workflow starts with `offset=0` and calls `fetchPage(offset, pageSize)` to retrieve the first page of records. |
| 55 | +2. It processes each record in the page by executing the `processRecord` Activity. |
| 56 | +3. After the page is fully processed, it calls `continueAsNew` with `offset + pageSize`, passing the updated offset to the next run. |
| 57 | +4. The next run begins with a clean history and repeats the same steps for the next page. |
| 58 | +5. When `fetchPage` returns fewer records than `pageSize`, the Workflow knows it has reached the last page and returns normally. |
| 59 | + |
| 60 | +## Implementation |
| 61 | + |
| 62 | +The following examples show how each SDK implements the Batch Iterator pattern. |
| 63 | + |
| 64 | +<Tabs groupId="language" queryString> |
| 65 | +<TabItem value="typescript" label="TypeScript"> |
| 66 | + |
| 67 | +```typescript |
| 68 | +// workflows.ts |
| 69 | +import { continueAsNew, log, proxyActivities } from "@temporalio/workflow"; |
| 70 | +import type * as activities from "./activities"; |
| 71 | +import { PAGE_SIZE } from "./shared"; |
| 72 | + |
| 73 | +const { fetchPage, processRecord } = proxyActivities<typeof activities>({ |
| 74 | + startToCloseTimeout: "10 seconds", |
| 75 | +}); |
| 76 | + |
| 77 | +export async function batchIteratorWorkflow( |
| 78 | + offset: number = 0, |
| 79 | + totalProcessed: number = 0 |
| 80 | +): Promise<number> { |
| 81 | + const page = await fetchPage(offset, PAGE_SIZE); |
| 82 | + |
| 83 | + for (const record of page) { |
| 84 | + await processRecord(record); |
| 85 | + totalProcessed++; |
| 86 | + } |
| 87 | + |
| 88 | + log.info(`Processed page at offset ${offset} (${page.length} records, running total: ${totalProcessed})`); |
| 89 | + |
| 90 | + if (page.length === PAGE_SIZE) { |
| 91 | + await continueAsNew<typeof batchIteratorWorkflow>(offset + PAGE_SIZE, totalProcessed); |
| 92 | + } |
| 93 | + |
| 94 | + return totalProcessed; |
| 95 | +} |
| 96 | +``` |
| 97 | + |
| 98 | +</TabItem> |
| 99 | +<TabItem value="python" label="Python"> |
| 100 | + |
| 101 | +```python |
| 102 | +# workflows.py |
| 103 | +from temporalio import workflow |
| 104 | +from temporalio.workflow import continue_as_new |
| 105 | +from datetime import timedelta |
| 106 | +from activities import fetch_page, process_record |
| 107 | +from shared import PAGE_SIZE |
| 108 | + |
| 109 | + |
| 110 | +@workflow.defn |
| 111 | +class BatchIteratorWorkflow: |
| 112 | + @workflow.run |
| 113 | + async def run(self, offset: int = 0, total_processed: int = 0) -> int: |
| 114 | + page = await workflow.execute_activity( |
| 115 | + fetch_page, |
| 116 | + args=[offset, PAGE_SIZE], |
| 117 | + start_to_close_timeout=timedelta(seconds=10), |
| 118 | + ) |
| 119 | + |
| 120 | + for record in page: |
| 121 | + await workflow.execute_activity( |
| 122 | + process_record, |
| 123 | + record, |
| 124 | + start_to_close_timeout=timedelta(seconds=10), |
| 125 | + ) |
| 126 | + total_processed += 1 |
| 127 | + |
| 128 | + workflow.logger.info( |
| 129 | + f"Processed page at offset {offset} ({len(page)} records, running total: {total_processed})" |
| 130 | + ) |
| 131 | + |
| 132 | + if len(page) == PAGE_SIZE: |
| 133 | + continue_as_new(args=[offset + PAGE_SIZE, total_processed]) |
| 134 | + |
| 135 | + return total_processed |
| 136 | +``` |
| 137 | + |
| 138 | +</TabItem> |
| 139 | +<TabItem value="go" label="Go"> |
| 140 | + |
| 141 | +```go |
| 142 | +// workflows.go |
| 143 | +package main |
| 144 | + |
| 145 | +import ( |
| 146 | + "time" |
| 147 | + |
| 148 | + "go.temporal.io/sdk/workflow" |
| 149 | +) |
| 150 | + |
| 151 | +func BatchIteratorWorkflow(ctx workflow.Context, offset int, totalProcessed int) (int, error) { |
| 152 | + ao := workflow.ActivityOptions{ |
| 153 | + StartToCloseTimeout: 10 * time.Second, |
| 154 | + } |
| 155 | + ctx = workflow.WithActivityOptions(ctx, ao) |
| 156 | + |
| 157 | + var page []Record |
| 158 | + if err := workflow.ExecuteActivity(ctx, FetchPage, offset, PageSize).Get(ctx, &page); err != nil { |
| 159 | + return totalProcessed, err |
| 160 | + } |
| 161 | + |
| 162 | + for _, record := range page { |
| 163 | + if err := workflow.ExecuteActivity(ctx, ProcessRecord, record).Get(ctx, nil); err != nil { |
| 164 | + return totalProcessed, err |
| 165 | + } |
| 166 | + totalProcessed++ |
| 167 | + } |
| 168 | + |
| 169 | + workflow.GetLogger(ctx).Info("Processed page", |
| 170 | + "offset", offset, |
| 171 | + "pageSize", len(page), |
| 172 | + "totalProcessed", totalProcessed) |
| 173 | + |
| 174 | + if len(page) == PageSize { |
| 175 | + return totalProcessed, workflow.NewContinueAsNewError(ctx, BatchIteratorWorkflow, offset+PageSize, totalProcessed) |
| 176 | + } |
| 177 | + |
| 178 | + return totalProcessed, nil |
| 179 | +} |
| 180 | +``` |
| 181 | + |
| 182 | +</TabItem> |
| 183 | +<TabItem value="java" label="Java"> |
| 184 | + |
| 185 | +```java |
| 186 | +// BatchIteratorWorkflow.java |
| 187 | +import io.temporal.activity.ActivityOptions; |
| 188 | +import io.temporal.workflow.*; |
| 189 | +import java.time.Duration; |
| 190 | +import java.util.List; |
| 191 | + |
| 192 | +@WorkflowInterface |
| 193 | +public interface BatchIteratorWorkflow { |
| 194 | + @WorkflowMethod |
| 195 | + int run(int offset, int totalProcessed); |
| 196 | +} |
| 197 | + |
| 198 | +// BatchIteratorWorkflowImpl.java |
| 199 | +public class BatchIteratorWorkflowImpl implements BatchIteratorWorkflow { |
| 200 | + private final Activities activities = Workflow.newActivityStub( |
| 201 | + Activities.class, |
| 202 | + ActivityOptions.newBuilder() |
| 203 | + .setStartToCloseTimeout(Duration.ofSeconds(10)) |
| 204 | + .build() |
| 205 | + ); |
| 206 | + |
| 207 | + @Override |
| 208 | + public int run(int offset, int totalProcessed) { |
| 209 | + List<Record> page = activities.fetchPage(offset, Shared.PAGE_SIZE); |
| 210 | + |
| 211 | + for (Record record : page) { |
| 212 | + activities.processRecord(record); |
| 213 | + totalProcessed++; |
| 214 | + } |
| 215 | + |
| 216 | + Workflow.getLogger(BatchIteratorWorkflowImpl.class).info( |
| 217 | + "Processed page at offset " + offset + " (" + page.size() + " records, total: " + totalProcessed + ")" |
| 218 | + ); |
| 219 | + |
| 220 | + if (page.size() == Shared.PAGE_SIZE) { |
| 221 | + BatchIteratorWorkflow next = Workflow.newContinueAsNewStub(BatchIteratorWorkflow.class); |
| 222 | + next.run(offset + Shared.PAGE_SIZE, totalProcessed); |
| 223 | + } |
| 224 | + |
| 225 | + return totalProcessed; |
| 226 | + } |
| 227 | +} |
| 228 | +``` |
| 229 | + |
| 230 | +</TabItem> |
| 231 | +</Tabs> |
| 232 | + |
| 233 | +## Best practices |
| 234 | + |
| 235 | +- **Choose a page size that keeps history under 2,000 events.** Each page produces roughly `3 × pageSize` history events (`ActivityTaskScheduled` + `ActivityTaskStarted` + `ActivityTaskCompleted`). A page size of 500–800 records is a safe target. |
| 236 | +- **Include `totalProcessed` (or a similar counter) in the `continueAsNew` args.** This lets you observe overall progress via the Workflow input visible in the UI without querying internal state. |
| 237 | +- **Fetch inside an Activity, not the Workflow.** The `fetchPage` call must be an Activity — not inline Workflow code — so it can interact with external systems and be retried independently. |
| 238 | +- **Make `processRecord` idempotent.** Activities have at-least-once execution semantics. If a worker crashes after an Activity completes externally but before the completion is recorded in history, Temporal will retry it. Your downstream system must tolerate receiving the same record more than once. |
| 239 | +- **Avoid accumulating large local state between pages.** `continueAsNew` does not carry over in-memory state; only the arguments you pass are available in the next run. |
| 240 | + |
| 241 | +## Common pitfalls |
| 242 | + |
| 243 | +- **Forgetting `continueAsNew` on the last page.** If you call `continueAsNew` unconditionally, the Workflow loops forever even when the data source is exhausted. Check whether the returned page is shorter than `pageSize` before continuing. |
| 244 | +- **Passing unnecessary state into `continueAsNew`.** All arguments are serialized and stored in history. Pass only the minimal state needed (offset, counters) — not accumulated result lists or large collections that grow with each page. |
| 245 | +- **Sequential processing bottlenecks.** The default implementation processes one record at a time per page. You can fan out Activities concurrently within a page using the SDK's async primitives for higher per-page throughput — note this increases per-page event count accordingly. If record-set-wide throughput matters more than rate limiting, consider [Sliding Window](/design-patterns/sliding-window) or [MapReduce Tree](/design-patterns/mapreduce-tree). |
| 246 | + |
| 247 | +## Related resources |
| 248 | + |
| 249 | +- [Continue-as-New pattern](/design-patterns/continue-as-new) — core concepts for history management via `continueAsNew` |
| 250 | +- [Sliding Window](/design-patterns/sliding-window) — bounded concurrency that progresses at the rate of the fastest processor |
| 251 | +- [MapReduce Tree](/design-patterns/mapreduce-tree) — fully parallel processing for maximum speed |
| 252 | +- [Temporal limits reference](/cloud/limits) |
| 253 | +- [Batch samples (Java)](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/batch/iterator) |
0 commit comments