Add guide: Migrate a Celery task queue to a Temporal Workflow - #5116
Draft
brianmacdonald-temporal wants to merge 4 commits into
Draft
Add guide: Migrate a Celery task queue to a Temporal Workflow#5116brianmacdonald-temporal wants to merge 4 commits into
brianmacdonald-temporal wants to merge 4 commits into
Conversation
Add docs/guides/celery-to-workflow.mdx, a Python guide that walks through converting a Celery application to Temporal one piece at a time: mapping Celery concepts to Temporal primitives, turning a task into an Activity, orchestrating it with a Workflow, running a Worker, replacing .delay() with client.start_workflow(), migrating task retries to a Retry Policy, replacing Celery Beat with a Temporal Schedule, and translating Canvas workflows. Link the page from the Guides sidebar and add a card to the Guides grid under a new "Migration" tag. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a Python guide for migrating Celery tasks, retries, schedules, and Canvas pipelines to Temporal.
Changes:
- Adds the migration guide and Python examples.
- Adds the guide to navigation.
- Adds a Migration card to the Guides grid.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 9 comments.
| File | Description |
|---|---|
docs/guides/celery-to-workflow.mdx |
Adds the migration tutorial. |
sidebars.js |
Adds the guide to the sidebar. |
src/components/GuidesGrid/guides-data.json |
Adds the guide card and Migration tag. |
Suppressed comments (3)
docs/guides/celery-to-workflow.mdx:100
- Capitalize “Temporal Service,” which is the required proper noun for the Temporal backend (
readme/STYLE.md:24-31).
You will see the installed version printed to your terminal. With the tools installed, you can start a local Temporal service.
docs/guides/celery-to-workflow.mdx:104
- Capitalize “Temporal Service,” which is the required proper noun for the Temporal backend (
readme/STYLE.md:24-31).
In Celery, work flows through a broker such as Redis. In Temporal, work flows through the Temporal service, which also stores each Workflow's durable history. In this step, you will start a local development server that stands in for that service.
docs/guides/celery-to-workflow.mdx:539
- Capitalize “Temporal Service,” which is the required proper noun for the Temporal backend (
readme/STYLE.md:24-31).
- Self-hosting the Temporal service or using [Temporal Cloud](https://docs.temporal.io/) instead of the development
server.
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| description: Migrate Celery tasks to Temporal Workflows and Activities. | ||
| sidebar_label: Migrate from Celery | ||
| toc_max_heading_level: 3 | ||
| author: n/a |
|
|
||
| [Celery](https://docs.celeryq.dev/en/stable/) is a distributed task queue that runs background jobs by pushing messages through a broker (such as Redis or RabbitMQ) to a pool of worker processes. It is a popular choice for sending emails, processing uploads, and running scheduled jobs. As applications grow, however, teams often want stronger guarantees than a broker provides: automatic recovery after a crash, a durable record of every job's progress, and built-in retries. | ||
|
|
||
| Temporal provides those guarantees. Instead of enqueueing a message and waiting for a worker to finish it, you run a _Workflow_ whose entire state is persisted by the Temporal service. If a Worker crashes mid-job, another Worker resumes exactly where it left off. |
| Result: sent to user@example.com | ||
| ``` | ||
|
|
||
| Note how the Celery patterns translate. A fire-and-forget `.delay()` corresponds to `start_workflow`, which returns a handle immediately. Retrieving the return value with `AsyncResult.get()` corresponds to `handle.result()`. The `id` you provide is a business identifier you choose (an order number, a user ID); Temporal uses it to guarantee that the same Workflow is never started twice, which is a built-in form of deduplication. |
|
|
||
| In Celery, retries are your responsibility: you set `max_retries` and call `self.retry()` inside the task. In Temporal, retries are automatic and declarative. In this step, you will restore your task's retry behavior by attaching a retry policy to the Activity call. | ||
|
|
||
| By default, Temporal retries a failed Activity indefinitely with exponential backoff. To reproduce the Celery task's limit of five attempts, update the `execute_activity` call in `workflows/onboarding.py`: |
| ) | ||
| ``` | ||
|
|
||
| Here, `maximum_attempts=5` mirrors Celery's `max_retries`, and `maximum_interval` caps the backoff between attempts. The `non_retryable_error_types` list names errors that should fail immediately without retrying — the equivalent of _not_ calling `self.retry()` for a permanent failure. To raise such an error from an Activity, use `ApplicationError` with `non_retryable=True`: |
Comment on lines
+338
to
+342
| retry_policy=RetryPolicy( | ||
| maximum_attempts=5, | ||
| maximum_interval=timedelta(minutes=1), | ||
| non_retryable_error_types=["InvalidUserError"], | ||
| ), |
Contributor
📖 Docs PR preview links
|
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
brianmacdonald-temporal
marked this pull request as draft
August 14, 2026 20:39
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Adds a new Python guide, Migrate a Celery Task Queue to a Temporal Workflow, that walks through converting a Celery application to Temporal one piece at a time:
.delay()withclient.start_workflow()max_retries/self.retry()to a Retry Policychain/group/chordpipelinesAlso links the page from the Guides sidebar and adds a card to the Guides grid under a new
Migrationtag.Notes to reviewers
yarn buildpasses.vale --config .vale-ci.ini docs/guides/celery-to-workflow.mdxreports 0 errors and 0 warnings. Three heading suggestions remain, all false positives on product names the rule does not know: "Celery task", "Celery Beat", and "Canvas workflows".Migrationis a new tag in the Guides grid — flagging in case you would rather reuse an existing one.🤖 Generated with Claude Code
┆Attachments: EDU-6960 Add guide: Migrate a Celery task queue to a Temporal Workflow