docs(platform): Add cloud data worker capacity enforcement documentation#75955
docs(platform): Add cloud data worker capacity enforcement documentation#75955Ian Alton (ian-at-airbyte) wants to merge 12 commits intomasterfrom
Conversation
- Add Queued connection status and queued connections section to connection status page - Add on-demand capacity section and workspace usage to manage-data-workers hub page - Add on-demand capacity credits section and billing notification updates to manage-credits - Add Queued syncs notification event and webhook payload to notifications page - Add Use on-demand capacity setting to connection configuration table - Add Queued status to connection timeline and cancellation context - Add System tags section with Burst tag documentation to tagging page - Add capacity queuing considerations to sync schedules - Add QUEUED state to job state machine diagram and explanation to jobs page - Add Pro and Enterprise Flex plan considerations to cloud limits page - Add capacity note to set-up-a-connection page Resolves airbytehq/airbyte-internal-issues#16129 Co-Authored-By: ian.alton@airbyte.io <ian.alton@airbyte.io>
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
👋 Greetings, Airbyte Team Member!Here are some helpful tips and reminders for your convenience. 💡 Show Tips and TricksPR Slash CommandsAirbyte Maintainers (that's you!) can execute the following slash commands on your PR:
📚 Show Repo GuidanceHelpful Resources
|
Remove blank line between Error and Action Required rows that would break Markdown table rendering. Co-Authored-By: ian.alton@airbyte.io <ian.alton@airbyte.io>
|
Deploy preview for airbyte-docs ready! ✅ Preview Built with commit 6d1c298. |
Review: Cross-referenced against platform-internal PRsI reviewed this PR against the merged implementation PRs in Show/Hide Detailed Review1. On-demand toggle exists during connection creation, not just Settings
2. Workspace usage graph replaces credit usage, not additive
3. Connection QUEUED maps to stream status PENDING — disambiguation note should mention thisThe stream-level disambiguation note in 4. Webhook payload for "Queued syncs" — verify backend actually sends thisThe webhook payload JSON in 5. Capacity banner is feature-flag gated — docs should note the prerequisite
6. "Queued syncs" notification is email-only — table shows webhook supportThe notification table adds 7. Missing:
|
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
docs/platform/cloud/managing-airbyte-cloud/manage-data-workers.md
Outdated
Show resolved
Hide resolved
docs/platform/cloud/managing-airbyte-cloud/manage-airbyte-cloud-notifications.md
Outdated
Show resolved
Hide resolved
|
Devin can you take a look at these? |
- manage-data-workers.md: Clarify workspace usage replaces credit usage for entitled orgs; add note that on-demand toggle appears during connection creation; note banner requires on-demand capacity contract - review-connection-status.md: Add stream Pending mapping to disambiguation note (streams show Pending when connection is Queued) - manage-airbyte-cloud-notifications.md: Remove webhook payload for Queued syncs (email-only event; reconciles table vs payload mismatch) - configuring-connections.md: Note toggle also appears during connection creation - set-up-a-connection.md: Revert inline capacity note to keep bullet list clean (capacity details live in manage-data-workers.md hub page) Co-Authored-By: ian.alton@airbyte.io <ian.alton@airbyte.io>
|
I've addressed the review comments in commit a594d8e. Here's what I changed and what still needs your verification: Fixed:
Needs your verification (Ian Alton (@ian-at-airbyte)):
|
| By default, all customers automatically review upcoming invoice notifications 3 and 7 days before the invoice is finalized. All billing notifications are sent to the billing email on file. | ||
|
|
||
| You can also enroll in billing notifications for your organization. Airbyte highly recommends enrolling in billing notifications to ensure you stay up-to-date on your upcoming invoices. | ||
| You can also enroll in billing notifications for your organization. Airbyte highly recommends enrolling in billing notifications to ensure you stay up-to-date on your upcoming invoices. If you use on-demand capacity, consider setting up billing alerts to monitor premium credit consumption. For more information about managing capacity, see [Monitor data worker usage](./manage-data-workers.md). |
There was a problem hiding this comment.
Is that even an option? I thought we just sent an email when syncs were queued.
There was a problem hiding this comment.
You're right — I removed that sentence in 5f33de1. The billing notifications paragraph is now back to its original text without the on-demand capacity suggestion.
…city Per reviewer feedback, there is no specific billing alert feature for on-demand credit consumption. Reverted the billing notifications paragraph to its original text. Co-Authored-By: ian.alton@airbyte.io <ian.alton@airbyte.io>
Code investigation of ConnectionManagerWorkflowImpl.kt shows there is no explicit FIFO queue ordering. Each connection's workflow independently polls for capacity every minute, and whichever poll runs first when capacity frees up gets the reservation. Removed 'run in the order they were triggered' claims from review-connection-status.md and manage-data-workers.md. Co-Authored-By: ian.alton@airbyte.io <ian.alton@airbyte.io>
- Replace 'Queued icon' placeholder in connection status table with actual hourglass icon (connection_queued.png) - Add Burst tag image to tagging.md system tags section - Add inline Burst tag image to manage-data-workers.md on-demand section Co-Authored-By: ian.alton@airbyte.io <ian.alton@airbyte.io>
Code-Verified Eval: Technical Accuracy ReviewI cross-referenced every technical claim in this PR against the actual implementation in Show/Hide Detailed FindingsFinding 1: "Queued syncs" notification supports BOTH email AND webhook in the UI — docs say email-onlySeverity: Medium — docs contradict the UI The docs say "Queued syncs notifications are email-only. Webhook payloads are not supported for this event type." However, the UI code tells a different story: In {isOnDemandCapacityEnabled && <NotificationItemField name="sendOnConnectionSyncQueued" />}Note that it does not pass <NotificationItemField name="sendOnBreakingChangeWarning" slackNotificationUnsupported />Without However, the backend trigger type
Recommendation: Investigate whether the backend actually fires notifications for this trigger. The docs should either:
Code locations:
Finding 2: Capacity banner is NOT gated on
|
docs/platform/cloud/managing-airbyte-cloud/manage-airbyte-cloud-notifications.md
Outdated
Show resolved
Hide resolved
…review - Remove 'If your organization has on-demand capacity in its contract' qualifier from banner description. Code shows CapacityReachedMessage renders for any workspace with queuedCount > 0 (no feature flag check). - Update queued syncs notification payload section: the UI allows both email and webhook configuration (no slackNotificationUnsupported prop). Note that no example payload is available yet. Co-Authored-By: ian.alton@airbyte.io <ian.alton@airbyte.io>
|
Addressed the three actionable findings from the eval review in ff96308: Finding 1 (Notification channel mismatch) — Fixed: Verified that Finding 2 (Banner gating) — Fixed: Removed "If your organization has on-demand capacity in its contract" from the banner description. Finding 3 (Feature flag vs plan gating) — Needs decision: The docs say "On capacity-based Cloud plans..." but the code shows enforcement is gated behind |
|
You can assume the back end will be completed. Don't account for rollout in the docs. Operate as if everyone will be getting this immediately. |
|
Got it — leaving the docs as "On capacity-based Cloud plans..." with no rollout caveat. Finding 3 is resolved, no changes needed. For Finding 1: the notification payload section now says both email and webhook are supported. Since we're assuming the backend will be completed, no further caveats needed there either. All three eval findings are addressed in ff96308. |
|
Devin Ian Alton (@ian-at-airbyte) IMO we should also mention that enabling “Run on demand” is only available to admins. A couple of related PRs were recently merged: |
Per intents.yaml, ManageConnectionOnDemandCapacity is restricted to ORGANIZATION_ADMIN, WORKSPACE_ADMIN, and ADMIN roles. The UI shows a disabled toggle with tooltip 'Only admins can enable or disable on-demand capacity' for other roles. Updated manage-data-workers.md and configuring-connections.md to document this restriction. Co-Authored-By: ian.alton@airbyte.io <ian.alton@airbyte.io>
|
Good catch! I verified this against the code in
Updated in adb0d6d:
|
| | [Destination Namespace](../../using-airbyte/core-concepts/namespaces) | Determines where the replicated data is written to in the destination | | ||
| | [Destination Stream Prefix](../../using-airbyte/configuring-schema) | (Optional) Adds a prefix to each table name in the destination | | ||
| | [Detect and propagate schema changes](../../using-airbyte/schema-change-management) | Set how Airbyte handles schema changes in the source | | ||
| | Use on-demand capacity (Cloud only) | When enabled, syncs always run even if committed capacity is exhausted, by consuming credits at a premium rate. Only organization admins and workspace admins can change this toggle. It also appears during connection creation. See [Monitor data worker usage](./manage-data-workers.md). | |
There was a problem hiding this comment.
Good catch — updated in 369ee3b. Changed to "Cloud Pro and Enterprise Flex only" here and across all other capacity-related labels (review-connection-status.md, review-connection-timeline.md, tagging.md, jobs.md).
|
|
||
| 4. On the **Status** tab for a connection, there is a list of associated Streams. To the left of the name for each Stream, there is an icon that displays its status. | ||
|
|
||
| ### Queued connections (Cloud only) |
There was a problem hiding this comment.
Good call — updated in 369ee3b. Changed from "Cloud only" to "Cloud Pro and Enterprise Flex only" to be more precise about which plans have capacity enforcement. Updated the table row description and the subsection heading.
…or capacity features Per reviewer feedback, capacity enforcement and on-demand capacity are specific to Pro and Enterprise Flex plans, not all Cloud plans. Updated parenthetical labels across all affected files. Co-Authored-By: ian.alton@airbyte.io <ian.alton@airbyte.io>
|
In Connection / Settings/ the text under Use-on-demand-capacity. Let's change it to: Enable on demand capacity for this connection. Syncs for this connection will never be queued. Syncs that run when committed data worker is exhausted will be charged a premium rate. |
Updated the quoted toggle description text to match the new UI copy: 'Enable on demand capacity for this connection. Syncs for this connection will never be queued. Syncs that run when committed data worker is exhausted will be charged a premium rate.' Co-Authored-By: ian.alton@airbyte.io <ian.alton@airbyte.io>
|
Updated in 01f4064 — the toggle description now reads: "Enable on demand capacity for this connection. Syncs for this connection will never be queued. Syncs that run when committed data worker is exhausted will be charged a premium rate." |
|
|
||
| ## On-demand capacity credits | ||
|
|
||
| If your organization uses [on-demand capacity](./manage-data-workers.md#on-demand-capacity), syncs that run when committed data worker capacity is exhausted consume organization credits at a premium rate. The premium rate is determined by your contract. Contact your Airbyte representative or [talk to sales](https://airbyte.com/company/talk-to-sales) for details. |
There was a problem hiding this comment.
| If your organization uses [on-demand capacity](./manage-data-workers.md#on-demand-capacity), syncs that run when committed data worker capacity is exhausted consume organization credits at a premium rate. The premium rate is determined by your contract. Contact your Airbyte representative or [talk to sales](https://airbyte.com/company/talk-to-sales) for details. | |
| If your organization uses [on-demand capacity](./manage-data-workers.md#on-demand-capacity), syncs that run when committed data worker capacity is exhausted consume organization credits at a premium rate. |
There was a problem hiding this comment.
Applied in 78b49dc — removed the contract/sales sentence, keeping just the one-liner about premium rate.
|
|
||
| If your organization uses [on-demand capacity](./manage-data-workers.md#on-demand-capacity), syncs that run when committed data worker capacity is exhausted consume organization credits at a premium rate. The premium rate is determined by your contract. Contact your Airbyte representative or [talk to sales](https://airbyte.com/company/talk-to-sales) for details. | ||
|
|
||
| On-demand credit consumption appears in your billing alongside other credit usage. If your organization runs out of credits, on-demand syncs are queued like standard syncs. Queuing does not incur any charges. Credits are only consumed when a sync runs using on-demand capacity. |
There was a problem hiding this comment.
| On-demand credit consumption appears in your billing alongside other credit usage. If your organization runs out of credits, on-demand syncs are queued like standard syncs. Queuing does not incur any charges. Credits are only consumed when a sync runs using on-demand capacity. |
There was a problem hiding this comment.
Removed contract/sales details and second paragraph about billing visibility and queuing behavior, keeping the section concise. Co-Authored-By: ian.alton@airbyte.io <ian.alton@airbyte.io>
|
|
||
| For critical data pipelines that must always run on time, you can enable on-demand capacity on individual connections. When committed capacity is available, the sync uses it at no extra cost. When committed capacity is exhausted, the sync consumes organization credits at a premium rate so it runs immediately instead of being queued. | ||
|
|
||
| On-demand capacity must be part of your signed contract. Contact your Airbyte representative or [talk to sales](https://www.airbyte.com/talk-to-sales) if you're interested. Once your organization administrator enables on-demand capacity at the organization level, organization admins and workspace admins can enable it per connection. Other roles can view the toggle but cannot change it. |
There was a problem hiding this comment.
lets remove
On-demand capacity must be part of your signed contract. Contact your Airbyte representative or talk to sales if you're interested
There was a problem hiding this comment.
Removed in 6d1c298 — deleted the contract/sales requirement sentence. The paragraph now starts with "Once your organization administrator enables on-demand capacity..."
|
|
||
| On-demand capacity must be part of your signed contract. Contact your Airbyte representative or [talk to sales](https://www.airbyte.com/talk-to-sales) if you're interested. Once your organization administrator enables on-demand capacity at the organization level, organization admins and workspace admins can enable it per connection. Other roles can view the toggle but cannot change it. | ||
|
|
||
| If your organization runs out of credits, syncs are queued even if on-demand capacity is enabled. For more information about credit consumption, see [Manage billing and credits](./manage-credits.md). |
There was a problem hiding this comment.
Remove - no concept of credits. It will run no matter what.
There was a problem hiding this comment.
Removed in 6d1c298 — deleted the entire "credits run out" paragraph. On-demand capacity runs no matter what.
|
|
||
| You can view more details about an individual invoice by clicking **View Invoice**. | ||
|
|
||
| ## On-demand capacity credits |
There was a problem hiding this comment.
Let's remove this section we're not using credits for that
There was a problem hiding this comment.
Removed in 6d1c298 — deleted the entire "On-demand capacity credits" section from manage-credits.md.
…eedback - Removed contract/sales requirement sentence from manage-data-workers.md - Removed 'credits run out' paragraph from manage-data-workers.md (on-demand runs no matter what) - Removed 'credits at a premium rate' phrasing from intro paragraph - Removed entire on-demand capacity credits section from manage-credits.md - Updated configuring-connections.md table to remove credit-based language Co-Authored-By: ian.alton@airbyte.io <ian.alton@airbyte.io>
4 participants
What
Adds documentation for the cloud data worker capacity enforcement feature, which queues newly triggered sync jobs when all committed data workers are in use on capacity-based plans (Pro, Enterprise Flex).
Resolves airbytehq/airbyte-internal-issues#16129
How
Adds or updates content across 9 documentation files:
connection_queued.png), new "Queued connections (Cloud Pro and Enterprise Flex)" subsection, disambiguation note for stream-level vs connection-level "Queued" (including that streams show Pending when a connection is queued)burst-tag.png), scoped to Pro & Enterprise Flexqueuedstate to Mermaid state machine diagram with transition labels, new "Queued state (Cloud Pro and Enterprise Flex)" explanatory subsectionNew image assets added:
docs/platform/cloud/managing-airbyte-cloud/assets/connection_queued.png— orange hourglass icon for queued connection statusdocs/platform/cloud/managing-airbyte-cloud/assets/burst-tag.png— Burst system tag icon (also copied todocs/platform/using-airbyte/images/burst-tag.pngfor tagging page)Review guide
manage-data-workers.md— Primary hub page. Verify: on-demand capacity section reads correctly, cross-links are accurate, workspace usage "replaces" (not "in addition to") wording is correct, connection-creation toggle note is accurate, admin-only restriction wording matches the UI, and inline Burst tag image renders at an appropriate sizereview-connection-status.md— Verify the new connection-level "Queued" row withconnection_queued.pngrenders consistently with the other status icon rows, and the stream-level disambiguation note is clearjobs.md— Mermaid state machine diagram now includesqueuedstate with\ntransition labels. Verified in Vercel preview that\nrenders as line breaksmanage-airbyte-cloud-notifications.md—slackNotificationUnsupportedprop). The table may need a webhook checkmark added, or the table/payload section need to be reconciled.sync-schedules.md— Scheduled-run replacement behavior (newer run replaces older queued one) was verified againstConnectionManagerWorkflowImpl.ktinairbyte-platform-internaltagging.md— Burst tag image added below the description paragraph. Verify sizing and renderingconfiguring-connections.md—manage-data-workers.mdintro paragraph no longer mentions credits/charges at all. Verify this is intentionally different — the table quotes the UI toggle; the hub page describes the feature more generically.Cross-cutting: All new sections link back to
manage-data-workers.mdas the hub. Verify anchors like#on-demand-capacityresolve correctly.Not included: The spec calls for additional screenshots from a test workspace beyond the two icons added here. Those still need to be captured and added.
Human review checklist
Verify— Replaced with actualQueued iconplaceholder inreview-connection-status.mdconnection_queued.pnghourglass iconmanage-airbyte-cloud-notifications.mdshows email-only (checkmark in Email column, blank Webhook column), but the payload details section says both email and webhook are supported. Code confirms the UI renders both toggles. Decide whether to add a webhook checkmark to the table or adjust the payload section.connection_queued.pngrenders at a consistent size with other status icons in the connection status tableburst-tag.pngrenders appropriately both inline (inmanage-data-workers.mdparagraph) and as a standalone image (intagging.md)Confirm FIFO ordering for the queue— Code investigation ofConnectionManagerWorkflowImpl.ktconfirmed there is no FIFO guarantee; each connection polls independently. FIFO claims removed.manage-data-workers.mdstep 3 matches the actual UI stringintents.yaml) mapsManageConnectionOnDemandCapacitytoORGANIZATION_ADMIN,WORKSPACE_ADMIN, andADMIN. Confirm these role names match what users see in the product.review-connection-status.mdandconfiguring-connections.mddon't wrap awkwardly.configuring-connections.mdand the quoted toggle text inmanage-data-workers.mdmention "charged a premium rate", while themanage-data-workers.mdintro paragraph does not mention credits or charges. Confirm this is intentional.Updates since initial revision
Commit a594d8e — Addressed Devin Review bot comments:
review-connection-status.mdset-up-a-connection.mdto keep the bullet list clean (capacity details live in hub page)Commit 5f33de1 — Addressed reviewer feedback on
manage-credits.md:Commit 9dc0636 — Code-verified queuing behavior in
airbyte-platform-internal:ConnectionManagerWorkflowImpl.ktcancels the old queued job viaCapacityWaitExceededExceptionwhen the next scheduled run arrives, then restarts the workflow for the new runreview-connection-status.mdandmanage-data-workers.mdCommit 376843f — Added icon image assets:
Queued iconplaceholder with actualconnection_queued.pnghourglass icon in the connection status tableburst-tag.pngtotagging.md(System tags section) and inline inmanage-data-workers.md(on-demand capacity section)Commit ff96308 — Addressed eval review findings (code-verified against
airbyte-platform-internal):NotificationSettingsForm.tsxrenders the toggle withoutslackNotificationUnsupported). No example payload available yet as the backend trigger is being finalized.CapacityReachedMessage.tsxshows the banner for any workspace withqueuedCount > 0— no feature flag or entitlement check.Commit adb0d6d — Added admin-only restriction for on-demand capacity toggle (per reviewer feedback referencing airbyte-platform-internal#18857 and #18851):
ManageConnectionOnDemandCapacityintent inintents.yamlis restricted toORGANIZATION_ADMIN,WORKSPACE_ADMIN, andADMINroles.RunOnDemandFormField.tsxdisables the toggle for other roles with tooltip "Only admins can enable or disable on-demand capacity".manage-data-workers.md: On-demand capacity section now states "organization admins and workspace admins can enable it per connection. Other roles can view the toggle but cannot change it." Enable step also notes the admin role requirement.configuring-connections.md: Connection settings table now notes "Only organization admins and workspace admins can change this toggle."Commit 369ee3b — Narrowed plan-scope labels from "Cloud only" to "Cloud Pro and Enterprise Flex" (per reviewer feedback):
review-connection-status.md(status table row + subsection heading),configuring-connections.md(settings table),review-connection-timeline.md(status table),tagging.md(Burst tag heading), andjobs.md(queued state heading).Commit 01f4064 — Updated toggle description text in
manage-data-workers.mdto match the latest UI copy per reviewer request:Commits 78b49dc & 6d1c298 — Removed credit references from on-demand capacity per reviewer feedback:
manage-data-workers.md("On-demand capacity must be part of your signed contract...")manage-data-workers.md— on-demand capacity runs regardlessmanage-data-workers.mdintro paragraphmanage-credits.md— on-demand capacity does not use the credits modelconfiguring-connections.mdtable description to avoid credit-based language (now says "charged a premium rate" matching the UI toggle copy)User Impact
Users on capacity-based Cloud plans (Pro, Enterprise Flex) will have documentation explaining:
No negative side effects — this is additive documentation only.
Can this PR be safely reverted and rolled back?
Link to Devin session: https://app.devin.ai/sessions/50c6288e98ee4e7b8f090c8485d81059
Requested by: Ian Alton (@ian-at-airbyte)