-
Notifications
You must be signed in to change notification settings - Fork 5.1k
docs(platform): Add cloud data worker capacity enforcement documentation #75955
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Changes from 9 commits
b1ab58d
bb07c97
a594d8e
5f33de1
9dc0636
376843f
ff96308
adb0d6d
369ee3b
01f4064
78b49dc
6d1c298
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
|
|
@@ -89,6 +89,12 @@ In the Invoices section, you can review any past invoices. All invoices have an | |||||
|
|
||||||
| You can view more details about an individual invoice by clicking **View Invoice**. | ||||||
|
|
||||||
| ## On-demand capacity credits | ||||||
|
|
||||||
| If your organization uses [on-demand capacity](./manage-data-workers.md#on-demand-capacity), syncs that run when committed data worker capacity is exhausted consume organization credits at a premium rate. The premium rate is determined by your contract. Contact your Airbyte representative or [talk to sales](https://airbyte.com/company/talk-to-sales) for details. | ||||||
|
||||||
| If your organization uses [on-demand capacity](./manage-data-workers.md#on-demand-capacity), syncs that run when committed data worker capacity is exhausted consume organization credits at a premium rate. The premium rate is determined by your contract. Contact your Airbyte representative or [talk to sales](https://airbyte.com/company/talk-to-sales) for details. | |
| If your organization uses [on-demand capacity](./manage-data-workers.md#on-demand-capacity), syncs that run when committed data worker capacity is exhausted consume organization credits at a premium rate. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Applied in 78b49dc — removed the contract/sales sentence, keeping just the one-liner about premium rate.
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| On-demand credit consumption appears in your billing alongside other credit usage. If your organization runs out of credits, on-demand syncs are queued like standard syncs. Queuing does not incur any charges. Credits are only consumed when a sync runs using on-demand capacity. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -4,7 +4,7 @@ products: cloud-plus | |
|
|
||
| # Monitor data worker usage | ||
|
|
||
| If you are on a capacity-based plan, you can monitor your data worker usage across your organization. To view data worker usage, you need the **organization admin** role. | ||
| If you are on a capacity-based plan, you can monitor your data worker usage across your organization and understand how capacity enforcement affects your syncs. When all committed data workers are in use, newly triggered sync jobs are queued until capacity becomes available. To view data worker usage, you need the **organization admin** role. | ||
|
|
||
| ## Open the usage chart | ||
|
|
||
|
|
@@ -24,16 +24,30 @@ Hover on a day to see more details about it. | |
|
|
||
| - To change the date range, click the date dropdown in the chart's upper right corner and choose a new date range. | ||
|
|
||
| ## Workspace-level data worker usage | ||
|
|
||
| On capacity-based plans with data worker entitlements, the workspace Usage page shows data worker usage instead of credit usage. Workspace editors can view data worker usage for their workspace. | ||
|
|
||
| 1. Click **Workspace Settings** > **Usage**. | ||
|
|
||
| 2. Review the line graph, which shows hourly data worker usage over a 7-day period for the current workspace. | ||
ian-at-airbyte marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| This helps you understand your workspace's contribution to overall organization capacity usage. | ||
|
|
||
| ## What to do if you hit your data worker limit | ||
|
|
||
| An infrequent instance of maximum usage probably isn't a problem. If you're regularly hitting your data worker limit, you have three options. | ||
| An infrequent instance of maximum usage probably isn't a problem. If you're regularly hitting your data worker limit, you have four options. | ||
|
|
||
| - Accept that Airbyte may queue or throttle your connections. | ||
| - Accept that Airbyte may queue your connections. If a connection already has a queued sync and its next scheduled run arrives, the newer run replaces the older queued one so the most recent data syncs when capacity frees up. | ||
|
|
||
| - Reschedule some connections so they run at different times of the day, week, or month. | ||
|
|
||
| - Buy more data workers to increase capacity. | ||
|
|
||
| - Enable [on-demand capacity](#on-demand-capacity) for critical connections so they always run, even when committed capacity is exhausted. | ||
|
|
||
| Manually triggered syncs that remain queued for 8 hours are automatically cancelled. | ||
|
|
||
| ### Optimize data worker usage | ||
|
|
||
ian-at-airbyte marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| If you can, it's preferable to optimize Airbyte by rescheduling connections outside of busy periods. | ||
|
|
@@ -53,3 +67,29 @@ If you can, it's preferable to optimize Airbyte by rescheduling connections outs | |
| ### Buy more data workers | ||
|
|
||
| If you've tried to optimize scheduling and still need more data workers, contact your Airbyte representative or [talk to sales](https://www.airbyte.com/talk-to-sales). | ||
|
|
||
| ## On-demand capacity | ||
|
|
||
| For critical data pipelines that must always run on time, you can enable on-demand capacity on individual connections. When committed capacity is available, the sync uses it at no extra cost. When committed capacity is exhausted, the sync consumes organization credits at a premium rate so it runs immediately instead of being queued. | ||
|
|
||
| On-demand capacity must be part of your signed contract. Contact your Airbyte representative or [talk to sales](https://www.airbyte.com/talk-to-sales) if you're interested. Once your organization administrator enables on-demand capacity at the organization level, organization admins and workspace admins can enable it per connection. Other roles can view the toggle but cannot change it. | ||
|
||
|
|
||
| If your organization runs out of credits, syncs are queued even if on-demand capacity is enabled. For more information about credit consumption, see [Manage billing and credits](./manage-credits.md). | ||
ian-at-airbyte marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| ### Enable on-demand capacity on a connection | ||
|
|
||
| 1. Click **Connections** and select the connection you want to configure. | ||
|
|
||
| 2. Click **Settings**. | ||
|
|
||
| 3. Toggle **Use on-demand capacity**. The toggle description reads: "Enable connection to use credits to always run syncs on demand. Syncs for this connection will never be queued, and only consume credits if no capacity is available." You must have the organization admin or workspace admin role to change this toggle. | ||
|
|
||
| You can also enable on-demand capacity when first creating a connection. The toggle appears in the connection configuration during setup. | ||
|
|
||
| When you enable on-demand capacity on a connection, Airbyte automatically applies a "Burst" tag () with an orange gradient background and a star icon. You can filter connections by the Burst tag to see all on-demand connections at a glance. If you disable on-demand capacity, Airbyte removes the Burst tag automatically. For more information about tags, see [Tagging connections](/platform/using-airbyte/tagging). | ||
|
|
||
| ### Identify queued connections | ||
|
|
||
| When your committed capacity is fully utilized, connections waiting for capacity display an orange hourglass icon and a "Queued" status. You can filter the Connections page by "Queued" status to find all queued connections. A dismissable yellow banner also appears at the top of the Connections page: "Maximum capacity currently reached, additional jobs will be queued until capacity is available." | ||
|
|
||
| For more information about connection statuses, see [Connection status](./review-connection-status.md). | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's remove this section we're not using credits for that
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Removed in 6d1c298 — deleted the entire "On-demand capacity credits" section from manage-credits.md.
Devin session