Skip to content
Open
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,7 @@ You can configure the following settings:
| [Destination Namespace](../../using-airbyte/core-concepts/namespaces) | Determines where the replicated data is written to in the destination |
| [Destination Stream Prefix](../../using-airbyte/configuring-schema) | (Optional) Adds a prefix to each table name in the destination |
| [Detect and propagate schema changes](../../using-airbyte/schema-change-management) | Set how Airbyte handles schema changes in the source |
| Use on-demand capacity (Cloud only) | When enabled, syncs always run even if committed capacity is exhausted, by consuming credits at a premium rate. See [Monitor data worker usage](./manage-data-workers.md). |

## Stream Settings

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,11 @@ You configure notifications for each workspace separately. Once you do, Airbyte
| **Sync Disabled - Repeated Failures** | ✓ | ✓ | Airbyte has turned off a connection due to repeated failures. It has failed 30 times consecutively and has not been successful in the last 7 days. |
| **Warning - Upgrade Required** | ✓ | | A new connector version is available, but you need to manually upgrade. |
| **Sync Disabled - Upgrade Required** | ✓ | | Airbyte turned off one or more connections automatically because you missed the deadline to upgrade the connector. |
| **Queued syncs** | ✓ | | A sync from any of your connections is queued due to capacity limits. Off by default. See [Monitor data worker usage](./manage-data-workers.md). |

:::note
"Queued syncs" notifications default to off because capacity enforcement queuing is expected behavior for most organizations. Enable this notification if you want awareness when syncs are waiting for capacity.
:::

### Enabling schema update notifications

Expand Down Expand Up @@ -190,6 +195,38 @@ Webhook doesn't contain payload and only works for Slack notifications.

Webhook doesn't contain payload and only works for Slack notifications.

</details>
<details>
<summary>Queued syncs</summary>

```json
{
"data": {
"workspace": {
"id":"b510e39b-e9e2-4833-9a3a-963e51d35fb4",
"name":"Workspace1",
"url":"https://link/to/ws"
},
"connection":{
"id":"64d901a1-2520-4d91-93c8-9df438668ff0",
"name":"Connection",
"url":"https://link/to/connection"
},
"source":{
"id":"c0655b08-1511-4e72-b7da-24c5d54de532",
"name":"Source",
"url":"https://link/to/source"
},
"destination":{
"id":"5621c38f-8048-4abb-85ca-b34ff8d9a298",
"name":"Destination",
"url":"https://link/to/destination"
},
"jobId":9988
}
}
```

</details>

### Configure Slack notifications
Expand Down
8 changes: 7 additions & 1 deletion docs/platform/cloud/managing-airbyte-cloud/manage-credits.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,11 +89,17 @@ In the Invoices section, you can review any past invoices. All invoices have an

You can view more details about an individual invoice by clicking **View Invoice**.

## On-demand capacity credits
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's remove this section we're not using credits for that

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed in 6d1c298 — deleted the entire "On-demand capacity credits" section from manage-credits.md.


Devin session


If your organization uses [on-demand capacity](./manage-data-workers.md#on-demand-capacity), syncs that run when committed data worker capacity is exhausted consume organization credits at a premium rate. The premium rate is determined by your contract. Contact your Airbyte representative or [talk to sales](https://airbyte.com/company/talk-to-sales) for details.
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
If your organization uses [on-demand capacity](./manage-data-workers.md#on-demand-capacity), syncs that run when committed data worker capacity is exhausted consume organization credits at a premium rate. The premium rate is determined by your contract. Contact your Airbyte representative or [talk to sales](https://airbyte.com/company/talk-to-sales) for details.
If your organization uses [on-demand capacity](./manage-data-workers.md#on-demand-capacity), syncs that run when committed data worker capacity is exhausted consume organization credits at a premium rate.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Applied in 78b49dc — removed the contract/sales sentence, keeping just the one-liner about premium rate.


Devin session


On-demand credit consumption appears in your billing alongside other credit usage. If your organization runs out of credits, on-demand syncs are queued like standard syncs. Queuing does not incur any charges. Credits are only consumed when a sync runs using on-demand capacity.
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
On-demand credit consumption appears in your billing alongside other credit usage. If your organization runs out of credits, on-demand syncs are queued like standard syncs. Queuing does not incur any charges. Credits are only consumed when a sync runs using on-demand capacity.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deleted this paragraph in 78b49dc.


Devin session


## Billing notifications

By default, all customers automatically review upcoming invoice notifications 3 and 7 days before the invoice is finalized. All billing notifications are sent to the billing email on file.

You can also enroll in billing notifications for your organization. Airbyte highly recommends enrolling in billing notifications to ensure you stay up-to-date on your upcoming invoices.
You can also enroll in billing notifications for your organization. Airbyte highly recommends enrolling in billing notifications to ensure you stay up-to-date on your upcoming invoices. If you use on-demand capacity, consider setting up billing alerts to monitor premium credit consumption. For more information about managing capacity, see [Monitor data worker usage](./manage-data-workers.md).
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is that even an option? I thought we just sent an email when syncs were queued.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right — I removed that sentence in 5f33de1. The billing notifications paragraph is now back to its original text without the on-demand capacity suggestion.


Devin session


Airbyte can notify you when:

Expand Down
44 changes: 41 additions & 3 deletions docs/platform/cloud/managing-airbyte-cloud/manage-data-workers.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ products: cloud-plus

# Monitor data worker usage

If you are on a capacity-based plan, you can monitor your data worker usage across your organization. To view data worker usage, you need the **organization admin** role.
If you are on a capacity-based plan, you can monitor your data worker usage across your organization and understand how capacity enforcement affects your syncs. When all committed data workers are in use, newly triggered sync jobs are queued until capacity becomes available. To view data worker usage, you need the **organization admin** role.

## Open the usage chart

Expand All @@ -24,16 +24,30 @@ Hover on a day to see more details about it.

- To change the date range, click the date dropdown in the chart's upper right corner and choose a new date range.

## Workspace-level data worker usage

In addition to the organization-level chart, workspace editors can view data worker usage for their workspace.

1. Click **Workspace Settings** > **Usage**.

2. Review the line graph, which shows hourly data worker usage over a 7-day period for the current workspace.

This helps you understand your workspace's contribution to overall organization capacity usage.

## What to do if you hit your data worker limit

An infrequent instance of maximum usage probably isn't a problem. If you're regularly hitting your data worker limit, you have three options.
An infrequent instance of maximum usage probably isn't a problem. If you're regularly hitting your data worker limit, you have four options.

- Accept that Airbyte may queue or throttle your connections.
- Accept that Airbyte may queue your connections. Queued syncs run in the order they were triggered. If a connection already has a queued sync and its next scheduled run arrives, the newer run replaces the older queued one so the most recent data syncs when capacity frees up.

- Reschedule some connections so they run at different times of the day, week, or month.

- Buy more data workers to increase capacity.

- Enable [on-demand capacity](#on-demand-capacity) for critical connections so they always run, even when committed capacity is exhausted.

Manually triggered syncs that remain queued for 8 hours are automatically cancelled.

### Optimize data worker usage

If you can, it's preferable to optimize Airbyte by rescheduling connections outside of busy periods.
Expand All @@ -53,3 +67,27 @@ If you can, it's preferable to optimize Airbyte by rescheduling connections outs
### Buy more data workers

If you've tried to optimize scheduling and still need more data workers, contact your Airbyte representative or [talk to sales](https://www.airbyte.com/talk-to-sales).

## On-demand capacity

For critical data pipelines that must always run on time, you can enable on-demand capacity on individual connections. When committed capacity is available, the sync uses it at no extra cost. When committed capacity is exhausted, the sync consumes organization credits at a premium rate so it runs immediately instead of being queued.

On-demand capacity must be part of your signed contract. Contact your Airbyte representative or [talk to sales](https://www.airbyte.com/talk-to-sales) if you're interested. Once your organization administrator enables on-demand capacity at the organization level, workspace editors can enable it per connection.

If your organization runs out of credits, syncs are queued even if on-demand capacity is enabled. For more information about credit consumption, see [Manage billing and credits](./manage-credits.md).
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove - no concept of credits. It will run no matter what.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed in 6d1c298 — deleted the entire "credits run out" paragraph. On-demand capacity runs no matter what.


Devin session


### Enable on-demand capacity on a connection

1. Click **Connections** and select the connection you want to configure.

2. Click **Settings**.

3. Toggle **Use on-demand capacity**. The toggle description reads: "Enable connection to use credits to always run syncs on demand. Syncs for this connection will never be queued, and only consume credits if no capacity is available."

When you enable on-demand capacity on a connection, Airbyte automatically applies a "Burst" tag with an orange gradient background and a star icon. You can filter connections by the Burst tag to see all on-demand connections at a glance. If you disable on-demand capacity, Airbyte removes the Burst tag automatically. For more information about tags, see [Tagging connections](/platform/using-airbyte/tagging).

### Identify queued connections

When your committed capacity is fully utilized, connections waiting for capacity display an orange hourglass icon and a "Queued" status. You can filter the Connections page by "Queued" status to find all queued connections. A dismissable yellow banner also appears at the top of the Connections page: "Maximum capacity currently reached, additional jobs will be queued until capacity is available."

For more information about connection statuses, see [Connection status](./review-connection-status.md).
Original file line number Diff line number Diff line change
Expand Up @@ -42,15 +42,28 @@ To view the connection status:

3. Select a single connection to view more details about the connection and for a breakdown of the status of each Stream in that connection.

| Icon | Status | Description |
| -------------------------------------------------- | ----------- | ------------------------------------------------------------------- |
| ![Healthy](./assets/connection_synced.png) | **Healthy** | The most recent sync for this connection succeeded |
| ![Failed](./assets/connection_action_required.png) | **Failed** | The most recent sync for this connection failed |
| ![Running](./assets/connection_syncing.png) | **Running** | The connection is currently actively syncing |
| ![Paused](./assets/connection_disabled.png) | **Paused** | The connection is disabled and isn't scheduled to run automatically |
| Icon | Status | Description |
| -------------------------------------------------- | ----------- | ---------------------------------------------------------------------------------------------------- |
| ![Healthy](./assets/connection_synced.png) | **Healthy** | The most recent sync for this connection succeeded |
| ![Failed](./assets/connection_action_required.png) | **Failed** | The most recent sync for this connection failed |
| ![Running](./assets/connection_syncing.png) | **Running** | The connection is currently actively syncing |
| ![Paused](./assets/connection_disabled.png) | **Paused** | The connection is disabled and isn't scheduled to run automatically |
| Queued icon | **Queued** | The connection's sync is waiting for data worker capacity to become available (Cloud only) |

4. On the **Status** tab for a connection, there is a list of associated Streams. To the left of the name for each Stream, there is an icon that displays its status.

### Queued connections (Cloud only)
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pro/Flex only?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good call — updated in 369ee3b. Changed from "Cloud only" to "Cloud Pro and Enterprise Flex only" to be more precise about which plans have capacity enforcement. Updated the table row description and the subsection heading.


Devin session


On capacity-based Cloud plans, when your organization's committed data worker capacity is fully utilized, newly triggered sync jobs enter a "Queued" state. Queued connections display an orange hourglass icon.

Queued jobs run in the order they were triggered. A queued job is cancelled when:

- The connection is updated, cancelled, deleted, or reset.
- The next scheduled sync for that connection arrives. The newer sync replaces the queued one so the most recent data syncs when capacity frees up.
- Eight hours have elapsed for manually triggered syncs.

You can filter the Connections page by "Queued" status to identify all queued connections. For information about monitoring and managing data worker capacity, see [Monitor data worker usage](./manage-data-workers.md).

## Review one stream's status

The stream status allows you to monitor an individual stream's latest status. Connections often sync more than one stream. This view allows you to more easily determine if there is a problem with a given stream that could be causing problems with the connection.
Expand All @@ -64,6 +77,10 @@ The stream status allows you to monitor an individual stream's latest status. Co
| ![Error](./assets/connection_incomplete.png) | **Error** | The connection didn't succeed on its most recent sync, but Airbyte expects it to recover on the next one |
| ![Action Required](./assets/connection_action_required.png) | **Action Required** | A breaking change related to the source or destination requires attention to resolve |

:::note
Stream-level "Queued" status means the stream is waiting to sync within an active connection sync. This is different from a connection-level "Queued" status, which means the entire sync job is waiting for data worker capacity. For more information about capacity-based queuing, see [Monitor data worker usage](./manage-data-workers.md).
:::

Once the sync is complete, each stream displays the time since Airbyte loaded the last record to the destination. You can click **Last record loaded** in the header to optionally display the exact datetime the last record loaded.

## Per-stream actions
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,9 @@ Completed syncs, [refreshes](/platform/operator-guides/refreshes), and [clears](
| Failed | The sync encountered a fatal error, and needs intervention to resolve. Airbyte loaded a subset or none of the data to the destination. |
| Cancelled | Someone cancelled the sync before it finished. |
| Running | The sync is currently running. |
| Queued | The sync is waiting for data worker capacity to become available. (Cloud only) |

Queued syncs may appear as "Cancelled" in the timeline if they were cancelled due to a connection update, the next scheduled run arriving, or the 8-hour timeout for manual syncs. For more information about capacity queuing, see [Monitor data worker usage](./manage-data-workers.md).

### How the Connection Timeline reflects moved data

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,12 @@ These limitations only apply to those using the Standard plan. If you upgrade to

- Size of a single record: 20MB\*\*

## Pro and Enterprise Flex plan considerations

Organizations on capacity-based plans (Pro, Enterprise Flex) have a contracted number of data workers. When all data workers are in use, additional syncs are queued until capacity becomes available. This is not a hard limit on the number of connections or syncs you can create, but a limit on concurrent sync execution.

You can enable on-demand capacity for critical connections so they run even when committed capacity is exhausted. See [Monitor data worker usage](./manage-data-workers.md) for details.

\*\* The effective maximum size of the record may vary based per destination. Some destinations may
fail to sync if a record cannot be stored, but Destinations which support
[typing and deduping](/platform/using-airbyte/core-concepts/typing-deduping) will adjust your record so that
Expand Down
17 changes: 16 additions & 1 deletion docs/platform/understanding-airbyte/jobs.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,7 @@ direction TB
state NonTerminal {
[*] --> pending
pending
queued
running
incomplete
note left of incomplete
Expand All @@ -97,7 +98,9 @@ note left of NonSuccess
All Non Terminal Statuses can be transitioned to cancelled or failed
end note

pending --> running
pending --> queued : capacity enforcement active,\nno capacity available
pending --> running : capacity available
queued --> running : capacity becomes available
running --> incomplete
incomplete --> running
running --> succeeded
Expand All @@ -118,6 +121,18 @@ stateDiagram-v2
running --> failed
```

### Queued state (Cloud only)

On capacity-based Cloud plans, the `queued` state is used when capacity enforcement is active and all committed data workers are in use. A job transitions from `pending` to `queued` when no capacity is available. The job stays in `queued` until capacity frees up, at which point it transitions to `running`.

A queued job is cancelled if:

- The connection is modified, cancelled, deleted, or reset.
- The next scheduled sync for that connection arrives. The newer sync replaces the queued one.
- Eight hours have elapsed for manually triggered syncs.

Capacity enforcement applies only to sync jobs, not to check, discover, or spec operations. For more information, see [Monitor data worker usage](/platform/cloud/managing-airbyte-cloud/manage-data-workers).

### Attempts and Retries

In the event of a failure, the Airbyte platform will retry the pipeline. Each of these sub-invocations of a job is called an attempt.
Expand Down
2 changes: 2 additions & 0 deletions docs/platform/using-airbyte/core-concepts/sync-schedules.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,8 @@ It can be triggered by clicking the "Sync Now" button at any time through the UI
- Only one sync per connection can run at a time.
- If a sync is scheduled to run before the previous sync finishes, the scheduled sync will start after the completion of the previous sync.
- Syncs can run at most every 60 minutes in Airbyte Cloud. Reach out to [Sales](https://airbyte.com/company/talk-to-sales) if you require replication more frequently than once per hour.
- On capacity-based Cloud plans, if all committed data workers are in use, newly triggered syncs are queued until capacity is available. You can enable [on-demand capacity](/platform/cloud/managing-airbyte-cloud/manage-data-workers#on-demand-capacity) for critical connections so they always run on time.
- If a connection has a queued sync and its next scheduled run arrives, the newer scheduled run replaces the queued one. This ensures that when capacity frees up, the most recent data is synced rather than stale data from the original trigger.

:::note
For Scheduled or cron scheduled syncs, Airbyte guarantees syncs will initiate with a schedule accuracy of +/- 30 minutes.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ A **connection** is the relationship between a source connector and a destinatio

- What data Airbyte should replicate
- How Airbyte should read and write data
- When Airbyte should initiate a data sync
- When Airbyte should initiate a data sync. On capacity-based Cloud plans, syncs may be queued if data worker capacity is fully utilized. See [Monitor data worker usage](/platform/cloud/managing-airbyte-cloud/manage-data-workers) for details.
- Where Airbyte should write replicated data
- How Airbyte should handle schema drift

Expand Down
Loading
Loading