fix(destination-mssql): reduce memory usage to prevent OOM during large syncs#76114
Draft
devin-ai-integration[bot] wants to merge 3 commits intomasterfrom
Draft
fix(destination-mssql): reduce memory usage to prevent OOM during large syncs#76114devin-ai-integration[bot] wants to merge 3 commits intomasterfrom
devin-ai-integration[bot] wants to merge 3 commits intomasterfrom
Conversation
…ge syncs Lower maxNumOpenLoaders from 8 to 4, batchEveryNRecords from 5000 to 2500, and recordBatchSizeBytes from 10MB to 5MB. These changes reduce the peak theoretical memory footprint of concurrent loaders from ~80MB to ~20MB, preventing OOM crashes during large syncs. Co-Authored-By: bot_apk <apk@cognition.ai>
Contributor
Author
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
Co-Authored-By: bot_apk <apk@cognition.ai>
Contributor
👋 Greetings, Airbyte Team Member!Here are some helpful tips and reminders for your convenience. 💡 Show Tips and TricksPR Slash CommandsAirbyte Maintainers (that's you!) can execute the following slash commands on your PR:
📚 Show Repo GuidanceHelpful Resources
|
Co-Authored-By: bot_apk <apk@cognition.ai>
Contributor
|
Deploy preview for airbyte-docs ready! ✅ Preview Built with commit 1729d78. |
Contributor
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Resolves https://github.com/airbytehq/oncall/issues/11899:
Original issue: #76113
The destination-mssql JVM process crashes with OOM during large syncs. The
-XX:+ExitOnOutOfMemoryErrorflag causes immediate termination without graceful shutdown, surfacing as "Broken pipe" and "No exit code" errors.How
Reduced three memory-related configuration defaults in
MSSQLConfiguration.kt:recordBatchSizeBytesbatchEveryNRecordsmaxNumOpenLoadersPeak theoretical loader memory drops from approximately 80 MB (8 x 10 MB) to approximately 20 MB (4 x 5 MB). The
synchronized(parent)block inMSSQLDirectLoader.executeBatchSafely()means only one loader flushes at a time — with fewer loaders, less data accumulates while waiting.Also removed the now-unused
ObjectStorageUploadConfigurationimport (the 10 MB default came from there; the new value is an inline literal).Review guide
MSSQLConfiguration.kt— the actual fix (3 constant changes + removed import)MSSQLConfigurationTest.kt— new unit tests asserting the tuned defaultsmetadata.yaml— version bump 2.2.15 → 2.2.16docs/integrations/destinations/mssql.md— changelog entryKey things to verify
recordBatchSizeBytes.synchronized(parent)serialization bottleneck inMSSQLDirectLoader.kt:67is unchanged — with 4 loaders instead of 8, contention is lower, but flush throughput is also lower.User Impact
Syncs that previously crashed with OOM / "Broken pipe" / "No exit code" errors on large datasets should now complete successfully. Throughput may decrease for very high-volume syncs due to the lower concurrency and smaller batch sizes.
Can this PR be safely reverted and rolled back?
Link to Devin session: https://app.devin.ai/sessions/04c46da808d046528937552da2b5a9c0