Skip to content

Adding 5K tables integration test for schema migration#1324

Merged
sm745052 merged 23 commits into
masterfrom
5ksmtit
May 27, 2026
Merged

Adding 5K tables integration test for schema migration#1324
sm745052 merged 23 commits into
masterfrom
5ksmtit

Conversation

@sm745052

@sm745052 sm745052 commented Apr 29, 2026

Copy link
Copy Markdown
Contributor

Adds a 5,000-table integration test for MySQL direct connect and introduces a BatchedInfoSchema interface to allow bulk metadata fetching, improving performance for large schemas.

Changes

  • BatchedInfoSchema: New interface in infoschema.go for bulk fetching of columns, constraints, FKs, and indexes.
  • Parallel Batching: Updates GenerateSrcSchema to process tables in parallel batches.
  • 5k Test: Adds TestIntegration_MYSQL_DirectConnect_5000Tables and 5,000 table SQL dump.
  • Corrected TestGetConstraints_CheckConstraintsTableAbsent: Updated the test to expect success when the CHECK_CONSTRAINTS table is missing. Now validates the fallback logic that uses a simpler query to fetch TABLE_CONSTRAINTS when the check constraints table is unavailable. Previously, the test was incorrectly asserting failure due to an incomplete mock setup for the fallback query.
  • Ignored ExpressionId in Source Schema Assertions: Modified the test helper assertSrcColDef in internal/test_util.go to ignore ExpressionId for both expected and actual columns. This avoids test brittleness caused by non-deterministic ID generation order during parallel batch processing.

Batch Size Experiments (5k Tables)

Batch Size Result Time
100 PASS ~57s
50 PASS ~94s
10 PASS ~378s
1 TIMEOUT >30m

Conclusion: Larger batch sizes significantly reduce schema processing time. The PR currently defaults to 50.

@sm745052 sm745052 requested a review from a team as a code owner April 29, 2026 09:17
@sm745052 sm745052 requested review from aasthabharill and pratickchokhani and removed request for a team April 29, 2026 09:17
@codecov

codecov Bot commented Apr 29, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 70.21944% with 95 lines in your changes missing coverage. Please review.
✅ Project coverage is 51.19%. Comparing base (03e1b89) to head (7bffe29).

Files with missing lines Patch % Lines
sources/common/infoschema.go 0.00% 59 Missing ⚠️
sources/mysql/infoschema.go 87.50% 21 Missing and 11 partials ⚠️
internal/test_util.go 0.00% 4 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1324      +/-   ##
==========================================
+ Coverage   51.15%   51.19%   +0.03%     
==========================================
  Files         202      202              
  Lines       22639    22796     +157     
  Branches      552      552              
==========================================
+ Hits        11582    11671      +89     
- Misses      10375    10437      +62     
- Partials      682      688       +6     
Components Coverage Δ
backend-apis 46.10% <ø> (ø)
backend-library 56.84% <70.21%> (-0.01%) ⬇️
cli 26.65% <ø> (ø)
frontend 39.34% <ø> (ø)
Files with missing lines Coverage Δ
internal/test_util.go 28.74% <0.00%> (-0.71%) ⬇️
sources/mysql/infoschema.go 79.43% <87.50%> (+1.99%) ⬆️
sources/common/infoschema.go 7.77% <0.00%> (-3.00%) ⬇️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@sm745052 sm745052 changed the title Adding 2.5K tables integration test for schema migration Adding 5K tables integration test for schema migration Apr 30, 2026
@sm745052 sm745052 force-pushed the 5ksmtit branch 2 times, most recently from f4084b5 to 169838d Compare May 11, 2026 08:22
@sm745052 sm745052 requested a review from rohitwali May 12, 2026 05:23
Comment thread sources/common/infoschema.go
Comment thread sources/common/infoschema.go
Comment thread sources/mysql/infoschema.go
Comment thread sources/mysql/infoschema.go
@sm745052 sm745052 merged commit 4dcf563 into master May 27, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants