Skip to content

Fix: truncate destination table on FULL_TABLE replication to prevent …#6077

Open
abbasikov wants to merge 1 commit intomage-ai:masterfrom
abbasikov:fix/data-integration-truncate
Open

Fix: truncate destination table on FULL_TABLE replication to prevent …#6077
abbasikov wants to merge 1 commit intomage-ai:masterfrom
abbasikov:fix/data-integration-truncate

Conversation

@abbasikov
Copy link
Copy Markdown

@abbasikov abbasikov commented Apr 1, 2026

1: What the issue was
FULL_TABLE replication is supposed to be a full replace, source is the single source of truth. But when
the destination table already existed, the code only ran ALTER TABLE (to add new columns) and then
INSERT — so deleted rows in the source were never removed from the destination, causing stale data to
accumulate indefinitely.

suggested Solution was to opt-in toggle, but the correct semantic of FULL_TABLE replication (per Singer spec) is always truncate-and-reload — it's not optional. Making it optional would be incorrect behavior. The Singer specification defines FULL_TABLE as a complete replacement of the destination table on every sync. Our implementation is therefore more correct than what the issue author suggested.

2: What was changed:
In build_query_strings(), when table_exists and replication_method == FULL_TABLE, we now prepend a
TRUNCATE TABLE command before the inserts. The ALTER TABLE path is preserved for INCREMENTAL
and LOG_BASED methods via the else branch.

@abbasikov
Copy link
Copy Markdown
Author

The test_web_server_windows (3.10) failure is unrelated to this change, it's a GitHub Actions artifact upload conflict (409 error) caused by parallel jobs uploading with the same artifact name. All actual tests passed

@wangxiaoyou1993
Copy link
Copy Markdown
Member

Have you tested with real Mage data integration pipeline?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants