Skip to content

[BugFix] FSE-v2 missed setting the schema for shared-data sorted schema change (backport #72235)#72347

Closed
mergify[bot] wants to merge 1 commit into
branch-4.0from
mergify/bp/branch-4.0/pr-72235
Closed

[BugFix] FSE-v2 missed setting the schema for shared-data sorted schema change (backport #72235)#72347
mergify[bot] wants to merge 1 commit into
branch-4.0from
mergify/bp/branch-4.0/pr-72235

Conversation

@mergify

@mergify mergify Bot commented Apr 29, 2026

Copy link
Copy Markdown
Contributor

Why I'm doing:

On shared-data, an ALTER that changes the sort key is wrongly cancelled
with a misleading "table not found" when the schema cache is disabled:

update information_schema.be_configs
    set value = "0" where name = "lake_metadata_cache_limit";

ALTER TABLE t1 ADD COLUMN k0 INT KEY DEFAULT "0" FIRST;
W table_schema_service.cpp:333 failed to get schema.
  db_id: 0, table_id: 0, schema_id: 10422, tablet_id: 10423, txn_id: 249,
  error: Table not exist: table not found, may be dropped, db id: 0, table id: 0

Walking the cause:

  • This alter runs through SortedSchemaChange, which writes the converted
    rowsets through DeltaWriter. The builder is only fed schema_id; the
    writer is left to resolve the schema itself.

  • Since [Enhancement] Add BE TableSchemaService for Fast Schema Evolution in shared-data #66699 (FSE-v2 in shared-data, merged Dec 2025), DeltaWriter
    resolves the schema through TableSchemaService, whose lookup chain is:

    schema cache (by schema_id)
       -> latest cached tablet metadata
       -> FE RPC (by db_id, table_id, schema_id)
    
  • Shared-data alter tasks (TAlterTabletReqV2) don't carry catalog ids.
    When the lookup falls through to the RPC, the request goes out with
    db_id=0, table_id=0 and FE rightly answers "Table not exist".

The bug has been latent because the cache (default 2 GB) and cached
tablet metadata almost always answer the schema lookup before the RPC.
Disabling the cache exposes the broken RPC path.

DirectSchemaChange is unaffected — it builds its writer with a
TabletSchemaPtr directly and never touches the service.

What I'm doing:

The sorted schema change already holds the new tablet schema; the
schema-service lookup is unnecessary. Hand the schema to DeltaWriter
directly.

  • DeltaWriterBuilder gains set_tablet_schema(TabletSchemaPtr). When
    set, DeltaWriter::init_tablet_schema() short-circuits and bypasses
    TableSchemaService entirely. set_schema_id is still required and
    validated against the preset schema's id.
  • SortedSchemaChange::process passes the new tablet schema into the
    builder. DirectSchemaChange is unchanged.

As a side effect this also removes one (normally successful) RPC per
sorted schema change in the cache-hit case — the residual TODO already
flagged in schema_change.cpp.

Fixes #issue

What type of PR is this:

  • BugFix
  • Feature
  • Enhancement
  • Refactor
  • UT
  • Doc
  • Tool

Does this PR entail a change in behavior?

  • Yes, this PR will result in a change in behavior.
  • No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

  • Interface/UI changes: syntax, type conversion, expression evaluation, display information
  • Parameter changes: default values, similar parameters but with different default values
  • Policy changes: use new policy to replace old one, functionality automatically enabled
  • Feature removed
  • Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

  • I have added test cases for my bug fix or my new feature
    • BE UT: a regression test that disables the metacache, hooks the
      schema RPC sync point, and asserts the sorted schema change runs
      end-to-end without firing the RPC.
    • SQL: test_fse_alter_with_metacache_off @cloud pinning the original
      repro.
  • This pr needs user documentation (for new or modified features or behaviors)
    • I have added documentation for my new feature or new function
    • This pr needs auto generate documentation
  • This is a backport pr

Bugfix cherry-pick branch check:

…ma change (#72235)

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: wanpengfei-git <wanpengfei91@163.com>
(cherry picked from commit d70921f)

# Conflicts:
#	be/test/storage/lake/schema_change_test.cpp
@mergify mergify Bot added the conflicts label Apr 29, 2026
@mergify

mergify Bot commented Apr 29, 2026

Copy link
Copy Markdown
Contributor Author

Cherry-pick of d70921f has failed:

On branch mergify/bp/branch-4.0/pr-72235
Your branch is up to date with 'origin/branch-4.0'.

You are currently cherry-picking commit d70921f1e4.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   be/src/storage/lake/delta_writer.cpp
	modified:   be/src/storage/lake/delta_writer.h
	modified:   be/src/storage/lake/schema_change.cpp
	new file:   test/sql/test_fast_schema_evolution/R/test_fse_alter_with_metacache_off
	new file:   test/sql/test_fast_schema_evolution/T/test_fse_alter_with_metacache_off

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   be/test/storage/lake/schema_change_test.cpp

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

@mergify

mergify Bot commented Apr 29, 2026

Copy link
Copy Markdown
Contributor Author

@mergify[bot]: Backport conflict, please reslove the conflict and resubmit the pr

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant