Skip to content

Commit a988bb1

Browse files
meegooclaude
andcommitted
[BugFix] review: skip redundant SCHEMA_{id} write on replay; note orphan historical entry GC
Two review follow-ups on apply_add_index STEP 4: - Gate create_schema_file on the schema id actually changing in this apply, so re-applying the same OpAddIndex (metadata replay on restart) does not repeat the remote object-store write. First apply behavior is unchanged. - Document that a fully-repointed old_schema_id can leave its historical_schemas entry unreferenced, and that apply_opcompaction's historical_schemas GC reclaims it on the next compaction. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BNsDjnXjbty1eDHzZnFAEQ Signed-off-by: meegoo <meegoo.sr@gmail.com>
1 parent c54a9d7 commit a988bb1

1 file changed

Lines changed: 12 additions & 5 deletions

File tree

be/src/storage/lake/meta_file.cpp

Lines changed: 12 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -377,6 +377,9 @@ void MetaFileBuilder::apply_add_index(const TxnLogPB_OpAddIndex& op) {
377377
// that a pinned schema_id exists in historical_schemas, so the archive
378378
// and the repoint must happen together. Guarded on historical_schemas so
379379
// replay is idempotent (on replay schema()->id() already == new id).
380+
// If every pin moved off old_schema_id, its historical_schemas entry is
381+
// left unreferenced here; the next compaction's historical_schemas GC
382+
// (see apply_opcompaction below) reclaims such entries.
380383
if (!_tablet_meta->rowset_to_schema().empty() && _tablet_meta->historical_schemas().count(new_schema_id) <= 0) {
381384
(*_tablet_meta->mutable_historical_schemas())[new_schema_id].CopyFrom(*schema);
382385
for (auto& entry : *_tablet_meta->mutable_rowset_to_schema()) {
@@ -388,11 +391,15 @@ void MetaFileBuilder::apply_add_index(const TxnLogPB_OpAddIndex& op) {
388391
// Best-effort persist the new schema as a standalone SCHEMA_{id} file so
389392
// any by-id cold reader (get_tablet_schema_by_id) resolves the indexed
390393
// schema. Non-fatal on failure: loads/compaction resolve via
391-
// metadata->schema() whose id now equals new_schema_id.
392-
if (auto* mgr = _tablet.tablet_mgr(); mgr != nullptr) {
393-
auto st = mgr->create_schema_file(_tablet_meta->id(), *schema);
394-
LOG_IF(WARNING, !st.ok()) << "apply_add_index: create_schema_file failed for tablet " << _tablet_meta->id()
395-
<< " schema_id " << schema->id() << ": " << st;
394+
// metadata->schema() whose id now equals new_schema_id. Gated on the id
395+
// actually changing in THIS apply so re-applying the same op (metadata
396+
// replay on restart) does not repeat the remote object-store write.
397+
if (old_schema_id != new_schema_id) {
398+
if (auto* mgr = _tablet.tablet_mgr(); mgr != nullptr) {
399+
auto st = mgr->create_schema_file(_tablet_meta->id(), *schema);
400+
LOG_IF(WARNING, !st.ok()) << "apply_add_index: create_schema_file failed for tablet "
401+
<< _tablet_meta->id() << " schema_id " << schema->id() << ": " << st;
402+
}
396403
}
397404
}
398405
}

0 commit comments

Comments
 (0)