[Enhancement] Add checksum protection for shared-data tablet metadata and txn log by luohaha · Pull Request #74924 · StarRocks/starrocks

luohaha · 2026-06-17T03:06:46Z

Why I'm doing:

In the shared-data architecture, lake tablet metadata and txn log files are
serialized as plain protobuf with no header and no checksum. Silent corruption of these
files (truncation, bit-rot, partial writes on object storage) cannot be detected on read,
and a corrupted tablet meta / txn log can lead to hard-to-diagnose failures.

The shared-nothing engine already protects its metadata with ProtobufFileWithHeader,
which carries an Adler-32 checksum, but that path hardcodes OLAP_FIX_HEADER_MAGIC_NUMBER
(== 0, a poor magic) and rejects anything else, so it cannot be reused as-is for lake.

What I'm doing:

Add an opt-in checksummed on-disk format for shared-data tablet metadata and txn logs,
with full backward/forward read compatibility.

Introduce LAKE_META_HEADER_MAGIC_NUMBER, chosen so its little-endian first on-disk
byte is 0xFE. A serialized TabletMetadataPB / TxnLogPB starts with the field-1
varint tag (0x08) and protobuf can never produce a leading 0xFE byte, so a reader
can distinguish the new checksummed format from legacy headerless protobuf with
effectively zero false-positive risk.
Parameterize ProtobufFileWithHeader with magic and allow_plain_protobuf_fallback
(both defaulted), so existing shared-nothing callers are byte-for-byte unchanged. When
fallback is enabled, load() parses legacy headerless protobuf when the magic is absent.
Add BE config lake_enable_protobuf_file_checksum (default false). It only controls the
write format; readers always auto-detect and verify the checksum when present,
regardless of the flag.
Route lake tablet meta, txn log / slog / vlog, and combined txn log through the new
save/load helpers in TabletManager, and fix vacuum.cpp's own metadata reader to use
the same auto-detecting load path.
For bundle tablet metadata, protect both layers: a per-tablet page checksum
(map<int64,uint32> tablet_meta_page_checksum in BundleTabletMetadataPB, same key as
tablet_meta_pages) plus a footer crc32 over the shared bundle header (schemas, mappings,
page pointers, and the page-checksum map itself). When the flag is on the footer becomes
[bundle_meta][crc32][size | high-bit flag]; readers detect the layout via the high bit of
the size field (collision-free, since a legacy bundle size is always well below 2^63) and
fall back to the legacy [bundle_meta][size] when it is absent.
Adapt meta_tool (print_lake_metadata / print_lake_txn_log /
print_lake_combined_txn_log) to load via the header-aware path so it parses both formats;
print_lake_bundle_metadata already goes through the updated parse_bundle_tablet_metadata.

Compatibility note: the read path is always backward compatible (a new BE reads legacy
headerless files via fallback; the checksummed format is auto-detected). The flag defaults to
false to stay safe during rolling upgrades and downgrades: while versions are mixed, an older
BE/CN uses the legacy reader and cannot parse newly-written checksummed files. Enable it only
after the whole cluster has been upgraded to a version that understands the format.

Fixes #issue

What type of PR is this:

Does this PR entail a change in behavior?

Yes, this PR will result in a change in behavior.
No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

Interface/UI changes: syntax, type conversion, expression evaluation, display information
Parameter changes: default values, similar parameters but with different default values
Policy changes: use new policy to replace old one, functionality automatically enabled
Feature removed
Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

I have added test cases for my bug fix or my new feature
This pr needs user documentation (for new or modified features or behaviors)
- I have added documentation for my new feature or new function
- This pr needs auto generate documentation
This is a backport pr

Bugfix cherry-pick branch check:

I have checked the version labels which the pr will be auto-backported to the target branch
- 4.1
- 4.0
- 3.5

github-actions · 2026-06-17T03:07:44Z

No new undocumented parameters detected by the param-drift check.

CelerData-Reviewer · 2026-06-17T03:08:52Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7c73cad15a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

github-actions · 2026-06-17T03:20:36Z

🌎 Translation Required?

✅ All translation files are up to date.
Great job! No translation actions are required for this PR.

🕒 Last updated: Sat, 20 Jun 2026 08:26:18 GMT

… and txn log Lake tablet metadata and txn logs were stored as plain protobuf with no checksum, so silent corruption could not be detected on read. - Add an opt-in checksummed format gated by BE config lake_enable_protobuf_file_checksum (default false; enable only after the whole cluster understands the format, to stay safe across rolling upgrade/downgrade). Readers always auto-detect and verify the checksum when present, regardless of the flag. - Single files (tablet meta, txn log/slog/vlog, combined txn log) use ProtobufFileWithHeader with a new LAKE_META_HEADER_MAGIC_NUMBER whose first on-disk byte is 0xFE, which no serialized protobuf can start with (wire type 6), so legacy headerless files are distinguished with zero false positives. - Bundle tablet metadata: per-tablet page checksum plus a footer crc32 over the shared header; the checksummed footer is marked by the high bit of the trailing size field (collision-free, since a legacy bundle size is well below 2^63), falling back to the legacy [bundle_meta][size] layout otherwise. - Route TabletManager and vacuum reads/writes through the new helpers, and adapt meta_tool (print_lake_metadata / print_lake_txn_log / print_lake_combined_txn_log) to parse both formats. - Add BE UTs for the single-file and bundle formats, legacy fallback, and corruption detection; update en/zh/ja BE config docs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: luohaha <18810541851@163.com>

CelerData-Reviewer · 2026-06-20T08:19:10Z

@codex review

chatgpt-codex-connector

💡 Codex Review

starrocks/be/src/storage/protobuf_file.cpp

Lines 172 to 175 in 15bd9d4

    
           if (data.size() < header.protobuf_length) { 
        
               return Status::Corruption(fmt::format("mismatched message size of protobuf data. real={} expect={}", 
        
                                                     data.size(), (int64_t)header.protobuf_length)); 
        
           }

Reject trailing bytes in checksummed protobuf files

When lake_enable_protobuf_file_checksum is enabled, lake metadata/txn-log reads go through this buffer loader with legacy fallback enabled. A file containing a valid checksummed header/body plus extra trailing bytes now passes this < size check, so those bytes are neither covered by the Adler-32 nor rejected; the non-fallback streaming path previously detects the same condition by reading protobuf_length + 1 and failing on an oversized file. This masks append/trailing-byte corruption in the new checksummed format, so the reader should require the remaining size or header.file_length to match exactly before parsing.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

github-actions · 2026-06-20T09:38:01Z

[Java-Extensions Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

github-actions · 2026-06-20T09:40:10Z

[FE Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

github-actions · 2026-06-20T09:51:33Z

[BE Incremental Coverage Report]

✅ pass : 98 / 114 (85.96%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	be/src/tools/meta_tool.cpp	0	15	00.00%	[1887, 1890, 1892, 1893, 1894, 1975, 1978, 1980, 1981, 1982, 1995, 1998, 2000, 2001, 2002]
🔵	be/src/storage/protobuf_file.cpp	28	29	96.55%	[145]
🔵	be/src/storage/tablet_meta.cpp	1	1	100.00%	[]
🔵	be/src/storage/lake/replication_txn_manager.cpp	1	1	100.00%	[]
🔵	be/src/storage/lake/vacuum.cpp	2	2	100.00%	[]
🔵	be/src/storage/lake/tablet_manager.cpp	59	59	100.00%	[]
🔵	be/src/storage/protobuf_file.h	7	7	100.00%	[]

github-actions · 2026-06-22T02:33:16Z

@Mergifyio backport branch-4.1

github-actions · 2026-06-22T02:33:18Z

@Mergifyio backport branch-4.0

mergify · 2026-06-22T02:33:33Z

backport branch-4.1

✅ Backports have been created

Details

#75102 [Enhancement] Add checksum protection for shared-data tablet metadata and txn log (backport #74924) has been created for branch branch-4.1 but encountered conflicts

Cherry-pick of 77a3cd6 has failed:

On branch mergify/bp/branch-4.1/pr-74924
Your branch is up to date with 'origin/branch-4.1'.

You are currently cherry-picking commit 77a3cd65e5.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   be/src/common/config.h
	modified:   be/src/common/config_fwd_headers_manifest.json
	modified:   be/src/storage/lake/replication_txn_manager.cpp
	modified:   be/src/storage/lake/vacuum.cpp
	modified:   be/src/storage/olap_define.h
	modified:   be/src/storage/protobuf_file.cpp
	modified:   be/src/storage/tablet_meta.cpp
	modified:   be/test/storage/lake/partial_update_test.cpp
	modified:   docs/en/administration/management/BE_parameters/shared_lake_other.md
	modified:   docs/ja/administration/management/BE_parameters/shared_lake_other.md
	modified:   docs/zh/administration/management/BE_parameters/shared_lake_other.md
	modified:   gensrc/proto/lake_types.proto

Unmerged paths:
  (use "git add/rm <file>..." as appropriate to mark resolution)
	deleted by us:   be/src/common/config_lake_fwd.h
	both modified:   be/src/storage/lake/tablet_manager.cpp
	both modified:   be/src/storage/protobuf_file.h
	both modified:   be/src/tools/meta_tool.cpp
	both modified:   be/test/storage/lake/tablet_manager_test.cpp
	both modified:   be/test/storage/protobuf_file_test.cpp

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

mergify · 2026-06-22T02:33:34Z

backport branch-4.0

✅ Backports have been created

Details

#75103 [Enhancement] Add checksum protection for shared-data tablet metadata and txn log (backport #74924) has been created for branch branch-4.0 but encountered conflicts

Cherry-pick of 77a3cd6 has failed:

On branch mergify/bp/branch-4.0/pr-74924
Your branch is up to date with 'origin/branch-4.0'.

You are currently cherry-picking commit 77a3cd65e5.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   be/src/common/config.h
	modified:   be/src/storage/lake/vacuum.cpp
	modified:   be/src/storage/olap_define.h
	modified:   be/src/storage/protobuf_file.cpp
	modified:   be/src/storage/tablet_meta.cpp
	modified:   be/test/storage/lake/partial_update_test.cpp
	modified:   docs/en/administration/management/BE_parameters/shared_lake_other.md
	modified:   docs/ja/administration/management/BE_parameters/shared_lake_other.md
	modified:   docs/zh/administration/management/BE_parameters/shared_lake_other.md
	modified:   gensrc/proto/lake_types.proto

Unmerged paths:
  (use "git add/rm <file>..." as appropriate to mark resolution)
	deleted by us:   be/src/common/config_fwd_headers_manifest.json
	deleted by us:   be/src/common/config_lake_fwd.h
	both modified:   be/src/storage/lake/replication_txn_manager.cpp
	both modified:   be/src/storage/lake/tablet_manager.cpp
	both modified:   be/src/storage/protobuf_file.h
	both modified:   be/src/tools/meta_tool.cpp
	both modified:   be/test/storage/lake/tablet_manager_test.cpp
	both modified:   be/test/storage/protobuf_file_test.cpp

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

… and txn log (#74924) Signed-off-by: luohaha <18810541851@163.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> (cherry picked from commit 77a3cd6) Signed-off-by: luohaha <18810541851@163.com>

… and txn log (backport #74924) (#75102) Signed-off-by: luohaha <18810541851@163.com> Co-authored-by: Yixin Luo <18810541851@163.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

… and txn log (#74924) Signed-off-by: luohaha <18810541851@163.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> (cherry picked from commit 77a3cd6) Signed-off-by: luohaha <18810541851@163.com>

… and txn log (backport #74924) (#75103) Signed-off-by: luohaha <18810541851@163.com> Co-authored-by: luohaha <18810541851@163.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

… and txn log (StarRocks#74924) Signed-off-by: luohaha <18810541851@163.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Oliver Layer <o.layer@celonis.com>

github-actions Bot added the behavior_changed label Jun 17, 2026

wanpengfei-git added the PROTO-REVIEW label Jun 17, 2026

wanpengfei-git requested a review from a team June 17, 2026 03:07

mergify Bot assigned luohaha Jun 17, 2026

github-actions Bot added 4.1 4.0 labels Jun 17, 2026

luohaha force-pushed the feat-lake-meta-crc branch from 7c73cad to 8a87c13 Compare June 17, 2026 03:09

github-actions Bot requested review from meegoo and xiangguangyxg June 17, 2026 03:10

github-actions Bot added the GLOBAL-REVIEW label Jun 17, 2026

luohaha force-pushed the feat-lake-meta-crc branch from 8a87c13 to e013a6c Compare June 17, 2026 03:12

github-actions Bot added the documentation Documentation changes label Jun 17, 2026

chatgpt-codex-connector Bot reviewed Jun 17, 2026

View reviewed changes

Comment thread be/src/common/config.h Outdated

github-actions Bot added 4.1 4.0 and removed 4.1 4.0 behavior_changed labels Jun 17, 2026

luohaha force-pushed the feat-lake-meta-crc branch from e013a6c to 1ae76fe Compare June 17, 2026 03:20

github-actions Bot added behavior_changed 4.1 4.0 and removed 4.1 4.0 behavior_changed labels Jun 17, 2026

luohaha dismissed xiangguangyxg’s stale review via 5e173b5 June 19, 2026 03:43

luohaha force-pushed the feat-lake-meta-crc branch from 82ee234 to 5e173b5 Compare June 19, 2026 03:43

luohaha force-pushed the feat-lake-meta-crc branch from 5e173b5 to 15bd9d4 Compare June 20, 2026 08:18

chatgpt-codex-connector Bot reviewed Jun 20, 2026

View reviewed changes

kevincai approved these changes Jun 21, 2026

View reviewed changes

luohaha enabled auto-merge (squash) June 22, 2026 02:19

xiangguangyxg approved these changes Jun 22, 2026

View reviewed changes

luohaha merged commit 77a3cd6 into StarRocks:main Jun 22, 2026
70 of 71 checks passed

github-actions Bot removed the 4.1 label Jun 22, 2026

github-actions Bot removed the 4.0 label Jun 22, 2026

This was referenced Jun 22, 2026

[Enhancement] Add checksum protection for shared-data tablet metadata and txn log (backport #74924) #75102

Merged

[Enhancement] Add checksum protection for shared-data tablet metadata and txn log (backport #74924) #75103

Merged

github-actions Bot added the 4.1-merged label Jun 22, 2026

huailiu1122 added the 4.0-merged label Jun 24, 2026

	if (data.size() < header.protobuf_length) {
	return Status::Corruption(fmt::format("mismatched message size of protobuf data. real={} expect={}",
	data.size(), (int64_t)header.protobuf_length));
	}

Uh oh!

Conversation

luohaha commented Jun 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why I'm doing:

What I'm doing:

What type of PR is this:

Checklist:

Bugfix cherry-pick branch check:

Uh oh!

github-actions Bot commented Jun 17, 2026

Uh oh!

CelerData-Reviewer commented Jun 17, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

github-actions Bot commented Jun 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🌎 Translation Required?

Uh oh!

CelerData-Reviewer commented Jun 20, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

github-actions Bot commented Jun 20, 2026

[Java-Extensions Incremental Coverage Report]

Uh oh!

github-actions Bot commented Jun 20, 2026

[FE Incremental Coverage Report]

Uh oh!

github-actions Bot commented Jun 20, 2026

[BE Incremental Coverage Report]

file detail

Uh oh!

Uh oh!

github-actions Bot commented Jun 22, 2026

Uh oh!

github-actions Bot commented Jun 22, 2026

Uh oh!

mergify Bot commented Jun 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Backports have been created

Uh oh!

mergify Bot commented Jun 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Backports have been created

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

luohaha commented Jun 17, 2026 •

edited

Loading

github-actions Bot commented Jun 17, 2026 •

edited

Loading

mergify Bot commented Jun 22, 2026 •

edited

Loading

mergify Bot commented Jun 22, 2026 •

edited

Loading