[BugFix] Carry forward untouched indexes into the file-bundling bundle on compaction publish (backport #76105)

[mergify]

Why I'm doing:

With file_bundling enabled (the default for shared-data tables since 4.0), one
physical bundle file (meta/0_<version>.meta) holds the metadata of all
tablets of a partition at a given version.

PublishVersionDaemon.publishPartition builds the tablet set to publish from the
transaction's point-in-time index view (getPartitionLoadedIndexes). For a lake
compaction that index set is snapshotted at transaction begin
(CompactionScheduler.beginTransaction). If a rollup / synchronous-MV index
becomes visible after the compaction transaction begins but before it
publishes, the stale view omits that index, so the compaction writes the new
version's bundle without the rollup index's tablets. Their metadata then
disappears at that version, and the next publish that reads a rollup tablet at
that base version fails with can not find tablet <id> from shared tablet metadata (a 404 on the bundle object), permanently wedging the partition's
publish queue.

Reproduced/diagnosed via StarRocksTest#11536 (file_bundling table + sync MV +
a subsequent import hangs). Confirmed on the affected cluster by reading the
bundle objects directly: 0_12.meta contained 20 tablets (base + rollup) while
0_13.meta, written by a compaction, contained only the 10 base tablets.

What I'm doing:

When publishing a file-bundling partition version, carry forward every
currently-visible NORMAL index that the transaction did not touch, publishing
those tablets as an empty-txn version bump within the same aggregate request
as the touched tablets — two separate aggregate publishes would each
truncate-write the same 0_<version>.meta and drop one of the two sets. This
keeps the new bundle a complete whole-partition snapshot.

Only affects the file-bundling path; a normal load already covers every visible
index, so nothing is carried forward and its behavior is unchanged. Added a unit
test for the carry-forward selection (collectFileBundlingCarryForwardTablets).

Fixes https://github.com/StarRocks/StarRocksTest/issues/11536

What type of PR is this:

• BugFix
• Feature
• Enhancement
• Refactor
• UT
• Doc
• Tool

Does this PR entail a change in behavior?

• Yes, this PR will result in a change in behavior.
• No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

• Interface/UI changes: syntax, type conversion, expression evaluation, display information
• Parameter changes: default values, similar parameters but with different default values
• Policy changes: use new policy to replace old one, functionality automatically enabled
• Feature removed
• Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

• I have added test cases for my bug fix or my new feature
• This pr needs user documentation (for new or modified features or behaviors)
  • I have added documentation for my new feature or new function
  • This pr needs auto generate documentation
• This is a backport pr

Bugfix cherry-pick branch check:

• I have checked the version labels which the pr will be auto-backported to the target branch
  • 4.1
  • 4.0
  • 3.5

---

This is an automatic backport of pull request #76105 done by [Mergify](https://mergify.com).

[mergify]

Cherry-pick of 35c68c6 has failed:

On branch mergify/bp/branch-4.0/pr-76105
Your branch is up to date with 'origin/branch-4.0'.

You are currently cherry-picking commit 35c68c6305.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   fe/fe-core/src/main/java/com/starrocks/transaction/PublishVersionDaemon.java
	both modified:   fe/fe-core/src/test/java/com/starrocks/transaction/PublishVersionDaemonTest.java

no changes added to commit (use "git add" and/or "git commit -a")

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

[mergify]

@mergify[bot]: Backport conflict, please reslove the conflict and resubmit the pr

[sevev]

Not backporting this fix to branch-4.0. The carry-forward mechanism depends on the no-op empty-transaction publish path (TxnInfoPB.no_op_publish, FE proto field + BE handling in transactions.cpp), introduced by feature PR #73553, which is not present in branch-4.0 — 4.0's TxnInfoPB has no no_op_publish field and its BE ignore_txn_log is driven by force_publish, so an empty carry-forward transaction cannot advance the version without data. It also uses PhysicalPartition.getLatestMaterializedIndices (feature PR #67595). Enabling this on 4.0 would require backporting those feature PRs (including a BE change) into a release branch, which we're choosing not to do. The fix is backported to branch-4.1 (#76190) and branch-4.1.3 (#76192); the 4.0 box on the source PR #76105 has been unchecked. Leaving this PR closed.