MB-72725: Improve efficiency of index compaction

[CascadingRadium]

• Fixed the planner picking a single-segment ("solo") fully live roster as the best candidate, which skipped a merge task. We now deliberately avoid hitting into this degenerate case, and only create merge tasks that actively reclaim deletes.
• Fixed the file-size budget being computed from raw file size, which is static and ignores logical deletes. It now uses live file size, so deleted data no longer inflates the budget.
• Fixed vector-segment eligibility being based on raw file size, so a 4GB vector segment with 99% of its vectors deleted would never become eligible for compaction. Eligibility now uses live file size.
• Fixed the empty-segment cleanup task being counted toward the budget, even though merging empty segments produces no output segment. Only real merges now count toward the budget.
• Overhauled unit tests to assert convergence and the expected segment layout after the index settles.
• The merger now introduces merged segments in one-shot; it creates one merged segment per task generated by the planner and introduces all merged segments in one go. Earlier we were introducing each merged segment one-by-one, which would reduce concurrency in high-ingest workloads due to channel contention.
• Increase test coverage by adding unit tests for scorch.

[coveralls]

coverage: 52.338% (+0.1%) from 52.2% — mergePlan into master

[Copilot]

Pull request overview

This PR improves Scorch index compaction planning/execution by making merge decisions account for logical deletions (via “live” sizing), avoiding degenerate single-segment plans that don’t reclaim deletes, and introducing merged segments in a single introducer push to reduce contention and improve concurrency.

Changes:

• Add “live file size” estimation and use it in merge planning (budgeting + vector eligibility) so heavily-deleted segments can become eligible for compaction.
• Adjust merge planning/roster selection to avoid non-converging “solo fully-live” merge tasks and to ensure only meaningful merge tasks count against the budget.
• Overhaul/add unit tests (Scorch + merge planner) to assert convergence and expected final segment layouts; update docs/examples for config keys.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
index/scorch/snapshot_segment.go	Adds LiveFileSize() estimation to SegmentSnapshot.
index/scorch/scorch_test.go	Expands persister/merge tests and adds a deletion-vs-merge-introduction regression test.
index/scorch/scorch_knn_test.go	Updates KNN persistence test to use refcounted snapshot + configured persister options.
index/scorch/rollback_test.go	Fixes test name typo (Rollback).
index/scorch/mergeplan/merge_plan.go	Updates planner to use “live” sizing, adds roster selection tweaks, and introduces budget currency handling.
index/scorch/mergeplan/merge_plan_test.go	Refactors tests to run multi-cycle convergence assertions and adds new planner coverage.
index/scorch/merge.go	Changes merge execution to introduce all merged segments in one shot; updates bookkeeping and history mapping.
index/scorch/introducer.go	Updates merge introduction logic to use the new batch-index field when applying deletes.
docs/persister.md	Updates configuration examples to match expected JSON key casing.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.