perf(dir-catalog): rewrite manifest mutations with copy-on-write

[jackye1995]

Summary

• Replace __manifest merge-insert/delete maintenance with streaming copy-on-write rewrites.
• Build strict-overwrite replacement scalar indices for object_id, object_type, and base_objects; delete table-version ranges without expanding every version id.

This PR is the cleaned-up, production implementation of the copy-on-write approach prototyped in #6794.

Expected performance

From the #6794 prototype benchmark (S3, multi-process, 1K manifest entries):

• Write throughput: ~1.6-2x higher (~2.3-2.5 vs ~1.2-1.5 ops/s)
• Write tail latency: ~3.5-4.1x lower p90/p99 (no periodic compaction spikes)
• Warm read at high concurrency: up to ~2.4x throughput (single-fragment reads)
• Cold read: ~11-15% lower latency

[codecov]

Codecov Report

❌ Patch coverage is 86.44068% with 216 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance-namespace-impls/src/dir/manifest.rs	85.82%	154 Missing and 53 partials ⚠️
rust/lance/src/dataset/write/commit.rs	94.49%	6 Missing ⚠️
rust/lance/src/io/commit.rs	75.00%	3 Missing ⚠️

📢 Thoughts on this report? Let us know!

[LuQQiu]

Curious about the performance number with this new approach

[LuQQiu]

Oh see the performance numbers.... lolll skip reviewing the PR descriptions fully

[LuQQiu]

Why
Cold read: ~11-15% lower latency

[jackye1995]

Why Cold read: ~11-15% lower latency

I think it's because in general just fewer files to read