Stabilize the flaky integration tests that false-fail in-flight PR CI (#516, #525)

[eduralph]

Surfaced repeatedly while shepherding tikv/client-rust#549 and tikv/client-rust#550: two pre-existing upstream flakes keep red-Xing our PRs' integration-test-txn job, each needing manual diagnosis + retrigger.

• tests: failpoint_tests::txn_cleanup_2pc_locks is flaky tikv/client-rust#516 — failpoint_tests::txn_cleanup_2pc_locks hangs until nextest's terminate kills it (SIGKILL/timeout). Hit #549; retrigger cleared it. The success-path wait is identical to the old try_join_all, so it is not caused by our changes.
• tests: "txn_cleanup_locks_batch_size" is flaky tikv/client-rust#525 family — txn_cleanup_range_async_commit_locks / txn_cleanup_locks_batch_size assert an exact count_locks that races (observed 513 vs 512, 575 vs 512). Hit #550; retrigger cleared it.

Both pass reliably in isolation locally (3/3, sub-second). The recurring cost is human triage + CI retriggers on every in-flight PR.

Scope: diagnose the shared root cause (async lock-cleanup timing under the aggressive region-split config) and either fix the tests' synchronization or file a tightened upstream repro. Relates to backlog #35 (failpoint flakiness, Phase 3) — pulled to Phase 0 because it is actively blocking the in-flight PRs.

Roadmap §3 / Phase 0 — unblocking in-flight work.

[eduralph]

Diagnosed and fixed. Root cause: the txn_cleanup_* tests counted locks over the whole keyspace, so the async-commit locks left draining by txn_cleanup_locks_batch_size inflated a later serial test's count (#525's over-counts) and amplified cleanup_locks past the timeout (#516). The fix scopes every lock count to the test's own keys and has the residue-leaving test clean up after itself — test-only, no src/ change.

Upstream PR: tikv/client-rust#551 (Fixes #525; left #516 open in case its hang has a deeper cause beyond the residue amplifier). Confirmed stable across 10+ multi-region runs locally; codex-reviewed clean.

[eduralph]

Splitting this issue's two flakes — they are not the same bug.

#525 (count over-counts) — FIXED by tikv/client-rust#551. Root cause confirmed: async-commit locks left draining by txn_cleanup_locks_batch_size inflated a later serial test's whole-keyspace count_locks. The fix scopes every count to the test's own keys.

#516 (the 600s txn_cleanup_2pc_locks timeout) — STILL OPEN, and #551 does not address it. My earlier reasoning that #551 would mitigate it was wrong: txn_cleanup_2pc_locks runs first in the serial suite (consistently reported ( 1/54 )), so no residue from the other cleanup tests can reach it. #551's body has been corrected accordingly.

I could not reproduce #516 locally — single-node and 3-node api-v2 clusters matching CI's --kv 3 topology, cold-started, and CPU-constrained to roughly a runner's budget; ~20 runs, all 1–2s with every phase in milliseconds. Ruled out along the way: ensure_region_split (capped at 15s), wait_for_locks_count (capped at 15s), and an unbounded CleanupLocks loop (cursor advances; region errors propagate to a bounded-backoff retry).

Rather than ship a speculative fix, tikv/client-rust#553 makes the next occurrence diagnosable: phase markers whose entry line survives the kill and names the stalled step, plus setup-cost timing. When CI next captures a stall, the offending phase should be obvious.

This issue stays open until #516 is actually understood and fixed.