[manager] make CacheReclaimer deletion asynchronous and prevent over-eviction

[shaohuaxi]

Background

Before this change, CacheReclaimer submitted a Location deletion by synchronously performing metadata Get, CAS, and Sync. These operations may access a remote metadata backend such as Redis, so a slow metadata operation could block the Reclaimer cron thread and delay reclamation work for every Instance Group handled by that loop.

Deletion may also be delayed by delay_before_delete_ms. During that interval, official storage usage and key count have not decreased yet, so subsequent cron iterations continue to observe an exceeded watermark and select more victims. The accumulated submissions can evict substantially more cache data than required. Making admission asynchronous also creates an accepted-before-CAS window in which a Location still appears selectable, so in-flight work must be accounted for and deduplicated immediately after acceptance.

Summary

• Add a true end-to-end asynchronous Location deletion API while preserving the existing synchronous Submit interface.
• Move metadata Get/CAS/Sync, delayed physical deletion, and final CAD onto SchedulePlanExecutor workers; delayed waits use the timer queue and do not occupy a worker.
• Track instance-scoped pending Locations to close the accepted-before-CAS duplicate-submission window.
• Credit in-flight delete bytes by Instance Group and base storage type, plus conservatively predicted deleted keys, so delayed deletion does not cause repeated over-eviction.
• Add credit deadlines, bounded per-Group/type and process-level backpressure, and no-progress polling backoff.
• Document the lifecycle, failure semantics, configuration defaults, and observability.

Key behavior

• SubmitAsync returns accepted=false with no valid Future when the initial enqueue is rejected; Reclaimer state is created only after acceptance.
• Every accepted request has one end-to-end Future. Worker failure, exception, secondary enqueue failure, shutdown cancellation, and normal completion all converge on an exactly-once promise completion path.
• delay_before_delete_ms starts after metadata Sync succeeds.
• Pending identity includes instance_id, block key, and location id, preserving strict Instance isolation.
• Delete bytes use the same URI-size accounting as official usage, with VCNS_HF3FS normalized to HF3FS; all effective-watermark subtraction is saturating.
• A credit timeout stops bytes/key-count deduction but intentionally retains pending identity and hard-limit quota until the Future reaches a terminal state.
• A saturated Group/type only blocks that type, while process-level limits stop all new admission.
• Watermark excess without an accepted non-empty request uses the normal polling interval instead of spinning.

Validation

All compilation and tests were run in a dedicated Linux container.

bazelisk test --jobs=1 --runs_per_test=10 --cache_test_results=no \
  //kv_cache_manager/manager/test:SchedulePlanExecutorTest \
  //kv_cache_manager/manager/test:CacheReclaimerTest \
  //kv_cache_manager/service/test:ServerConfigTest

Results:

• SchedulePlanExecutorTest: 16 tests, target passed 10/10 runs.
• CacheReclaimerTest: 75 tests, target passed 10/10 runs.
• ServerConfigTest: 3 tests, target passed 10/10 runs.
• //kv_cache_manager:kv_cache_manager_bin: built successfully with --jobs=1.
• C/C++ changed-line formatting, Python formatting for the separately exercised integration harness, and git diff --check: passed.

[oldsharp]

A high-quality, well-tested change with a clear design rationale.

Two questions to concern in this round of review:

1. about the SubmitAsync()'s accepting task type;
2. about the integration test implementation.

See the inline comments for more detail.

[oldsharp]

Should SubmitAsync accept CacheMetaDelRequest too?

No such use case under the current code base, but the case is reasonable.

CacheMetaDelRequest has the same problem the V1 async design was built to solve: Get/CAS/Sync may hit remote Redis and block the caller thread. And by making SubmitAsync accept CacheMetaDelRequest type task can let it align better to the other two Submit variants.

[shaohuaxi]

Thanks for reviewing.
Addressed by adding SubmitAsync(const CacheMetaDelRequest&) and reusing the same admission, delayed physical deletion, shutdown cancellation, and exactly-once Future completion pipeline as CacheLocationDelRequest. Existing synchronous callers remain unchanged. Unit tests cover worker execution, CLS_DELETING idempotency, failures, cancellation, and enqueue rejection.

[oldsharp]

Strong integration test verification design; would be better to add corresponding test implementations (similar to what #161 did in integration_test/reclaimer/reclaiming_test.py).

[shaohuaxi]

Addressed. Added Reclaimer integration tests covering byte/key-count credit under large delete delay, multiple Instances in the same Group, Future-state metric cleanup, and no-progress backoff. The multi-Instance case now uses metric polling instead of fixed sleep and passed 10 consecutive runs on the 193 test environment.