Fix stale reference in loadAllPushes() causing stuck push after controller restart

[haoxu07]

Summary

• Fix a bug in AbstractPushMonitor.loadAllPushes() where checkPushStatus() uses a stale loop variable instead of the refreshed object from topicToPushMap, causing batch pushes to get permanently stuck at END_OF_PUSH_RECEIVED after a controller restart.
• Add LoadAllPushesStaleReferenceTest to reproduce and verify the fix.

Problem

During controller STANDBY→LEADER transition, loadAllPushes():

1. Line 176: Bulk-loads OfflinePushStatus with partition statuses from ZK (snapshot T1)
2. Line 190: Registers ZK watchers for partition status changes
3. Line 195: updateOfflinePush() reads fresh data from ZK (snapshot T2) — all replicas now COMPLETED — and replaces the entry in topicToPushMap
4. Line 199-204: checkPushStatus() is called with the stale loop variable (T1), not the refreshed object (T2)

If replicas complete between T1 and T2, the stale object has incomplete partition statuses. Since no ZK watcher callbacks fire for already-completed partitions (their final write happened before watcher registration), the push is stuck at END_OF_PUSH_RECEIVED permanently, blocking all future batch pushes for the store.

Evidence

Observed in EI on 2026-04-03:

• heartbeat_inc_push_mt-0 v2420 on mt-0: all 60 partitions × 3 replicas COMPLETED in ZK, but controller's in-memory OfflinePushStatus.currentStatus = END_OF_PUSH_RECEIVED (confirmed via heap dump, ODP event 4075972)
• Controller logs showed DeferredVersionSwapService: "Skipping store as parent version 2420 status is PARTIALLY_ONLINE" for 8+ hours
• Required manual --kill-job to unblock

Fix

One-line change after updateOfflinePush():

offlinePushStatus = topicToPushMap.get(offlinePushStatus.getKafkaTopic());

This ensures checkPushStatus() evaluates the latest partition data from ZK.

Test plan

• LoadAllPushesStaleReferenceTest.testLoadAllPushesStaleReferenceAfterUpdateOfflinePush — reproduces the race condition (fails without fix, passes with fix)
• All existing AbstractPushMonitorTest inherited tests pass
• CI

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Fixes a controller failover race in AbstractPushMonitor.loadAllPushes() where checkPushStatus() could evaluate a stale OfflinePushStatus instance (loaded before watcher registration) instead of the refreshed object after updateOfflinePush(), leaving batch pushes stuck at END_OF_PUSH_RECEIVED after restart.

Changes:

• Refresh the offlinePushStatus loop variable from topicToPushMap immediately after updateOfflinePush() so subsequent status evaluation uses the latest ZK snapshot.
• Add LoadAllPushesStaleReferenceTest to reproduce the stale-reference scenario and verify the push transitions to COMPLETED.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
services/venice-controller/src/main/java/com/linkedin/venice/pushmonitor/AbstractPushMonitor.java	Re-reads the refreshed OfflinePushStatus from topicToPushMap after updateOfflinePush() to avoid using stale partition status data.
services/venice-controller/src/test/java/com/linkedin/venice/pushmonitor/LoadAllPushesStaleReferenceTest.java	Adds a regression test that simulates stale (T1) vs refreshed (T2) snapshots around watcher subscription and verifies completion + version swap.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.