Skip to content

Fix[MQBC]: skip FSM events if primary goes down - #1193

Merged
678098 merged 1 commit into
bloomberg:mainfrom
678098:260313_fix_nodeId_assert
Mar 25, 2026
Merged

Fix[MQBC]: skip FSM events if primary goes down#1193
678098 merged 1 commit into
bloomberg:mainfrom
678098:260313_fix_nodeId_assert

Conversation

@678098

@678098 678098 commented Mar 13, 2026

Copy link
Copy Markdown
Collaborator
  1. Add missing state transition UNKNOWN -> RST_PRIMARY -> UNKNOWN
    This updates primary in partition thread.

  2. In FSM mode, update primary only from FSM.
    Keep directly calling storageManager->setPrimaryForPartition and storageManager->clearPrimaryForPartition in legacy mode.

  3. Skip enqueueing an event to FSM if primaryLeaseId is outdated or there is a primary mismatch.

  4. Update UT to catch new "Primary mismatch" error.

@678098
678098 requested a review from a team as a code owner March 13, 2026 19:53
@678098
678098 force-pushed the 260313_fix_nodeId_assert branch from 8ec0af2 to d1dc15c Compare March 13, 2026 19:56
@678098
678098 requested a review from kaikulimu March 13, 2026 20:05
@678098
678098 force-pushed the 260313_fix_nodeId_assert branch 2 times, most recently from b898ca8 to 985570f Compare March 16, 2026 21:17
Comment thread src/groups/mqb/mqbc/mqbc_clusterstatetable.h
@678098
678098 force-pushed the 260313_fix_nodeId_assert branch from 985570f to 8e2a519 Compare March 17, 2026 15:25
kaikulimu
kaikulimu previously approved these changes Mar 17, 2026
@678098
678098 force-pushed the 260313_fix_nodeId_assert branch 11 times, most recently from 8266fae to 236a881 Compare March 18, 2026 20:10
<< d_partitionInfoVec[partitionId].primaryLeaseId();

bmqp_ctrlmsg::ControlMessage controlMsg;
controlMsg.rId() = evt.requestId();

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The event may not be a request in which case there is no need to build and send a response.

@678098
678098 force-pushed the 260313_fix_nodeId_assert branch 6 times, most recently from 2ec5435 to d54459a Compare March 20, 2026 18:51
failureResponse.code() = mqbi::ClusterErrorCode::e_NOT_REPLICA;
failureResponse.message() = "Not a replica";
failureResponse.code() = mqbi::ClusterErrorCode::e_UNKNOWN;
failureResponse.message() = "Primary mismatch";

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test was failing due to the early reject mechanism.
I have updated the expected output.
The error log from UT:

TEST /home/runner/work/blazingmq/blazingmq/src/groups/mqb/mqbc/mqbc_storagemana 
ger.t.cpp CASE 4                                                                  
20MAR2026_18:33:15.939 18823:140470192740224 WARN /home/runner/work/blazingmq/bla 
zingmq/src/groups/mqb/mqbc/mqbc_storagemanager.cpp:401 MQBC.STORAGEMANAGER        
Cluster (testCluster) Partition [0]: dropping stale event: source [W1, 3], event  
primaryLeaseId [1], current primaryLeaseId [1], event primary [[W1, 3]], current  
primary [[E2, 2]]                                                                 
Error /home/runner/work/blazingmq/blazingmq/src/groups/mqb/mqbc/mqbc_storagemanag 
er.t.cpp(593): 'message' ([ rId = 1 choice = [ status = [ category = E_REFUSED    
code = -10 message = "Primary mismatch" ] ] ]) == 'failureMessage' ([ rId = 1     
choice = [ status = [ category = E_REFUSED code = -207 message = "Not a replica"  
] ] ])    (failed)  

@678098
678098 force-pushed the 260313_fix_nodeId_assert branch from d54459a to 3ad83b2 Compare March 20, 2026 19:08
@678098
678098 requested a review from kaikulimu March 20, 2026 19:08
Signed-off-by: Evgeny Malygin <emalygin@bloomberg.net>
@kaikulimu

Copy link
Copy Markdown
Collaborator

A splendidly crafted PR!

@678098
678098 merged commit c0e9830 into bloomberg:main Mar 25, 2026
43 of 45 checks passed
@678098
678098 deleted the 260313_fix_nodeId_assert branch March 25, 2026 18:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants