Skip to content

Fix[mqb]: Always call onHandleConfiguredDispatched - #1685

Merged
dorjesinpo merged 1 commit into
mainfrom
fix/deconfigure-while-converting
Aug 3, 2026
Merged

Fix[mqb]: Always call onHandleConfiguredDispatched#1685
dorjesinpo merged 1 commit into
mainfrom
fix/deconfigure-while-converting

Conversation

@dorjesinpo

Copy link
Copy Markdown
Collaborator
FATAL mqbblp_queue.cpp:72 Assertion failed: inDispatcherThread(), stack trace:
  (0): bmqAssertHandler(char const*, char const*, int)+0x84 at 0x555555fc5ae4 source:bmqbrkr.m.cpp in /usr/local/bin/bmqbrkr
  (1): BloombergLP::bsls::Assert::invokeHandler(BloombergLP::bsls::AssertViolation const&)+0x18 at 0x555556d21008 in /usr/local/bin/bmqbrkr
  (2): BloombergLP::mqbblp::Queue::onHandleDeconfigured(BloombergLP::bmqp_ctrlmsg::Status const&, BloombergLP::bmqp_ctrlmsg::StreamParameters const&,
  BloombergLP::mqbi::QueueHandle*)+0xef at 0x5555562a4e2f in /usr/local/bin/bmqbrkr
  (3): BloombergLP::bslma::SharedPtrOutofplaceRep<BloombergLP::mqbblp::RelayQueueEngine::ConfigureContext,
  BloombergLP::bslma::Allocator*>::disposeObject()+0x5f at 0x55555635dcef in /usr/local/bin/bmqbrkr
  (4): BloombergLP::bslma::SharedPtrRep::releaseRef()+0x25 at 0x555556d15f55 in /usr/local/bin/bmqbrkr
  (5): BloombergLP::bdlf::Bind_BoundTuple7<BloombergLP::mqbblp::RelayQueueEngine*, bsl::weak_ptr<BloombergLP::mqbblp::RelayQueueEngine>,
  BloombergLP::bdlf::PlaceHolder<1>, BloombergLP::bdlf::PlaceHolder<2>, BloombergLP::mqbi::QueueHandle*, BloombergLP::bmqp_ctrlmsg::StreamParameters,
  bsl::shared_ptr<BloombergLP::mqbblp::RelayQueueEngine::ConfigureContext> >::~Bind_BoundTuple7()+0x23 at 0x55555635e9b3 in /usr/local/bin/bmqbrkr
  (6): BloombergLP::bslstl::Function_Rep::ManagerRet BloombergLP::bslstl::Function_Rep::functionManager<BloombergLP::bdlf::Bind<BloombergLP::bslmf::Nil,
  void (BloombergLP::mqbblp::RelayQueueEngine::*)(bsl::weak_ptr<BloombergLP::mqbblp::RelayQueueEngine> const&, BloombergLP::bmqp_ctrlmsg::Status
  const&, BloombergLP::bmqp_ctrlmsg::StreamParameters const&, BloombergLP::mqbi::QueueHandle*, BloombergLP::bmqp_ctrlmsg::StreamParameters const&,
  bsl::shared_ptr<BloombergLP::mqbblp::RelayQueueEngine::ConfigureContext> const&),
  BloombergLP::bdlf::Bind_BoundTuple7<BloombergLP::mqbblp::RelayQueueEngine*, bsl::weak_ptr<BloombergLP::mqbblp::RelayQueueEngine>,
  BloombergLP::bdlf::PlaceHolder<1>, BloombergLP::bdlf::PlaceHolder<2>, BloombergLP::mqbi::QueueHandle*, BloombergLP::bmqp_ctrlmsg::StreamParameters,
  bsl::shared_ptr<BloombergLP::mqbblp::RelayQueueEngine::ConfigureContext> > >, false>(BloombergLP::bslstl::Function_Rep::ManagerOpCode,
  (7): BloombergLP::bslstl::Function_Rep::~Function_Rep()+0x2c at 0x555556d24edc in /usr/local/bin/bmqbrkr
  (8): BloombergLP::bslma::SharedPtrInplaceRep<BloombergLP::bdlf::Bind<BloombergLP::bslmf::Nil, void
  (BloombergLP::mqbblp::ClusterQueueHelper::*)(bsl::shared_ptr<BloombergLP::bmqp::RequestManagerRequest<BloombergLP::bmqp_ctrlmsg::ControlMessage,
  BloombergLP::bmqp_ctrlmsg::ControlMessage> > const&, BloombergLP::bmqt::Uri const&, BloombergLP::bmqp_ctrlmsg::StreamParameters const&, unsigned long
  long, bsl::function<void (BloombergLP::bmqp_ctrlmsg::Status const&, BloombergLP::bmqp_ctrlmsg::StreamParameters const&)> const&),
  BloombergLP::bdlf::Bind_BoundTuple6<BloombergLP::mqbblp::ClusterQueueHelper*, BloombergLP::bdlf::PlaceHolder<1>, BloombergLP::bmqt::Uri,
  BloombergLP::bmqp_ctrlmsg::StreamParameters, unsigned long long, bsl::function<void (BloombergLP::bmqp_ctrlmsg::Status const&,
  BloombergLP::bmqp_ctrlmsg::StreamParameters const&)> > > const>::disposeObject()+0x1c at 0x5555562425dc in /usr/local/bin/bmqbrkr
  (9): BloombergLP::bslma::SharedPtrRep::releaseRef()+0x25 at 0x555556d15f55 in /usr/local/bin/bmqbrkr
  (10): BloombergLP::bslstl::Function_Rep::ManagerRet
  BloombergLP::bslstl::Function_Rep::functionManager<BloombergLP::bdlf::BindWrapper<BloombergLP::bslmf::Nil, void
  (BloombergLP::mqbblp::ClusterQueueHelper::*)(bsl::shared_ptr<BloombergLP::bmqp::RequestManagerRequest<BloombergLP::bmqp_ctrlmsg::ControlMessage,
  BloombergLP::bmqp_ctrlmsg::ControlMessage> > const&, BloombergLP::bmqt::Uri const&, BloombergLP::bmqp_ctrlmsg::StreamParameters const&, unsigned long
  long, bsl::function<void (BloombergLP::bmqp_ctrlmsg::Status const&, BloombergLP::bmqp_ctrlmsg::StreamParameters const&)> const&),
  BloombergLP::bdlf::Bind_BoundTuple6<BloombergLP::mqbblp::ClusterQueueHelper*, BloombergLP::bdlf::PlaceHolder<1>, BloombergLP::bmqt::Uri,
  BloombergLP::bmqp_ctrlmsg::StreamParameters, unsigned long long, bsl::function<void (BloombergLP::bmqp_ctrlmsg::Status const&,
  BloombergLP::bmqp_ctrlmsg::StreamParameters const&)> > >, true>(BloombergLP::bslstl::Function_Rep::ManagerOpCode, BloombergLP::bslstl::Function_Rep*,
  void*)+0xde at 0x55555624428e in /usr/local/bin/bmqbrkr
  (11): BloombergLP::bslstl::Function_Rep::makeEmpty()+0x25 at 0x555556d25125 in /usr/local/bin/bmqbrkr
  (12): BloombergLP::bmqp::RequestManagerRequest<BloombergLP::bmqp_ctrlmsg::ControlMessage, BloombergLP::bmqp_ctrlmsg::ControlMessage>::clear()+0x6d at
  0x5555560950ed in /usr/local/bin/bmqbrkr
  (13): BloombergLP::bmqp::RequestManagerRequest<BloombergLP::bmqp_ctrlmsg::ControlMessage,
  BloombergLP::bmqp_ctrlmsg::ControlMessage>::~RequestManagerRequest()+0x33 at 0x5555560951e3 in /usr/local/bin/bmqbrkr
  (14): BloombergLP::bslma::SharedPtrRep::releaseRef()+0x56 at 0x555556d15f86 in /usr/local/bin/bmqbrkr
  (15): BloombergLP::bmqp::RequestManager<BloombergLP::bmqp_ctrlmsg::ControlMessage,
  BloombergLP::bmqp_ctrlmsg::ControlMessage>::processResponse(BloombergLP::bmqp_ctrlmsg::ControlMessage const&)+0x2dd at 0x5555561a6aad in
  /usr/local/bin/bmqbrkr
  (16): BloombergLP::mqbblp::Cluster::processResponseDispatched(BloombergLP::bmqp_ctrlmsg::ControlMessage const&, BloombergLP::mqbnet::ClusterNode*)+0x92
  at 0x55555617a752 in /usr/local/bin/bmqbrkr
  (17): BloombergLP::mqba::Dispatcher::EventCallback::operator()(bsl::shared_ptr<BloombergLP::mqbi::DispatcherEvent> const&)+0x183 at 0x5555560fb153 in
  /usr/local/bin/bmqbrkr
  (18): BloombergLP::bmqc::MultiQueueThreadPool<BloombergLP::mqbi::DispatcherEvent>::processQueue(BloombergLP::bmqc::MultiQueueThreadPool<BloombergLP::mq
  bi::DispatcherEvent>::QueueInfo&)+0xd1 at 0x55555610e0f1 in /usr/local/bin/bmqbrkr
  (19): BloombergLP::bdlmt::ThreadPool::workerThread()+0x345 at 0x555556cc19b5 in /usr/local/bin/bmqbrkr
  (20): ThreadPoolEntry+0xd at 0x555556cc1a8d in /usr/local/bin/bmqbrkr
  (21): --unknown-- at 0x7fffff2b6aa4 in /lib/x86_64-linux-gnu/libc.so.6
  (22): __clone+0x44 at 0x7fffff343a64 in /lib/x86_64-linux-gnu/libc.so.6

⏺ Root cause

Queue::onHandleDeconfigured (mqbblp_queue.cpp:72) asserts it's running on the queue's own dispatcher thread. But it's being invoked as a side effect of
a shared_ptrRelayQueueEngine::ConfigureContext destructing on the cluster dispatcher thread instead.

Chain of events:

  1. RelayQueueEngine::configureApp (mqbblp_relayqueueengine.cpp:868-879) binds a ConfigureContext (whose destructor unconditionally fires a completion
    callback, mqbblp_relayqueueengine.h:629-656) into the response callback of an upstream configure-queue request, along with a weak_ptr to the engine
    itself.
  2. That callback ends up stored in a RequestManagerRequest (mqbblp_clusterqueuehelper.cpp:3705-3746).
  3. Meanwhile the RelayQueueEngine gets destroyed (e.g. queue GC / Local↔Remote conversion) — ~RelayQueueEngine() invalidates only strong self-refs, not
    the weak one held by the in-flight request, so this race is expected/normal.
  4. When the response arrives, it's processed on the cluster dispatcher thread (Cluster::processResponseDispatched, mqbblp_cluster.cpp:1913).
    RelayQueueEngine::onHandleConfigured (mqbblp_relayqueengine.cpp:327-357) runs, self.lock() fails (engine is gone), and it does an early return — without
    re-dispatching context onto the queue's own thread the way the success path does (onHandleConfiguredDispatched).
  5. That leaves the last shared_ptr reference held only by the request's bound response-callback functor. When
    RequestManagerRequest::~RequestManagerRequest() (bmqp_requestmanager.h:947-966) tears down and releases it — still on the cluster thread —
    ~ConfigureContext() fires synchronously, calls its stored callback, which reaches Queue::onHandleDeconfigured, and trips
    BSLS_ASSERT_SAFE(inDispatcherThread()).

Signed-off-by: dorjesinpo <129227380+dorjesinpo@users.noreply.github.com>
@dorjesinpo
dorjesinpo requested a review from a team as a code owner August 3, 2026 19:19
@dorjesinpo dorjesinpo changed the title Always call onHandleConfiguredDispatched Fix[mqb]: Always call onHandleConfiguredDispatched Aug 3, 2026
@dorjesinpo
dorjesinpo requested review from 678098 and removed request for a team August 3, 2026 19:38
@dorjesinpo
dorjesinpo merged commit 83f776c into main Aug 3, 2026
66 of 67 checks passed
@dorjesinpo
dorjesinpo deleted the fix/deconfigure-while-converting branch August 3, 2026 20:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants