Fix[mqb]: revisiting open queue failures handling

[dorjesinpo]

Upon open queue failure, either buffer the request or retry immediately, or respond

[678098]

Suggested change

	// Do retry if the state is k_OPEN && isQueuePrimaryAvailable
	bool retryNow = subQueueContext.d_state == SubQueueContext::k_OPEN
	? isQueuePrimaryAvailable(*qcontext, otherThan)
	: false;
	bool retryNow = (subQueueContext.d_state == SubQueueContext::k_OPEN)
	&& isQueuePrimaryAvailable(*qcontext, otherThan);

Possible to replace ternary operator with operations mentioned in the comment.
This way, the code becomes self-explanatory, and the comment duplicating behaviour can be removed.

[678098]

To be precise, use_count only shows the number of shared_ptr copies, not the number of pending requests. This might be misleading during debug.

For example, there might be 1 shared_ptr with 3 copies, numPendingReopenQueueRequests() == 3 which is not true. We might spend time investigating "where are these 3 reopen requests?", when in reality we only have 1.

Binary +1 or +0 makes more sense here:

Suggested change

	sum += cit->second.use_count();
	sum += static_cast<int>(cit->second.use_count() > 0);

or

Suggested change

	sum += cit->second.use_count();
	sum += static_cast<int>(!cit->second.expired());

[dorjesinpo]

Using cit->second.use_count() > 0) would return number of partitions being reopened.
The use_count can have the error margin of +1 - upon reopen response we may bind the cycle to reconfigure or retry callback before releasing. It is like having two requests for the same queue - one is completing and another is starting.
But. Since numPendingReopenQueueRequests() is executed in the cluster thread - where we process reopen responses - the "completing" count should be gone and the number should be precise.

[678098]

I think that we should also check that the current d_reopenCycle[partitionId] has the same primary lease id as cycle before deleting.
What if we free the last PartitionReopenCycle from the previous primary at the time when we have a new primary and another partition cycle for the same partition id?

[dorjesinpo]

This runs in the cluster thread only. We setAsFailure if generationCounts mismatch). Not sure about checking generationCount - 1) there is no shared_ptr anymore; 2) the actual ClusterStatePartitionInfo::primaryLeaseId() could change before we start new cycle.

[678098]

If it runs on cluster thread, let's add a comment

// Thread: CLUSTER dispatcher

And add a precondition

BSLS_ASSERT_SAFE(d_cluster_p->inDispatcherThread());

[678098]

Suggested change

	cycle.reset(new (*d_allocator_p)
	PartitionReopenCycle(this, generationCount, partitionId),
	d_allocator_p);
	cycle = bsl::allocate_shared<PartitionReopenCycle>(d_allocator_p, this, generationCount, partitionId);