broker-client-shared-internal-executor issue

[programmerahul]

I see that broker-client-shared-internal-executor thread is used to load system topic policies in SystemTopicBasedTopicPoliciesService.java while doing initPolicesCache. After loading the topic policies, same thread is used to call listener.onUpdate ( to update the topic policy). But internally listener.onUpdate operation is a blocking operation. onUpdate(PersistentTopic) ->
applyUpdatedTopicPolicies ->
checkMessageExpiry->
sub.expireMessages->
getNumberOfEntriesInBacklog->
topic.isOldestMessageExpired->
cursor.getNthEntry->
counter.await(ledger.getConfig().getMetadataOperationsTimeoutSeconds(), TimeUnit.SECONDS);
Here the default operation timeout is 1minute.
That means, if there is a single topic for which the oldest ledger is not accessible for the time being, it will block loading all other topic for a significant time, depending on the number of subscription present on the topic, eg. if there are 20 subscription on the topic , then policies loading of other system-topics can be delayed by 20mins(This happens due to recursive loading of policies event, eg. an ns with 20 topics, has to complete 20 callbacks one after other, to load the topic policy) I have observed this in out heap dump

[lhotari]

@programmerahul Thanks for bringing this up. Please file an issue at https://github.com/apache/pulsar/issues/new/choose