Skip to content

[BUG] Cross Cluster Search to same node fails auth #5846

Open
#6474
Open
[BUG] Cross Cluster Search to same node fails auth#5846
#6474
Labels
bugSomething isn't workingtriagedIssues labeled as 'Triaged' have been reviewed and are deemed actionable.

Description

@dforste
dforste
opened on Dec 9, 2025

What is the bug?
Cross Cluster Search to same node fails authentication with:

How can one reproduce the bug?
Steps to reproduce the behavior:

  1. Create a cluster.
  2. Create a remote cluster with same node as seed.
  3. Search remote cluster.

What is the expected behavior?
Expect credentials to be passed and permissions to be honored.

What is your host/environment?

  • OS: Redhat/Docker Official image.
  • Version 3.3.0
  • Plugins

Do you have any additional context?
https://forum.opensearch.org/t/ccs-query-to-same-cluster-is-not-working/17995/4
Looks like only solution is to not auth for CCS which doesn't seem right.

Here is the stacktrace I observed:

[2025-12-09T19:24:02,331][WARN ][r.suppressed             ] [opensearch-1] path: /*%3Alogs/_search, params: {ignore_unavailable=true, preference=1765306631469, index=*:notifydis, timeout=30000ms, track_total_hits=true}
org.opensearch.transport.RemoteTransportException: [error while communicating with remote cluster [local]]
Caused by: org.opensearch.transport.RemoteTransportException: [opensearch-1][172.19.0.4:9300][indices:data/read/search]
Caused by: org.opensearch.OpenSearchSecurityException: No user found for indices:data/read/search
	at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:373) ~[?:?]
	at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:175) ~[?:?]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218) ~[opensearch-3.3.2.jar:3.3.2]
	at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:81) ~[?:?]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218) ~[opensearch-3.3.2.jar:3.3.2]
	at org.opensearch.action.support.TransportAction.execute(TransportAction.java:190) ~[opensearch-3.3.2.jar:3.3.2]
	at org.opensearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:133) ~[opensearch-3.3.2.jar:3.3.2]
	at org.opensearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:129) ~[opensearch-3.3.2.jar:3.3.2]
	at org.opensearch.security.ssl.transport.SecuritySSLRequestHandler.messageReceivedDecorate(SecuritySSLRequestHandler.java:209) ~[?:?]
	at org.opensearch.security.transport.SecurityRequestHandler.messageReceivedDecorate(SecurityRequestHandler.java:353) ~[?:?]
	at org.opensearch.security.ssl.transport.SecuritySSLRequestHandler.messageReceived(SecuritySSLRequestHandler.java:157) ~[?:?]
	at org.opensearch.security.OpenSearchSecurityPlugin$6$1.messageReceived(OpenSearchSecurityPlugin.java:939) ~[?:?]
	at org.opensearch.indexmanagement.rollup.interceptor.RollupInterceptor$interceptHandler$1.messageReceived(RollupInterceptor.kt:120) ~[?:?]
	at org.opensearch.performanceanalyzer.transport.PerformanceAnalyzerTransportRequestHandler.messageReceived(PerformanceAnalyzerTransportRequestHandler.java:44) ~[?:?]
	at org.opensearch.performanceanalyzer.transport.RTFPerformanceAnalyzerTransportRequestHandler.messageReceived(RTFPerformanceAnalyzerTransportRequestHandler.java:63) ~[?:?]
	at org.opensearch.wlm.WorkloadManagementTransportInterceptor$RequestHandler.messageReceived(WorkloadManagementTransportInterceptor.java:63) ~[opensearch-3.3.2.jar:3.3.2]
	at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:108) ~[opensearch-3.3.2.jar:3.3.2]
	at org.opensearch.transport.NativeMessageHandler.handleRequest(NativeMessageHandler.java:297) [opensearch-3.3.2.jar:3.3.2]
	at org.opensearch.transport.NativeMessageHandler.handleMessage(NativeMessageHandler.java:169) [opensearch-3.3.2.jar:3.3.2]
	at org.opensearch.transport.NativeMessageHandler.messageReceived(NativeMessageHandler.java:149) [opensearch-3.3.2.jar:3.3.2]
	at org.opensearch.transport.InboundHandler.messageReceivedFromPipeline(InboundHandler.java:152) [opensearch-3.3.2.jar:3.3.2]
	at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:144) [opensearch-3.3.2.jar:3.3.2]
	at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:804) [opensearch-3.3.2.jar:3.3.2]
	at org.opensearch.transport.InboundBytesHandler.forwardFragments(InboundBytesHandler.java:137) [opensearch-3.3.2.jar:3.3.2]
	at org.opensearch.transport.InboundBytesHandler.doHandleBytes(InboundBytesHandler.java:77) [opensearch-3.3.2.jar:3.3.2]
	at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:124) [opensearch-3.3.2.jar:3.3.2]
	at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:113) [opensearch-3.3.2.jar:3.3.2]
	at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:95) [transport-netty4-client-3.3.2.jar:3.3.2]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.125.Final.jar:4.1.125.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.125.Final.jar:4.1.125.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.125.Final.jar:4.1.125.Final]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) [netty-handler-4.1.125.Final.jar:4.1.125.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.125.Final.jar:4.1.125.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.125.Final.jar:4.1.125.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.125.Final.jar:4.1.125.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:107) ~[netty-codec-4.1.125.Final.jar:4.1.125.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.125.Final.jar:4.1.125.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.125.Final.jar:4.1.125.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.125.Final.jar:4.1.125.Final]
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1519) [netty-handler-4.1.125.Final.jar:4.1.125.Final]
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1377) [netty-handler-4.1.125.Final.jar:4.1.125.Final]
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1428) [netty-handler-4.1.125.Final.jar:4.1.125.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530) [netty-codec-4.1.125.Final.jar:4.1.125.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469) [netty-codec-4.1.125.Final.jar:4.1.125.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) [netty-codec-4.1.125.Final.jar:4.1.125.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.125.Final.jar:4.1.125.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.125.Final.jar:4.1.125.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.125.Final.jar:4.1.125.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357) [netty-transport-4.1.125.Final.jar:4.1.125.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.125.Final.jar:4.1.125.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.125.Final.jar:4.1.125.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868) [netty-transport-4.1.125.Final.jar:4.1.125.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.125.Final.jar:4.1.125.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:796) [netty-transport-4.1.125.Final.jar:4.1.125.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:697) [netty-transport-4.1.125.Final.jar:4.1.125.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:660) [netty-transport-4.1.125.Final.jar:4.1.125.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.125.Final.jar:4.1.125.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998) [netty-common-4.1.125.Final.jar:4.1.125.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.125.Final.jar:4.1.125.Final]
	at java.lang.Thread.run(Thread.java:1447) [?:?]

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingtriagedIssues labeled as 'Triaged' have been reviewed and are deemed actionable.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions