java.lang.OutOfMemoryError: unable to create new native thread

[slaskawi]

Hi!

When I try to run the Hawkular APM on my local OpenShift cluster [1] using the following snippet:

oc create -f https://raw.githubusercontent.com/jboss-dockerfiles/hawkular-apm/master/openshift-templates/hawkular-apm-server-deployment.yml

The Pods are failing with this error in logs:

oc logs hawkular-apm-1-srh9n
...
14:19:44,597 INFO  [org.jboss.weld.Bootstrap] (Weld Thread Pool -- 4) WELD-001125: Illegal bean type org.hawkular.apm.server.api.services.Cache<org.hawkular.apm.server.api.services.Property<?>> ignored on [EnhancedAnnotatedTypeImpl] public @Singleton class org.hawkular.apm.server.infinispan.InfinispanPropertyCache
14:19:44,671 INFO  [org.jboss.weld.Bootstrap] (Weld Thread Pool -- 2) WELD-001125: Illegal bean type org.hawkular.apm.server.api.services.Cache<org.hawkular.apm.server.api.services.Property<?>> ignored on [EnhancedAnnotatedTypeImpl] public @Singleton class org.hawkular.apm.server.infinispan.InfinispanPropertyCache
14:19:44,988 WARN  [org.jboss.weld.Validator] (MSC service thread 1-2) WELD-001471: Interceptor method init defined on class org.hawkular.apm.server.elasticsearch.ElasticsearchClient is not defined according to the specification. It should not throw java.lang.Exception, which is a checked exception.
        at org.hawkular.apm.server.elasticsearch.ElasticsearchClient.init(ElasticsearchClient.java:165)
  StackTrace
14:19:44,988 WARN  [org.jboss.weld.Validator] (MSC service thread 1-4) WELD-001471: Interceptor method init defined on class org.hawkular.apm.server.elasticsearch.ElasticsearchClient is not defined according to the specification. It should not throw java.lang.Exception, which is a checked exception.
        at org.hawkular.apm.server.elasticsearch.ElasticsearchClient.init(ElasticsearchClient.java:165)
  StackTrace
14:19:45,330 INFO  [org.elasticsearch.plugins] (ServerService Thread Pool -- 72) [Aragorn] loaded [], sites []
14:19:45,354 ERROR [org.apache.activemq.artemis.core.client] (Thread-5 (ActiveMQ-remoting-threads-ActiveMQServerImpl::serverUUID=58820dc6-c6bf-11e6-a41d-7f3dd349aa5d-55856758-1323698166)) AMQ214017: Caught unexpected Throwable: java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method)
        at java.lang.Thread.start(Thread.java:714)
        at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
        at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1357)
        at org.apache.activemq.artemis.utils.OrderedExecutorFactory$OrderedExecutor.execute(OrderedExecutorFactory.java:85)
        at org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection.write(InVMConnection.java:163)
        at org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection.write(InVMConnection.java:151)
        at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.send(ChannelImpl.java:259)
        at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.send(ChannelImpl.java:201)
        at org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.doConfirmAndResponse(ServerSessionPacketHandler.java:579)
        at org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.access$000(ServerSessionPacketHandler.java:116)
        at org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler$1.done(ServerSessionPacketHandler.java:561)
        at org.apache.activemq.artemis.core.persistence.impl.journal.OperationContextImpl.executeOnCompletion(OperationContextImpl.java:161)
        at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.afterCompleteOperations(JournalStorageManager.java:666)
        at org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.sendResponse(ServerSessionPacketHandler.java:546)
        at org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.handlePacket(ServerSessionPacketHandler.java:531)
        at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:567)
        at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:349)
        at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:331)
        at org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:605)
        at org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1.run(InVMConnection.java:171)
        at org.apache.activemq.artemis.utils.OrderedExecutorFactory$OrderedExecutor$ExecutorTask.run(OrderedExecutorFactory.java:100)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

14:19:45,354 ERROR [org.apache.activemq.artemis.core.client] (Thread-21 (ActiveMQ-remoting-threads-ActiveMQServerImpl::serverUUID=58820dc6-c6bf-11e6-a41d-7f3dd349aa5d-55856758-1323698166)) AMQ214017: Caught unexpected Throwable: java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method)
        at java.lang.Thread.start(Thread.java:714)
        at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
        at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1357)
        at org.apache.activemq.artemis.utils.OrderedExecutorFactory$OrderedExecutor.execute(OrderedExecutorFactory.java:85)
        at org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection.write(InVMConnection.java:163)
        at org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection.write(InVMConnection.java:151)
        at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.send(ChannelImpl.java:259)
        at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.send(ChannelImpl.java:201)
        at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQPacketHandler.handleCreateSession(ActiveMQPacketHandler.java:185)
        at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQPacketHandler.handlePacket(ActiveMQPacketHandler.java:76)
        at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:567)
        at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:349)
        at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:331)
        at org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:605)
        at org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1.run(InVMConnection.java:171)
        at org.apache.activemq.artemis.utils.OrderedExecutorFactory$OrderedExecutor$ExecutorTask.run(OrderedExecutorFactory.java:100)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

Could you please have a look into that?

Thanks,
Sebastian

[1] OpenShift versions:

oc version
oc v1.3.0-alpha.3
kubernetes v1.3.0+507d3a7
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://192.168.0.17:8443
openshift v1.3.0-alpha.3
kubernetes v1.3.0+507d3a7

[jpkrohling]

What is the host OS you are using? We are aware of similar issues on Fedora 24 when using Docker from the Fedora repository. Using Docker from Docker's repository is known to work on this case.

The Docker as provided by Fedora 25 is known to work as well.

[slaskawi]

Exactly... I'm running F24 and installed Docker from Fedora repository:

> rpm -q docker 
docker-1.10.3-55.gite03ddb8.fc24.x86_64

So it seems I need to install Docker from the official Docker repository.

[jpkrohling]

That's correct. Or Fedora 25 :)