Intermittent connection refused errors

[toboid]

I have started getting the following error when running StormDocker. It was working fine, now i've started getting the following error in StormUI. It works intermittently if i refresh the page a couple of times.

I'm seeing similar errors in the supervisor log file.

I've tried re-pulling all of the images down and restarting the docker service but this issue keeps coming and going whatever I do.

I'm running on Ubuntu 12.04 64bit. Any ideas?

org.apache.thrift7.transport.TTransportException: java.net.ConnectException: Connection refused
at org.apache.thrift7.transport.TSocket.open(TSocket.java:183)
at org.apache.thrift7.transport.TFramedTransport.open(TFramedTransport.java:81)
at backtype.storm.thrift$nimbus_client_and_conn.invoke(thrift.clj:75)
at backtype.storm.ui.core$cluster_summary.invoke(core.clj:455)
at backtype.storm.ui.core$fn__8223.invoke(core.clj:789)
at compojure.core$make_route$fn__3365.invoke(core.clj:93)
at compojure.core$if_route$fn__3353.invoke(core.clj:39)
at compojure.core$if_method$fn__3346.invoke(core.clj:24)
at compojure.core$routing$fn__3371.invoke(core.clj:106)
at clojure.core$some.invoke(core.clj:2443)
at compojure.core$routing.doInvoke(core.clj:106)
at clojure.lang.RestFn.applyTo(RestFn.java:139)
at clojure.core$apply.invoke(core.clj:619)
at compojure.core$routes$fn__3375.invoke(core.clj:111)
at ring.middleware.reload$wrap_reload$fn__7540.invoke(reload.clj:14)
at backtype.storm.ui.core$catch_errors$fn__8268.invoke(core.clj:858)
at ring.middleware.keyword_params$wrap_keyword_params$fn__4029.invoke(keyword_params.clj:27)
at ring.middleware.nested_params$wrap_nested_params$fn__4068.invoke(nested_params.clj:65)
at ring.middleware.params$wrap_params$fn__4001.invoke(params.clj:55)
at ring.middleware.multipart_params$wrap_multipart_params$fn__4096.invoke(multipart_params.clj:103)
at ring.middleware.flash$wrap_flash$fn__4277.invoke(flash.clj:14)
at ring.middleware.session$wrap_session$fn__4266.invoke(session.clj:43)
at ring.middleware.cookies$wrap_cookies$fn__4197.invoke(cookies.clj:160)
at ring.adapter.jetty$proxy_handler$fn__7179.invoke(jetty.clj:16)
at ring.adapter.jetty.proxy$org.mortbay.jetty.handler.AbstractHandler$0.handle(Unknown Source)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:385)
at java.net.Socket.connect(Socket.java:546)
at org.apache.thrift7.transport.TSocket.open(TSocket.java:178)

[toboid]

here is what i get in the supervisor.log

2014-11-30 15:55:01 o.a.c.f.i.CuratorFrameworkImpl [INFO] Starting
2014-11-30 15:55:01 o.a.z.ZooKeeper [INFO] Initiating client connection, connectString= sessionTimeout=20000 watcher=org.apache.curator.ConnectionState@
2014-11-30 15:55:01 o.a.z.ClientCnxn [INFO] Opening socket connection to server stormdocker_zookeeper_1/. Will not attempt to authenticate using SASL (unknown error)
2014-11-30 15:55:01 o.a.z.ClientCnxn [WARN] Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.6.0_32]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:601) ~[na:1.6.0_32]
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350) ~[zookeeper-3.4.5.jar:3.4.5-1392090]
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068) ~[zookeeper-3.4.5.jar:3.4.5-1392090]

[igorfrenkel]

+1. I'm also seeing these. Intermittently.

[toboid]

Restarting the box seems to clear it for me.

[sudeepd]

The nimbus (and others) IP changes when you restart the containers. The UI 's storm.yaml still points to the old nimbus IP.

[lq08025107]

got the same error, not Intermittenly, always

[lq08025107]

In my case, nimbus's storm.yaml's zookeeper address is 172.17.0.1, but you know that is wrong, this address is the docker0's address, I don't know why.........