AsyncAppender-Worker-query-log-asnc threads still alive after stopping the web application on tomcat

[jkhlam]

TsdMetricsFactory.newInstance() starts threads named [AsyncAppender-Worker-query-log-async]. When I stop the application on tomcat:

(1) From catalina log, there are two SEVERE message indicating that it has failed to stop a thread:
SEVERE: The web application [/somewebapp] appears to have started a thread named [AsyncAppender-Worker-query-log-async] but has failed to stop it. This is very likely to create a memory leak.
(2) From JConsole, method com.arpnetworking.metrics.ch.qos.logback.core.AsyncAppenderBase$Worker.run() is still active, and there are two threads with name "AsyncAppender-Worker-query-log-asnc" running.

Steps to reproduce:
(1) Use TdsMetricsFactory.newInstance() to create a new MetricsFactory when the servlet starts.
(2) Deploy and start the web application
(3) Without using the MetricsFactory, Immediately stop the deployed web application.

[vjkoskela]

Thanks for the bug report! We don't unfortunately have a lot of use cases with servlet containers; most of our Java installations are into embedded web servers (e.g. Jetty, Akka, Play, Vert.x). So please bear with us.

Can you confirm which version of the client you are using? The thread in question is from Logback and we have a shutdown hook with the JVM to close the context. You can see that here:

https://github.com/ArpNetworking/metrics-client-java/blob/master/src/main/java/com/arpnetworking/metrics/impl/BaseFileSink.java#L166

In your repro steps you mention that the MetricsFactory instance was not used; are the threads shutdown if the instance is used?

Can you elaborate on how you stop the deployed web application? Are you unloading the servlet? Shutting down the web server? (I'm guessing not the latter as you state you are able to observe the threads post-stop)

Two final questions:

1. If you restart your servlet on the same web server how many threads are there?

2. If we provide an explicit shutdown mechanism for the MetricsFactory instance, how are you planning to invoke it from the container lifecycle (we need to be able to repro the fix).

Thanks,
Ville