Home

Connecting to a remote JVM Process via JMX interface using ssh tunneling

Scenario :

Suppose you have JVM processes or a server running on a remote host and that network is private or restricted to only ssh access, and say we had to connect to the JVM to debug or to inspect anything. We would use something like JVisualVM and try to connect to the JVM via the JMX interface. With the above scenario, as it comes out-of-the-box, it is not really straight-forward to achieve a working setup and this a hands-on tutorial walks through the processing of setting this up.

Constraints and Challenges

The JVM process is running on a remote host in remote network
Only access to the network is via ssh via bastion/jumper host with the network
- client is--> |F| <--> bastion-host <--> server
Only way we can reach the intented destination hosts is via ssh
We cannot ssh directly to the destination-host and have to hop over the bastion/jumper host
We need to connect to the java process via jvisualvm for debugging, monitoring and other operations. Basically the use-case could be to do anything via JMX interface to a java process
We cannot run the jvisualvm on the remote/destination host as this is network intensive and may not privileges. The bandwidth required for this locality is not practical for operations
The default/standard way of using JMX via RMI does not work as RMI protocol cannot be tunneled. It is an arcance non-OSI 'layer' compliant protocol
We need the data over the network to be secure ( security provided by ssh )

Solution

The network connectivity problems are solved using powerful networking primitives which exploit the OSI layering abstractions. These features are all provided by SSH. We will use two main features of SSH to achive this
- SSH Local Port Forwarding : SSH port-forwarding primitives provide us means to communicate with abitrary ports on arbitrary networks over an uniform SSH connection
- SSH transparent proxying using ProxyCommand and netcat : Provides us means to transparently proxy the ssh-tunnel across mutliple host hops.
Standard JMX works over RMI protocol which is not a transparent layered protocol and hence cannot be tunneled across networks. It requires special knowledge of hosts and has unnecessay complex control protocol which is arcane. Fortunately there is another protocol upon which JMX can be interface called JMXMP. Unfortunately, it does not come standard out-of-the-box and requires some non-trivial setup. But once you have set this up on the server once, it is does not require any more special effort

Environment For Walkthrough

myhost.naboo.org |F| <- WAN -> |F| bastion.deathstar.org <--> bastion.deathstar.org

.

$myhost.naboo>netcat -vz bastion.deathstar.org 22
bastion.deathstar.org [178.85.206.132] 22 (ssh) open

$myhost.naboo>netcat -vz node-01.deathstat.org 22
Error: Couldn't resolve host "node-01.deathstat.org" ( or timeout )

$myhost.naboo>netcat -vz node-01.deathstat.org 59991
Error: Couldn't resolve host "node-01.deathstat.org" ( or timeout )

$bastion.deathstat>netcat -vz node-01.deathstat.org 22
node-01.deathstat.org [10.0.0.132] 22 (ssh) open

$node-01.deathstar> lsof -i TCP -n -P
sshd      2179      root   3u  IPv4    13812      0t0  TCP *:22 (LISTEN)
java     24601 		 foo  148u  IPv6 22442856      0t0  TCP *:8888 (LISTEN)
java     24942 		 foo  153u  IPv6 22449275      0t0  TCP 127.0.0.1:59991 (LISTEN)

Hands-On

JMXMP Setup

JVM Server / Process Side

Download JMX Reference Impl jar from Oracle/Java or either locations described here (see refs) and include it in the classpath for your server/process. This jar is available via some Maven repos too.
During the start of your Java process/server, start a JMX server which uses JMXMP protocol. I have included an example of how I did it with DropWizarb based server during bootstrap

    /**
     * Usage JVM Arg Example :
     * -Djavax.management.remote.JMXServiceURL=service:jmx:jmxmp://127.0.0.1:59991
     */
    private static class JMXMPJmxConnectorServerBundle extends RuntimeBundle {
        static final String JMXServiceURLPropKey = "javax.management.remote.JMXServiceURL";
        @Override
        public void run(Environment environment) {
            try {
                String jmxServiceUrlPropVal = System.getProperty(JMXServiceURLPropKey);
                JMXServiceURL jmxUrl = new JMXServiceURL(jmxServiceUrlPropVal);
                Map<String, String> jmxEnvironment = new ImmutableMap.Builder<String, String>()
                    .put("jmx.remote.server.address.wildcard", "false")
                    .build();
                MBeanServer mbs = ManagementFactory.getPlatformMBeanServer();
                JMXConnectorServer jmxRemoteServer = JMXConnectorServerFactory.newJMXConnectorServer(
                    jmxUrl, jmxEnvironment, mbs);
                jmxRemoteServer.start();
            } catch(Exception e) {
                throw new RuntimeException(e);
            }
        }
    }

Client Side

Download JMX Reference Impl jar from Oracle/Java or either locations described here and include it in the classpath for your server/process. This jar is available via some Maven repos too.
Invoke jvisualvm as follows

/usr/bin/jvisualvm -cp:a <full path to jmxmp jar> --openjmx service:jmx:jmxmp://127.0.0.1:59991

SSH Transparent Proxying Setup

Configure ssh client to do transparent proxying using netcat
Make sure you setup ssh-keys and publish them on the remote network for passwordless logins if you need convenience.
Sections in your .ssh/config look like this :

Host bastion-deathstar
    Hostname bastion.deathstar.org
    User hansolo
    PreferredAuthentications publickey
    IdentityFile /Users/hansolo/ssh-keys/id_rsa_deathstar
    IdentitiesOnly yes

Host node-01-deathstar
    Hostname node-01.deathstar.org
    User hansolo
    ProxyCommand ssh bastion-deathstar nc %h %p
    PreferredAuthentications publickey
    IdentityFile /Users/hansolo/ssh-keys/id_rsa_deathstar
    IdentitiesOnly yes

SSH Tunelling Setup

ssh -L 59991:127.0.0.1:59991 bastion.deathstar.org -N

Putting it all together ... Voila moment

Download the jar for JMXMP reference implementation for Oracle or other (see refs)
Setup the java server/processs with JMX server which uses JMXMP protocol
Setup SSH transparent proxying to hop via bastion/jumper hosts
Setup a SSH tunnel to the destination host ( using the transparent SSH proxying )
Invoke jvisualvm with JMXMP enabled protocol to connect to the destination host and JMX port via SSH local port forwarding ( SSH tunneling )

Navigation Menu