Tuesday, August 24, 2010

JConsole via a SOCKS ssh tunnel

I've been learning how to use cassandra lately, and one of the most useful tools when working with it is to be able to query status via all the JMX hooks it exposes. At first I was using jmxterm to query all the available settings, and while this works great, it was tough as a cassandra newb to figure out which beans I needed to query.

What I wanted to do was use JConsole so that I could browse the available beans without having to know a priori what they were. Unfortunately, our production system is all remote (ec2), so this makes it tricky to use JConsole.

At first I tried to use a basic ssh port mapping, but this didn't work due to RMI's need to create its own ports in the JConsole runtime to which the monitored process tries to connect back to - only the single port is tunnelled by ssh. There are a number of documented solutions to doing this, but they all seemed very intrusive and hard to setup correctly (e.g. running a hacked jmx agent within your monitored process).

I then tried using JConsole running remotely but displaying on my local X server, but the performance wasn't that great, and its a pain to have to install all the X runtime on my production system just for running JConsole.

Eventually I smacked my forehead after remembering that ssh can also act as a SOCKs proxy, which works great for any application that supports it. Fortunately, JConsole does (in jdk 1.6 at least).

Here is the bash alias I use to setup the socks connection and connect to my cassandra instance via jconsole. I have multiple remote remote cassandra hosts, so this lets me run JConsole independently for each.


function jc {
host=$1
proxy_port=${2:-8123}
jconsole_host=wrongway
ssh -f -D$proxy_port $host 'while true; do sleep 1; done' ssh_pid=`ps ax | grep "[s]sh -f -D$proxy_port" | awk '{print $1}'`
jconsole -J-DsocksProxyHost=localhost -J-DsocksProxyPort=$proxy_port ser
vice:jmx:rmi:///jndi/rmi://${jconsole_host}:8181/jmxrmi
kill $ssh_pid
}


Once this was in my shell environment, I simply run "jc remote_host", and I get a jconsole connected to my cassandra instance on that host.

Note that you need to set "jconsole_host" in the alias for this to work for you. I'm not sure why, but Its sensitive to the value in here, and only worked for me for the first entry I had in /etc/hosts (not localhost) that maps my local hostname to 127.0.0.1

13 comments:

  1. Matt, you're a hero! thanx a lot for sharing this.

    ReplyDelete
  2. Awesome post, sir. I'm using this as the basis for a post about the same thing I'm doing in my environment, and how your work makes mine easier. I'm linking to this as well.

    ReplyDelete
  3. For a server behind NAT rounter/firewall, you need to specify -Djava.rmi.server.hostname={server_name} as server jvm argument.
    Use it as your jconsole_host at client. Also as mentioned above, you need to modify /etc/hosts of both server and client to add it to the 127.0.0.1 entry. So it will look like if your server name is foo

    127.0.0.1 foo localhost

    foo does not need to be a valid internet name, just put it into both server and client /etc/hosts will do the trick.

    ReplyDelete
  4. You save my day.
    One suggestion I used ssh -N instead of the while loop and it works. For dummies like me
    the entry in /etc/host must be
    127.0.0.1 wrongway localhost

    ReplyDelete
  5. I have a similar situation, but i want to connect jConsole to server that i can only connect via a gateway.
    So, i can do
    ssh user@gateway
    and then on the gateway
    ssh user@prodServer
    Can anyone know how to use jconsole in this situation

    ReplyDelete
  6. +1 on the gratitude. A great solution. Thanks!

    ReplyDelete
  7. Agree with opensource21 -- using -N instead of the while loop is more desirable, since (for me anyway) the latter method launches a process on the target machine that's not killed when the tunnel is killed.

    ReplyDelete
  8. Thank You for article and all Your comments!
    Here is my final version if somebody is interested:

    function jc {
    jmx_host=$1
    jmx_port=${2:-5000}
    proxy_host=${3:-$jmx_host}
    proxy_port=${4:-8123}

    echo "connecting jconsole to $jmx_host:$jmx_port via SOCKS proxy $proxy_host using local port $proxy_port"
    ssh -f -ND $proxy_port $proxy_host
    jconsole -J-DsocksProxyHost=localhost -J-DsocksProxyPort=${proxy_port} service:jmx:rmi:///jndi/rmi://${jmx_host}:${jmx_port}/jmxrmi
    kill $(ps ax | grep "[s]sh -f -ND $proxy_port" | awk '{print $1}')
    }

    ReplyDelete
  9. For the server behind a gateway case, does your gateway SSH server support port forwarding itself? The trick I use is

    ssh -L 2222:prodServer:22 user@gateway

    then in a different terminal

    ssh -p 2222 -D 8123 user@localhost

    This effectively establishes a tunnel from your machine to the gateway, then you make another SSH connection through this tunnel from your local machine through to the target prodServer.

    ReplyDelete
  10. You can also use Your /.ssh/config and nc

    Host proxy2
    Hostname proxy2
    User usernameOnProxy2
    ProxyCommand ssh -q usernameOnProxy1@proxy1 nc -w 180 %h %p

    more here
    http://www.balamut.eu/lukasz/lockpicking-production

    ReplyDelete
  11. An easier approach which doesn't involve changing /etc/hosts on both systems is to set on the JVM opts "-Djava.rmi.server.hostname=your_ip" and then with jconsole connect to your_ip:your_port. Spent a few hours to figure this out and this post helped alot.

    ReplyDelete
  12. Do you guys need to have the jmx_port open when running in EC2 ?

    I assume that the request as proxied through localhost to the remote host on port 22.

    With jmx_port open it works for me, butI'm curious if I can get away with not opening the JMX port.

    ReplyDelete