What I wanted to do was use JConsole so that I could browse the available beans without having to know a priori what they were. Unfortunately, our production system is all remote (ec2), so this makes it tricky to use JConsole.
At first I tried to use a basic ssh port mapping, but this didn't work due to RMI's need to create its own ports in the JConsole runtime to which the monitored process tries to connect back to - only the single port is tunnelled by ssh. There are a number of documented solutions to doing this, but they all seemed very intrusive and hard to setup correctly (e.g. running a hacked jmx agent within your monitored process).
I then tried using JConsole running remotely but displaying on my local X server, but the performance wasn't that great, and its a pain to have to install all the X runtime on my production system just for running JConsole.
Eventually I smacked my forehead after remembering that ssh can also act as a SOCKs proxy, which works great for any application that supports it. Fortunately, JConsole does (in jdk 1.6 at least).
Here is the bash alias I use to setup the socks connection and connect to my cassandra instance via jconsole. I have multiple remote remote cassandra hosts, so this lets me run JConsole independently for each.
function jc {
host=$1
proxy_port=${2:-8123}
jconsole_host=wrongway
ssh -f -D$proxy_port $host 'while true; do sleep 1; done' ssh_pid=`ps ax | grep "[s]sh -f -D$proxy_port" | awk '{print $1}'`
jconsole -J-DsocksProxyHost=localhost -J-DsocksProxyPort=$proxy_port ser
vice:jmx:rmi:///jndi/rmi://${jconsole_host}:8181/jmxrmi
kill $ssh_pid
}
Once this was in my shell environment, I simply run "jc remote_host", and I get a jconsole connected to my cassandra instance on that host.
Note that you need to set "jconsole_host" in the alias for this to work for you. I'm not sure why, but Its sensitive to the value in here, and only worked for me for the first entry I had in /etc/hosts (not localhost) that maps my local hostname to 127.0.0.1
Great blog post
ReplyDeleteMatt, you're a hero! thanx a lot for sharing this.
ReplyDeleteAwesome post, sir. I'm using this as the basis for a post about the same thing I'm doing in my environment, and how your work makes mine easier. I'm linking to this as well.
ReplyDeleteFor a server behind NAT rounter/firewall, you need to specify -Djava.rmi.server.hostname={server_name} as server jvm argument.
ReplyDeleteUse it as your jconsole_host at client. Also as mentioned above, you need to modify /etc/hosts of both server and client to add it to the 127.0.0.1 entry. So it will look like if your server name is foo
127.0.0.1 foo localhost
foo does not need to be a valid internet name, just put it into both server and client /etc/hosts will do the trick.
You save my day.
ReplyDeleteOne suggestion I used ssh -N instead of the while loop and it works. For dummies like me
the entry in /etc/host must be
127.0.0.1 wrongway localhost
I have a similar situation, but i want to connect jConsole to server that i can only connect via a gateway.
ReplyDeleteSo, i can do
ssh user@gateway
and then on the gateway
ssh user@prodServer
Can anyone know how to use jconsole in this situation
+1 on the gratitude. A great solution. Thanks!
ReplyDeleteAgree with opensource21 -- using -N instead of the while loop is more desirable, since (for me anyway) the latter method launches a process on the target machine that's not killed when the tunnel is killed.
ReplyDeleteThank You for article and all Your comments!
ReplyDeleteHere is my final version if somebody is interested:
function jc {
jmx_host=$1
jmx_port=${2:-5000}
proxy_host=${3:-$jmx_host}
proxy_port=${4:-8123}
echo "connecting jconsole to $jmx_host:$jmx_port via SOCKS proxy $proxy_host using local port $proxy_port"
ssh -f -ND $proxy_port $proxy_host
jconsole -J-DsocksProxyHost=localhost -J-DsocksProxyPort=${proxy_port} service:jmx:rmi:///jndi/rmi://${jmx_host}:${jmx_port}/jmxrmi
kill $(ps ax | grep "[s]sh -f -ND $proxy_port" | awk '{print $1}')
}
For the server behind a gateway case, does your gateway SSH server support port forwarding itself? The trick I use is
ReplyDeletessh -L 2222:prodServer:22 user@gateway
then in a different terminal
ssh -p 2222 -D 8123 user@localhost
This effectively establishes a tunnel from your machine to the gateway, then you make another SSH connection through this tunnel from your local machine through to the target prodServer.
You can also use Your /.ssh/config and nc
ReplyDeleteHost proxy2
Hostname proxy2
User usernameOnProxy2
ProxyCommand ssh -q usernameOnProxy1@proxy1 nc -w 180 %h %p
more here
http://www.balamut.eu/lukasz/lockpicking-production
An easier approach which doesn't involve changing /etc/hosts on both systems is to set on the JVM opts "-Djava.rmi.server.hostname=your_ip" and then with jconsole connect to your_ip:your_port. Spent a few hours to figure this out and this post helped alot.
ReplyDeleteDo you guys need to have the jmx_port open when running in EC2 ?
ReplyDeleteI assume that the request as proxied through localhost to the remote host on port 22.
With jmx_port open it works for me, butI'm curious if I can get away with not opening the JMX port.