Monday, March 19, 2012

ServiceMix: Remote JMX Connection

I recently came across an issue where I was was unable to make a remote JMX connection to an instance of Apache ServiceMix which was running on a remote Linux server.  In this post I will document the configuration needed to make a successful remote JMX connection to ServiceMix deployed on Cent OS Server 6.2 using JConsole or VisualVM.

List of binaries used in this example:
When Apache ServiceMix is deployed to a Windows system you should be able to make a remote JMX connection using JConsole or VisualVM without an issue straight out-of-the-box.  However, with Linux there may be additional configuration required before this remote JMX connection can be made.  In order to make a remote JMX connection, the following property should be set on the JVM:

 -Dcom.sun.management.jmxremote  

A quick check of the [install_dir]/bin/servicemix script will confirm this property is being set on the Linux JVM, so why can't the connection be made?

If you are experiencing an issue making a remote JMX connection to an instance of Apache ServiceMix running on a Linux server check the following two items:

1) The result of hostname -i

 hostname -i  

If this command returns with the address 127.0.0.1 then the /etc/hosts file will need to be edited so that the hostname resolves to the host IP address.  The IP address of the host can found by running the the following command:

 [jsherman@saturn apache-servicemix-4.4.1-fuse-03-06]$ ifconfig  
 eth0   Link encap:Ethernet HWaddr 00:0C:29:85:EE:F9   
      inet addr:192.168.26.176 Bcast:192.168.26.255 Mask:255.255.255.0  
      inet6 addr: fe80::20c:29ff:fe85:eef9/64 Scope:Link  
      UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1  
      RX packets:11428 errors:0 dropped:0 overruns:0 frame:0  
      TX packets:8272 errors:0 dropped:0 overruns:0 carrier:0  
      collisions:0 txqueuelen:1000   
      RX bytes:1544128 (1.4 MiB) TX bytes:5504745 (5.2 MiB)  
      Interrupt:19 Base address:0x2000   
 lo    Link encap:Local Loopback   
      inet addr:127.0.0.1 Mask:255.0.0.0  
      inet6 addr: ::1/128 Scope:Host  
      UP LOOPBACK RUNNING MTU:16436 Metric:1  
      RX packets:680 errors:0 dropped:0 overruns:0 frame:0  
      TX packets:680 errors:0 dropped:0 overruns:0 carrier:0  
      collisions:0 txqueuelen:0   
      RX bytes:54936 (53.6 KiB) TX bytes:54936 (53.6 KiB)  

From this output you can see the IP address is 192.168.26.176

I have noticed that if the result of "hostname -i" returns something similar to the following, the remote JMX connection should still work as long as a domain has been configured on the localhost entry in /etc/hosts:

 [jsherman@saturn apache-servicemix-4.4.1-fuse-03-06]$ hostname -i  
 127.0.0.1 192.168.26.176  

If a domain has not been configured on the localhost entry then the remote JMX connection will fail.  If for some reason you cannot or choose not to have a domain on the localhost entry, you can edit the JMX serviceURL property in the org.apache.karaf.management.cfg file, which is located in the [install_dir]/etc, as follows by changing "localhost" to the host IP address, "192.168.26.176" in my case:

 serviceUrl = service:jmx:rmi://192.168.26.176:${rmiServerPort}/jndi/rmi://192.168.26.176:${rmiRegistryPort}/karaf-${karaf.name}  

Note: Normally the serviceURL should not have to be edited. However, adding the machine's IP address will allow you to connect for the above scenario.

2) The result of iptables -L -v

The second issue that may be preventing you from making a remote JMX connection is the machine's firewall. Running the above command will result in something similar to the following:

 [root@saturn etc]# iptables -L -v  
 Chain INPUT (policy DROP 0 packets, 0 bytes)  
  pkts bytes target   prot opt in   out   source        destination       
  679 56388 ACCEPT   all -- lo   any   anywhere       anywhere        
 18653 2145K ACCEPT   all -- any  any   anywhere       anywhere      state RELATED,ESTABLISHED   
   2  128 ACCEPT   tcp -- any  any   anywhere       anywhere      tcp dpt:ssh   
 Chain FORWARD (policy DROP 0 packets, 0 bytes)  
  pkts bytes target   prot opt in   out   source        destination       
 Chain OUTPUT (policy ACCEPT 12 packets, 5734 bytes)  
  pkts bytes target   prot opt in   out   source        destination       
 [root@saturn etc]#   

This shows the INPUT chain will drop all packets that do not match one of the configured rules on the INPUT chain.

To allow the remote JMX connection to be accepted I needed to add the following rules:

 iptables -A INPUT -p tcp --dport 1099 -j ACCEPT  
 iptables -A INPUT -p tcp --dport 44444 -j ACCEPT  

Which results in the following rules for the INPUT chain:

 [root@saturn etc]# iptables -L -v  
 Chain INPUT (policy DROP 0 packets, 0 bytes)  
  pkts bytes target   prot opt in   out   source        destination       
  694 57879 ACCEPT   all -- lo   any   anywhere       anywhere        
 19486 2214K ACCEPT   all -- any  any   anywhere       anywhere      state RELATED,ESTABLISHED   
   2  128 ACCEPT   tcp -- any  any   anywhere       anywhere      tcp dpt:ssh   
   0   0 ACCEPT   tcp -- any  any   anywhere       anywhere      tcp dpt:rmiregistry   
   2  112 ACCEPT   tcp -- any  any   anywhere       anywhere      tcp dpt:44444   
 Chain FORWARD (policy DROP 0 packets, 0 bytes)  
  pkts bytes target   prot opt in   out   source        destination       
 Chain OUTPUT (policy ACCEPT 19 packets, 1850 bytes)  
  pkts bytes target   prot opt in   out   source        destination       
 [root@saturn etc]#   

These rules could have been configured more securely by adding a specific source for the connection, however for this example I will accept connections from any source on ports 1099 and 44444.

The last thing you will want to do is save these changes to the iptables so that they persist a system reboot by running the following command:
 /sbin/service iptables save  

Seeing it in action

Now that the configuration has been taken care of you should now be able to make a remote JMX connection to Apache ServiceMix using the following URL:

 service:jmx:rmi:///jndi/rmi://localhost:1099/karaf-root  

Where "localhost" would be replaced by the IP address or hostname of the remote machine.

Remember that the default configuration for ServiceMix 4.4.1 requires the username/password of smx/smx on the JMX connection.

Conclusion

At first it might seem baffling as to why you are unable to make a remote JMX connection to a Linux server but the resolution is really quite simple. This issue is listed on the Oracle page regarding the FAQ of JConsole and Remote Management. Have a look at the FAQ if you have any further questions, or to see how the connection can be filtered to a specific source address.