Unable to remotely connect to JMX?

Question

For some weird reason I am not able to connect using VisualVM or jconsole to a JMX.

The parameters used to start the VM to be monitored:

-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=1100

I checked, and I can telnet to this port, from both locally and remotely.

Still, VisualVM or jconsole are failing to connect, after spending some considerably time trying to.

REMOTE MACHINE with JMX (debian)
java version "1.6.0_33"
Java(TM) SE Runtime Environment (build 1.6.0_33-b03-424-11M3720)
Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03-424, mixed mode)

MY WORKSTATION (OS X)
java version "1.6.0_26"
Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode)

What is the problem?

After supplying only -Dcom.sun.management.jmxremote.port=1100, can you run VisualVM locally, but connect via the port instead of through attach? — Ross Judson, Aug 02 '12 at 20:17
If there is a firewall in play see [this issue.][1] [1]: http://stackoverflow.com/a/19315119/2870472 — supdog, Oct 11 '13 at 10:10
@sorin , I have tried all the suggested answers below still facing `cannot connect to remote` while trying to connect a remote ec2 process from local. Could you please help ? — Jet, Jan 30 '18 at 06:55
On weblogic10.3.4 (maybe other versions as well), you MUST use the weblogic console's username/password for authentication EVEN IF you've added the java start argument: -Dcom.sun.management.jmxremote.authenticate=false (!!!) — Pierre, May 17 '18 at 14:24

score 94 · Answer 1 · edited May 29 '17 at 13:22

94

Add -Djava.rmi.server.hostname = host ip. Even i faced the same problem and this did the trick.

Addition of this -Djava.rmi.server.hostname = host ip forces RMI service to use the host ip instead of 127.0.0.1

edited May 29 '17 at 13:22

Arpit Aggarwal

21,748
13
80
99

answered Aug 16 '12 at 13:46

Sujith

1,238
7
11

2

I can confirm that adding this setting solved my problem with connecting from VisualVM to a remote VM. Nice answer. – axel22 Oct 06 '14 at 19:12
1

I did a TCP dump on the communication when JConsole tries to connect. Apparently in the handshake with the JMX port, it relays back the host given at -Djava.rmi.server.hostname and then it tries to connect to it. – Asaf Mesika Sep 08 '15 at 18:54
Where to add suggests variable change ? – Salman Jun 23 '16 at 08:49
12

This answer helped me to connect to a JVM running in Kubernetes via a port forwarding: -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=10099 -Dcom.sun.management.jmxremote.rmi.port=10099 -Djava.rmi.server.hostname=127.0.0.1 -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false I.e. setting the hostname to 127.0.0.1 and setting both jmx and rmi port to the same value! – Gerd Aschemann Aug 07 '17 at 12:28

score 37 · Answer 2 · edited Sep 17 '19 at 18:52

37

These are the steps that worked for me (Debian behind firewall on the server side was reached over VPN from my local Mac):

Check server public ip

ifconfig
Use JVM params:

    -Dcom.sun.management.jmxremote
    -Dcom.sun.management.jmxremote.port=[jmx port]
    -Dcom.sun.management.jmxremote.local.only=false
    -Dcom.sun.management.jmxremote.authenticate=false
    -Dcom.sun.management.jmxremote.ssl=false
    -Djava.rmi.server.hostname=[server ip from step 1]

Run application
Find process ID of the running java process
Check all ports used by JMX/RMI

netstat -lp | grep [pid from step 4]
Open all ports from step 5 on the firewall

Voila.

edited Sep 17 '19 at 18:52

Adrian

2,002
17
32

answered Feb 10 '15 at 20:20

Mariusz

2,419
1
18
25

Thanks! I was missing `netstat -lp | grep [pid from step 4]` – user454322 Apr 22 '15 at 10:25
2

In my case, I was missing the "java.rmi.server.hostname" ... after, adding that property, I was able to connect to my remote VM usign java VisualVM – Carlitos Way Apr 14 '16 at 19:04
In our case, turned out it more ports were required than just the one specified via `jmxremote.port`. We asked ops to open all ports to the machine behind VPN and were able to connect. Unexpected and have not reduced port exposure the machine yet to give more data. – fionbio Apr 21 '16 at 18:53
@Mariusz port opening to be done in remote machine or local ? – Jet Jan 30 '18 at 06:46
@Jet - remote machine – Mariusz Feb 03 '18 at 17:37
@Mariusz ... Still not resolved, please help me here https://stackoverflow.com/q/48516329/3301316 – Jet Feb 05 '18 at 04:44
ifconfig is better suitable for knowing the server public ip instead of hostname -i – Andrés Alcarraz Jan 17 '19 at 11:49

score 10 · Answer 3 · edited Jul 24 '12 at 11:08

10

In addition to listening to the port you specified (1100) the JMX server also listens to a randomly chosen (ephemeral) port. Check, e.g. with lsof -i|grep java if you are on linux/osx, which ports the java process listens to and make sure your firewall is open for the ephemeral port as well.

edited Jul 24 '12 at 11:08

sorin

137,198
150
472
707

answered Jul 24 '12 at 10:20

johlo

5,182
1
16
31

This is on intranet and there is no firewall, in addition I even tried to connect locally using the DHCP IP and it fails. Interesting but VisualVM is able to detect local instances and connect to them, but probably it does not use TCP ports for these. – sorin Jul 24 '12 at 11:04
1

The VM parameters looks correct. Wrong options/url to jconsole? – johlo Jul 24 '12 at 11:59
On VisualVM I right clicked on remote add JMX Connection and introduced `subdomain.example.com:1100`. – sorin Jul 24 '12 at 12:10
I also tried the second syntax `service:jmx:rmi:///jndi/rmi://subdomain.example.com:1100/jmxrmi` but the results were the same: a long, long delay of "connecting to service" that fails, probably with a timeout. – sorin Jul 24 '12 at 12:12
1

Have you tried using `-Djava.rmi.server.hostname=subdomain.example.com`, could be that the RMI service uses localhost. – johlo Jul 24 '12 at 12:37

score 9 · Answer 4 · answered Jul 25 '14 at 17:09

I experienced the problem where it said 'Adding ' forever and didn't seem to be able to connect. I got passed the problem by changing the jvisualvm proxy settings (Tools->options->network). Once I changed the option to No Proxy, I was able to connect. My jvm was started with the following options:

-Dcom.sun.management.jmxremote 
-Dcom.sun.management.jmxremote.port=2222 
-Dcom.sun.management.jmxremote.authenticate=false 
-Dcom.sun.management.jmxremote.ssl=false 
-Djava.rmi.server.hostname=<external_IP_of_server>

Then when I added the jmx connection, I specified "external_IP_of_server:2222"

score 7 · Answer 5 · edited May 23 '17 at 12:34

I had a similar issue when using port forwarding. I have a remote machine with Tomcat listening for JMX interactions on localhost:9000.

From my local machine, I'm used to do port-forwarding with:

ssh -L 9001:localhost:9000 tomcat.example.com

(so remote port 9000 is forwarded to my local machine's port 9001).

Then when I tried to use VisualVM to connect to localhost:9001, the connection was refused. JMX seems to require port numbers on both sides to be identical.

So my solution was using port numbers 9000 and 9000:

    ssh -L 9000:localhost:9000 tomcat.example.com

Now my local machine's VisualVM connects successfully to the remote machine's Tomcat via localhost:9000.

Make sure that you don't have any other service (Tomcat on dev machine?) listening on the same port.

Also take a look at setting up parameters correctly.

score 5 · Answer 6 · answered Feb 25 '19 at 20:57

Since I just joined I can't upvote Hett's answer, but it saved my life from another week of trial and error!

This is an example of a working Dockerfile:

FROM store/oracle/serverjre:8 

RUN mkdir -p /opt/app

ENV APP_PATH /opt/app

WORKDIR $APP_PATH   

COPY . $APP_PATH

CMD ["java", \
     "-Dcom.sun.management.jmxremote", \
     "-Dcom.sun.management.jmxremote.port=9010", \
     "-Dcom.sun.management.jmxremote.rmi.port=9010", \
     "-Dcom.sun.management.jmxremote.authenticate=false", \
     "-Dcom.sun.management.jmxremote.ssl=false", \
     "-Djava.rmi.server.hostname=12.345.67.89", \
     "-jar", \
     "app-service-0.0.1-SNAPSHOT.jar"]

EXPOSE 9010

score 5 · Answer 7 · answered Apr 05 '19 at 11:27

5

My two cents to the above answers..

I see most of the answers has mentioned only about the hostnames but no port. If we haven't specified ports, then server will dynamically assign RMI port. There wont be any issues, if both the servers are in same subnet or no firewall issues. If there is any concerns, we can add below JVM parameter to freeze.

-Dcom.sun.management.jmxremote.rmi.port

Ex:

<option name="-Dcom.sun.management.jmxremote.rmi.port" value="11001"/>

Make sure both RMI and JMX ports should be the same. For more, click here

answered Apr 05 '19 at 11:27

vardhan

1,087
10
11

1

Thanks for that answer. I already spend hours to find that hint. Now everything works for me: – Tomate_Salat Nov 26 '19 at 11:00
This was also the solution for us! thanks! – haikalpribadi Feb 12 '21 at 12:50

Hett · Answer 8 · 2019-10-24T12:50:16.387

3

If you run jar file (via -jar option), you must specifie all other jvm options before -jar option!

edited Oct 24 '19 at 12:50

answered Sep 20 '18 at 07:46

Hett

2,529
2
26
45

score 2 · Answer 9 · answered Apr 28 '14 at 10:29

I found the problem, my rmi service was running on the host ip which was "127.0.0.1". To connect remotely to the jvm I had to bind the external ip to the hostname. To do this in unix systems use command hostname to get the name of the hostname. Then check the ip that is assigned to the hostname, to find out this use ping $(hostname) you will see that system is pinging the hosname's ip. If your host ip was the default "127.0.0.1" and you wanted to change it, just edit the file /etc/hosts as superuser. After rebooting the rmi service, you can reach it from the remote machine.

`hostname` will give you the wrong answer in some environments. To get the correct external IP on AWS EC2 use: `ip=$(curl -s http://169.254.169.254/latest/meta-data/public-ipv4)` — Alex R, Oct 24 '16 at 04:44

score 0 · Answer 10 · answered Feb 27 '19 at 16:12

0

look in /etc/hosts if you don't have a wrong IP for your machine example : 127.0.0.1 localhost 127.0.0.2 your_machine 185.12.58.2 your_machine (the good IP for your machine)

JMX take the IP 127.0.0.2 and forget the other

answered Feb 27 '19 at 16:12

Dominique MITRE

1

Unable to remotely connect to JMX?

10 Answers10

Linked