Profiling a running Java application in command line

Question

I profile running Java applications often with VisualVM but it needs X to run on the machine.

I know I can connect through management port but that will be an offline sampled profiling which is not enough for me.

So I'm looking for a solution with which I can profile the CPU usage of the methods of a running Java application from command-line. It's enough for me to collect data on the server and then the collected data can be analyzed on a different machine.

Update:

It seems I need to be more specific. I want to profile a running Java application from command line, I don't want to stop it and rerun it.

Can you collect 10 or 20 stack samples with *jstack*? Then if *Foo* is a method, its overall time usage is the fraction of samples containing it. Its CPU usage is the fraction of those samples that don't terminate in I/O or a system call. Its "self time" is the fraction of samples in which it itself is the terminus. — Mike Dunlavey, Jul 27 '11 at 18:30
that would be the same as the VisualVM offline profiling, won't be? — KARASZI István, Jul 28 '11 at 13:56
[Check the doc.](http://visualvm.java.net/profiler.html) It doesn't tell you, by line (not function) the percent of inclusive time (not exclusive) that line is responsible for, and overall time (not just CPU). It suffers from [these problems](http://stackoverflow.com/questions/1777556/alternatives-to-gprof/1779343#1779343). Stack sampling is not pretty, but it finds the performance problems very quickly. Other tools are pretty, and they give you lots of numbers to puzzle over, but they don't take you straight to the problem, whatever it is. — Mike Dunlavey, Jul 28 '11 at 17:12
I tried to get the stack traces but it refuses to run without the `-F` flag, with which it freezes my app. — KARASZI István, Jul 28 '11 at 17:48
Hey, I don't need anything pretty. I either run it under the IDE and collect them that way, or use something like jstack that snapshots the stack of a running app. — Mike Dunlavey, Jul 29 '11 at 01:35
@Mike could you post your comment as an answer? I would accept it! — KARASZI István, Jul 29 '11 at 13:07

score 33 · Accepted Answer · edited May 29 '17 at 22:46

33

The jvmtop application is a convenient tool for profiling from the commandline. No need to stop the jvm. Usage:

jvmtop.sh --profile <PID>

Will give you output like this which will be updating while the app runs:

  Profiling PID 24015: org.apache.catalina.startup.Bootstrap
  36.16% (    57.57s) hudson.model.AbstractBuild.calcChangeSet()
  30.36% (    48.33s) hudson.scm.SubversionChangeLogParser.parse()
   7.14% (    11.37s) org.kohsuke.stapler.jelly.JellyClassTearOff.parseScript()
  ...

The advantage is that it does not take the use of instrumentation. The classes of the to-be-profiled jvm will not be altered.

If you are looking for something more visual then have a look at jvm-mon which is based on jvmtop

edited May 29 '17 at 22:46

rogerdpack

50,731
31
212
332

answered Jan 19 '16 at 21:29

Andrejs

24,146
10
98
92

4

I've been messing witth and cursing visualvm for years. Thanks for pointing me to this handy tool. Works without JVM agents, server restarts, etc. Simply untar on the box you want to profile, point JAVA_HOME to the right place and run. – Jilles van Gurp Feb 01 '16 at 14:15
4

This is an excellent tool but the project looks like it is dead. no release since 2015 – user1870400 Nov 07 '18 at 11:18
jvm-mon is a ...sub-optimal tool, profiling should mean also stack trace. – peterh Sep 02 '20 at 14:07
Looks like jvmtop doesn't work any more since JDK 9. – Rok Povsic May 09 '21 at 19:17

rogerdpack · Answer 2 · 2020-01-22T08:11:46.123

Looks like the "built-in" way to profile a java app from the command line is to start it with profiling command line parameters, like this

$ java -Xrunhprof:cpu=samples,file=myprogram.hprof ...

Then examine the file "myprogram.hprof" with some GUI tool (or web server tool like jhat) or command line tool after the process exits (and the file is created at that time).

If you use the "QUIT" signal trick, mentioned https://stackoverflow.com/a/2344436/32453 then you can generate a file at will without exiting the JVM (it appears to append to the previous output file). Or wait until the process exits and it will generate the file.

This (built-in) profiler does a sample infrequently so typically low slowdown/impact overall.

ref: http://web.archive.org/web/20160623224137/https://thunderguy.com/semicolon/2004/04/18/profiling-a-java-program-easily/

You could also just do the "poor man's profiler" by collecting lots of jstacks and dumping them into ex: a flamegraph or some other analyzer/conglomerator...

Thanks for the pointer! Note for future readers: hprof is sampling threads in the Java RUNNABLE state (so it's not equivalent of aggregating multiple `jstack` dumps, like I expected, but neither does it measure CPU consumption). It has other gotchas - see http://www.brendangregg.com/blog/2014-06-09/java-cpu-sampling-using-hprof.html — Nickolay, Jan 12 '18 at 17:12

score 14 · Answer 3 · edited May 23 '17 at 12:34

14

Can you collect 10 or 20 stack samples with jstack? Then if Foo is a method, its overall time usage is the fraction of samples containing it. Its CPU usage is the fraction of those samples that don't terminate in I/O or a system call. Its "self time" is the fraction of samples in which it itself is the terminus.

I don't need anything pretty. I either run it under the IDE and collect them that way, or use something like jstack that snapshots the stack of a running app.

That's the random-pause technique.

edited May 23 '17 at 12:34

Community

1
1

answered Jul 29 '11 at 14:26

Mike Dunlavey

38,662
12
86
126

1

I selected this idea: http://svn.apache.org/repos/asf/jackrabbit/sandbox/jackrabbit-j3/src/test/java/org/apache/jackrabbit/j3/Profiler.java – KARASZI István Jul 29 '11 at 15:05
This solution helped me to find a bottleneck with DNS resolution on the server. Thank you again! – KARASZI István Aug 19 '11 at 21:55
@KARASZI István: Glad it helped. – Mike Dunlavey Aug 19 '11 at 22:59

Sanjay T. Sharma · Answer 4 · 2015-02-11T16:15:12.770

6

We have used hprof on our servers and it definitely is better than sysouts in case you can't run a full fledged VisualVM session.

Examples of using hprof are plenty out there:

edited Feb 11 '15 at 16:15

answered Jul 27 '11 at 14:36

Sanjay T. Sharma

21,620
4
54
70

but I don't really get how can I attach it to a pid – KARASZI István Jul 27 '11 at 14:45
1

Sorry, AFAIK, `hprof` doesn't allow for attaching to a process. Your best bet would be to look into tools like jprofile and jtop. http://java.sun.com/developer/technicalArticles/J2SE/monitoring/ – Sanjay T. Sharma Jul 27 '11 at 15:27
@SanjayT.Sharma i am using HPROF for CPU profiling by using cpu=times,and generating the data after each specific run of the application,but i am not able to analyze the data like which process for a specific run had max cpu usage.Using jhat didn't helped much,plz enlighten. – Manas Pratim Chamuah Jan 16 '15 at 05:42
Sorry to downvote, but all of the java.sun.com links are dead. The javalobby.org link still works :) – blong Feb 11 '15 at 13:45
1

@blong: Good spot, fixed the broken links. – Sanjay T. Sharma Feb 11 '15 at 16:15
1

@SanjayT.Sharma , great, thanks so much! Downvote converted to upvote :) – blong Feb 11 '15 at 18:00

Vadzim · Answer 5 · 2020-01-06T13:25:06.927

The most precise profiling can be achieved with https://github.com/jvm-profiling-tools/async-profiler.

This project is a low overhead sampling profiler for Java that does not suffer from Safepoint bias problem. It features HotSpot-specific APIs to collect stack traces and to track memory allocations. The profiler works with OpenJDK, Oracle JDK and other Java runtimes based on HotSpot JVM.

Here is my script to install and run it from command-line:

async-profiler.sh

if [ ! -d profiler ]; then
  mkdir profiler && cd profiler && curl -L https://github.com/jvm-profiling-tools/async-profiler/releases/download/v1.6-ea/async-profiler-1.6-ea-linux-x64.tar.gz | tar xvz
  echo 1 > /proc/sys/kernel/perf_event_paranoid
  echo 0 > /proc/sys/kernel/kptr_restrict
  #apt install openjdk-8-dbg
else
  cd profiler
fi

#jps

./profiler.sh -d 60 -f dump_`date +%Y-%m-%d_%H-%M-%S`.jfr `jps -q`

It assumes that app is run under same user and there is a single java process PID to be listed by jps. Profiling duration is 60 seconds.

No modification of app's startup options or app restart is needed.

GUI for examining dumps is built-in into IntelliJ IDEA Ultimate: https://www.jetbrains.com/help/idea/cpu-profiler.html.

Nice illustrated article on async-profiler: https://hackernoon.com/profiling-java-applications-with-async-profiler-049s2790 — Vadzim, Sep 04 '19 at 06:01

rogerdpack · Answer 6 · 2018-08-29T17:57:09.760

3

One way to profile an "already started" JVM is to aggregate multiple jstacks taken over time.

You can for instance parse and display them as a FlameGraph (see details at the various answers for that link, I won't redundantly include them here).

edited Aug 29 '18 at 17:57

answered Aug 29 '18 at 17:18

rogerdpack

50,731
31
212
332

score 2 · Answer 7 · answered Jul 27 '11 at 14:42

You can run most commercial profilers remotely so an agent is run on the server then connect to that agent through a client on your dev machine. My absolute favorite profiler is JProfiler. It's fairly reasonable purchase, and it's very stable (which not all commercial profilers that's true).

http://www.ej-technologies.com/products/jprofiler/overview.html

Other commercial profilers that are stable, but not my favorite are YourKIT.

http://www.yourkit.com/

Those smaller vendors make good tools. These tools will provide you tons of information about method timings, memory use, GC, etc. Much more than jconsole.

Profiling a running Java application in command line

7 Answers7

Linked