102

I use cProfile now but I find it tedious to write pstats code just to query the statistics data.

I'm looking for a visual tool that shows me what my Python code is doing in terms of CPU time and memory allocation.

Some examples from the Java world are visualvm and JProfiler.

  • Does something like this exist?
  • Is there an IDE that does this?
  • Would dtrace help?

I know about KCachegrind for Linux, but I would prefer something that I can run on Windows/Mac without installing KDE.

Community
  • 1
  • 1
Frederik
  • 12,867
  • 9
  • 42
  • 52
  • 6
    If a program like this does not yet exist, it would be a great open source project. – carl Jul 31 '10 at 17:14
  • @cvondrick A program like this does exist, and was already mentioned: KCachegrind. – Devin Jeanpierre Jul 31 '10 at 18:12
  • 1
    @Devin, yes but see the question. :-) – carl Jul 31 '10 at 20:59
  • Are you aware of this technique: http://stackoverflow.com/questions/375913/what-can-i-use-to-profile-c-code-in-linux/378024#378024 It's not visual, but neither is it tedious, and it's hard to beat for effectiveness. – Mike Dunlavey Aug 01 '10 at 18:25
  • @carl creating a new project to reimplement KCacheGrind is called "reinventing the wheel". "not KCacheGrind" is not a good reason motivation for creating new software. Maybe if KCacheGrind is unable to do a thing, and the inability is fundamental to what KCacheGrind is or how it works... – Devin Jeanpierre Aug 02 '10 at 20:30
  • 4
    @KCacheGrind lovers: Any reason to not have to install KDE is a good reason not to use KCacheGrind. – Matt Joiner Aug 03 '10 at 00:31
  • @MikeDunlavey That's fine when a single bottleneck is taking up 50% of your CPU. Not much use when you've got 20 using up 5%. It also doesn't cover memory usage, thread blocking, etc, etc – Basic Nov 05 '13 at 11:22
  • @Basic: Many people assume (just guessing, really) that their programs are actually that close to optimal. That's far from my experience. [*This post*](http://scicomp.stackexchange.com/a/2719/1262) discusses the math behind it, and if you're up for a very amateur 8-minute video demonstration [*there's one here*](https://www.youtube.com/watch?v=xPg3sRpdW1U). – Mike Dunlavey Nov 05 '13 at 13:08

12 Answers12

89

A friend and I have written a Python profile viewer called SnakeViz that runs in a web browser. If you are already successfully using RunSnakeRun SnakeViz may not add that much value, but SnakeViz is much easier to install.

Edit: SnakeViz supports Python 2 and 3 and works on all major systems.

jiffyclub
  • 1,617
  • 1
  • 15
  • 9
41

I'm only aware of RunSnakeRun.

There was also some talk some time ago about an integrated profiler in PyDev (Eclipse), but I don't know if that will ever see the light of day.

Update: Unfortunately it seems that RunSnakeRun is no longer maintained, and it does not support Python 3.

nikow
  • 19,957
  • 6
  • 45
  • 69
  • +1 for RunSnakeRun. The best tool IMHO. – codeape Dec 19 '12 at 10:43
  • 4
    RunSnakeRun is good, but unfortunately it doesn't currently work in Python 3. (True for June 2014.) – Ram Rachum Jun 19 '14 at 20:37
  • @Ram: Thanks for the info, that is unfortunate :-(. – nikow Jun 20 '14 at 08:11
  • 2
    I've used pyinstrument instead. It's a different animal, but it's useful. – Ram Rachum Jun 21 '14 at 09:15
  • I like RunSnakeRun, but I strongly dislike how it displays callers and callees. RunSnakeRun will show the "total" values for call counts, local time and cumulative time for each caller/callee. So, you can have things like a function that has cumulative time of 5 seconds but a callee which has cumulative time of 100 seconds. Not that intuitive, and not as useful as what pstats provides. pstats uses contextual information related to the function in question to give more meaningful numbers for the caller/callee statistics. Not aware of another viewer which makes this easily viewable. – Vultaire Jun 24 '16 at 04:58
  • Also, pstats can be run via "python -m pstats", removing the need for "writ[ing] pstats code just to query the statistics data". No GUI though. – Vultaire Jun 24 '16 at 05:00
14

I use gprof2dot.py. The result looks like this. I use those commands:

  python -m cProfile -o profile.dat my_program.py
  gprof2dot.py -f pstats profile.dat | dot -Tpng -o profile.png

You need graphviz and gprof2dot.py installed. You might like a convenience shell script.

maxy
  • 3,933
  • 20
  • 23
  • If you output svg instead of png (with dot -Tsvg -o profile.svg) you'll be able to search the output graph with your browser, and you'll be able to scale the image without jaggies. – razeh May 14 '15 at 14:54
7

Spyder also provides a pretty nice gui for cProfile:

enter image description here

Frederik
  • 12,867
  • 9
  • 42
  • 52
jsexauer
  • 621
  • 2
  • 11
  • 20
5

Python Tools for Visual Studio contains a very well done graphical profiler: http://www.youtube.com/watch?v=VCx7rlPyEzE&hd=1

http://pytools.codeplex.com/

Jake
  • 2,267
  • 3
  • 25
  • 39
  • 2
    Yeah, I use this profiler for my .Net stuff. It just highlights how woeful cprofile is... – Basic Nov 05 '13 at 11:24
4

KCacheGrind includes a version called QCacheGrind which does run on Mac OS X and on Windows.

akaihola
  • 24,161
  • 5
  • 52
  • 64
  • Moreover OP seems to misinterpret dependencies for KCachegrind on Linux. On Debian/Ubuntu/Mint all you need is `apt-get install kcachegrind` which installs just 3 KDE-related libraries. – saaj Sep 29 '16 at 10:31
  • 2
    @saaj on Ubuntu 17.04 installation, `apt install kcachegrind` wants to install 102 packages, including ~40 KDE libraries. – Mark E. Haase Aug 20 '17 at 20:37
  • @mehaase On fresh `ubuntu:xenial` it may be three times as many, but it's for an development environment and I barely see it as an issue. And actually `apt-cache depends --recurse --no-recommends --no-suggests --no-conflicts --no-breaks --no-replaces --no-enhances --no-pre-depends kcachegrind | grep kde | grep Depends | sort -u | wc -l` says just 13. – saaj Aug 21 '17 at 08:48
4

This person created a graphical profile, described here. Maybe you could use that as a starting point for your own work.

PaulMcG
  • 56,194
  • 14
  • 81
  • 122
3

Try out Snakeviz. Very easy to install (via pip) and it's browser based.

https://jiffyclub.github.io/snakeviz/

BangTheBank
  • 651
  • 2
  • 9
  • 24
2

Python Call Graph generates pics very similar to those in maxy's answer. It also shows total time for each function, for some reason it's not reflected in the example graphs.

Community
  • 1
  • 1
Lev Levitsky
  • 55,704
  • 18
  • 130
  • 156
1

I've written a browser-based visualization tool, profile_eye, which operates on the output of gprof2dot.

gprof2dot is great at grokking many profiling-tool outputs, and does a great job at graph-element placement. The final rendering is a static graphic, which is often very cluttered.

Using d3.js it's possible to remove much of that clutter, through relative fading of unfocused elements, tooltips, and a fisheye distortion.

For comparison, see profile_eye's visualization of the canonical example used by gprof2dot. For Python in particular, see a cProfile output example.

Ami Tavory
  • 66,807
  • 9
  • 114
  • 153
1

Consider pyflame + flamegraph

Pyflame: A Ptracing Profiler For Python + flamegraph

https://github.com/uber/pyflame

You can trace towards a running python process using pyflame.

McKelvin
  • 3,360
  • 1
  • 24
  • 22
0

I have used plop and found it to be very light-weight. Gives a quick insight into the perf.

auny
  • 1,782
  • 4
  • 15
  • 35
  • [vprof](https://github.com/nvdv/vprof) is a similar tool that also uses the flamegraph, but it can also do memory profiling. Unfortunately it only shows cumulative time and not the total time spent in a function. – goodmami Sep 13 '18 at 00:03