37

I was thinking that JIT compilers will eventually beat AOT compilers in terms of the performance of the compiled code, due to the inherent advantage of JIT (can use information available only at runtime). One argument is that AOT compilers can spend more time compiling code, but a server VM could spend a lot of time, too.

I do understand that JIT does seem to beat AOT compilers in some cases, but they still seem to lag behind in most cases.

So my question is, what are the specific, tough problems that is preventing JIT compilers to beat AOT compilers?

EDIT:
Some common arguments:

  • AOT compilers can spend more time doing advanced optimizations -> If you are running a server VM for days, you can spend the same amount of time, if not longer.
  • Byte code interpretation has cost -> Most JIT compilers cache native machine instructions anyways these days.

Yet another edit:
For a specific example, see this article: Improving Swing Performance: JIT vs AOT Compilation. From what I can gather from this article, the authors are basically saying that when there are no hotspots, the advantage of having runtime information decreases and thus AOT without the overhead of JIT, wins. But by 40%?? That doesn't seem to make a lot of sense. Is it simply that the JIT compiler that was compared wasn't tuned for this situation? Or is it something more fundamental?

Seki
  • 10,429
  • 5
  • 39
  • 63
Enno Shioji
  • 25,422
  • 13
  • 67
  • 104
  • 3
    You seem to completely miss the idea of "just". Compiler perf is measured by humans, not machines. Nobody puts up with a 10 second wait. – Hans Passant Sep 29 '11 at 00:51
  • 1
    @Hans Passant: In a client VM, I see the point. But in a server VM? The JVM for example has that distinction. – Enno Shioji Sep 29 '11 at 01:20
  • I guess this just boils down to "Why is it hard to write an compiler that emits well-optimized machine code?". Also, comparing Java/JIT to C++/AOT is comparing Apples to Oranges. (Unless there is a *specific* Java AOT/JIT being talked about?) While it is true that an AOT "has more time" to spend on the entire process, don't forget javac has already done good chunks of compiling -- lexing, parsing, some optimizing, albeit to a "virtual machine" -- itself. –  Sep 29 '11 at 01:38
  • @Enno Shioji: From the article you linked: *"Where does the 40% performance win come from? We found the answer unexpectedly when we aimed at further improving Swing performance: the absence of hot methods."* – Rob Sep 29 '11 at 05:49
  • @Rob: Yeah, but does it make sense that that will result in 40% performance difference? At minimum, the JIT could start behaving like AOT once it detects that hotspot optimization is not worth it. That seems like a reasonably easy thing to fix. – Enno Shioji Sep 29 '11 at 06:51
  • @EnnoShioji: Just saw you accepted; I had forgotten about this question. A few things; a JIT compiler can't just "switch" to AOT at runtime, because AOT by definition occurs before execution time. Unfortunately, a large number of implementation details ultimately go into making up that 40% difference, so it's difficult to pinpoint. – Rob Oct 13 '11 at 13:03
  • See the second two paragraphs on this page: http://publib.boulder.ibm.com/infocenter/javasdk/v5r0/index.jsp?topic=%2Fcom.ibm.java.doc.diagnostics.50%2Fdiag%2Ftools%2Fjitpd_diagnose.html -- it seems that in the JVM, 'hot' methods are compiled to native instructions when an internal call counter is exceeded. Otherwise byte-code interpretation is performed as normal. (My point is that AOT compilation is not relevant at runtime.) – Rob Oct 13 '11 at 13:16

2 Answers2

32

There's a definite trade-off between JIT and AOT (ahead-of-time) compilation.

As you stated, JIT has access to run-time information that can aid in optimization. This includes data about the machine it's executing on, enabling platform-specific native optimization. However, JIT also has the overhead of translating byte-code to native instructions.

This overhead often becomes apparent in applications where a fast start-up or near real-time responses are necessary. JIT is also not as effective if the machine does not have sufficient resources for advanced optimization, or if the nature of the code is such that it cannot be "aggressively optimized."

For example, taken from the article you linked:

... what should we improve in the absence of clear performance bottlenecks? As you may have guessed, the same problem exists for profile-guided JIT compilers. Instead of a few hot spots to be aggressively optimized, there are plenty of "warm spots" that are left intact.

AOT compilers can also spend as much time optimizing as they like, whereas JIT compilation is bound by time requirements (to maintain responsiveness) and the resources of the client machine. For this reason AOT compilers can perform complex optimization that would be too costly during JIT.

Also see this SO question: JIT compiler vs offline compilers

Community
  • 1
  • 1
Rob
  • 5,013
  • 5
  • 37
  • 60
  • 8
    The more time one is willing to spend building a piece of code, the faster it can run. If users of AOT compilers are willing to give the build process more time, it will be able to produce more efficient code. An advantage of JIT compilation, however, is that is *infinitely* faster than AOT compilation for code which is never run. In some cases involving .NET generics, a finite (small even) amount of source code can specify an unbounded family of methods which would each require separate machine code; if only a dozen such methods are run, only those twelve will be compiled to machine code. – supercat Feb 20 '15 at 18:55
  • What about profile-guided AOT compilation? – Jessie Lesbian May 10 '21 at 06:41
1

Compilation and optimization are a function of code, context, and time.

Original Tradeoff

AOT can use as much time as needed, with deep code analysis, knowledge of execution platform, and additional hints for optimization. The disadvantages are slower builds, less portable code, bad optimizations based on wrong assumptions, and lacking support for advanced features like reflection during runtime.

JIT is much more portable and can support advanced code changes at runtime, but takes longer to start and has limited time and resources which means limited optimizations.

Modern Developments

The latest JIT compilers are much faster and start much quicker while still producing good code, closing most of the gap with AOT for startup latency.

Multiple compilation passes allow JIT to continually optimize the program as it runs, leading to similar performance as AOT after the warm-up period. The JIT can look at what code is actually used, how often it's used, and other factors like the running hardware that aren't available during AOT compilation.

The most advanced JIT compilers can meet or exceed AOT performance and maintain it under more scenarios. AOT is still the best option for fast startup and consistent/predictable operation, but no longer the default option for best overall performance.

Mani Gandham
  • 5,581
  • 43
  • 54
  • AOT can be just as portable as JIT (and in some cases the exact same byte-code is used for both AOT and JIT - e.g. Microsoft's CIL); and JIT can be "not portable at all" (e.g. no JIT compiler for the system you want to use). JIT can also have missing or incorrect optimizations (and almost always has deliberately missed optimizations because they were too expensive to do at run-time). – Brendan Oct 08 '20 at 21:16
  • For performance, AOT is typically slower that it could/should be (e.g. optimized for a generic target by developer & not optimized for user's specific machine) but that's a weakness in current tools (not a weakness of AOT itself); and modern JIT still fails to be as good as "not great AOT". To make up for how awful "advanced JIT" performance is they also (almost always) rely on large libraries full of "AOT compiled" native code to hide the performance disaster. There has never been any "advanced JIT" that is able to beat AOT in a fair comparison (and never will be). – Brendan Oct 08 '20 at 21:22
  • @Brendan Those are pedantic what-if's, not an argument about the concepts. CIL is irrelevant. JIT is more portable because it can run the same artifact on many machines instead of requiring a matching prebuilt binary. Having no JIT compiler available is the same as having no AOT support for the platform (an implementation detail). And as stated, the secondary passes overcome any runtime limitation by amortizing compilation across app lifetime while matching actual usage. JIT can do everything AOT can, but can also do better by using input factors that AOT is fundamentally unable to use. – Mani Gandham Oct 09 '20 at 03:33
  • Do me a few favors - install Gentoo Linux on a computer before trying to tell me AOT can't optimize for the specific machine, and/or learn what "run-time dispatch" (that Intel's ICC compiler does) is; then try to run Java on MS-DOS to find out how portable that is. Then try benchmarking "pure" Java code (without using a library full of AOT compiled native code; and without AOT compiling it to byte-code first). Finally, try amortizing the unnecessary performance disaster on a short lived small utility (where an AOT program will finish before you've even loaded a huge JIT compiler). – Brendan Oct 09 '20 at 06:35
  • @Brendan Obviously if JIT doesn't have ever have as much time as AOT then it's physically impossible to accomplish the same results. This is already covered in the answer. You're so busy with anecdotes and some dogmatic fight against JIT that you're conflating concepts, implementations, and edge cases. Let's end it here. – Mani Gandham Oct 09 '20 at 14:19