How to use Caliper for benchmarking?

Question

I am trying to figure out how to use Caliper to do benchmark testing in Eclipse and I am getting nowhere. I tried following the 26 minute tutorial found here: https://code.google.com/p/caliper/ but I get lost quickly. I have downloaded the Caliper jar file but I'm not sure what folder it should be in. I've also downloaded Maven for Eclipse plugin but I'm not even sure if that is necessary. Is it possible to install Caliper from the 'Install New Software..' option in the Help menu in Eclipse? I just want to be able to do very simple speed tests for some of the algorithms I've created for a Data Structures and Algorithms class I am taking.

Answer 1

This answer is now obsolete. Caliper has worked in Windows for more than a year, at least: https://code.google.com/p/caliper/issues/detail?id=167

---

Caliper doesn't work in Windows. See this case. You need to use version 0.5-rc1, which has other issues but is still pretty okay and is missing a lot of features, but it does work in Windows.

• If you know how to use Maven, add this pom snippet to your pom.xml.

  <dependency>
      <groupId>com.google.caliper</groupId>
      <artifactId>caliper</artifactId>
      <version>0.5-rc1</version>
  </dependency>
  

  • If you want to learn maven, first read this: http://maven.apache.org/guides/getting-started/maven-in-five-minutes.html
  • Convert your project to a maven project (Right click on project -> Configure -> Convert to Maven Project)
• If you don't know how to use Maven (here is a guide to how to do this with pictures):
  • Download the 0.5-rc1 jar
  • Right click on the project you want to use and choose Build Path -> Configure Build Path
  • Add it to your libraries tab using Add External Jar

Once you've done that, you can start writing benchmarks. Here is an example of a benchmark I wrote for a different Stack Overflow question.

Answer 2

Here is how you set up a working Caliper class using the latest version of Caliper as of this writing, caliper-1.0-beta2 . As far as I can tell this procedure isn't documented anywhere outside of inline comments in the Caliper code files.

First install caliper-1.0-beta2 in pom.xml or by downloading the jar file. Then make a file like this:

import com.google.caliper.Benchmark;
import com.google.caliper.runner.CaliperMain;

public class DemoBenchmark {
  public static void main(String[] args) {
    CaliperMain.main(DemoBenchmark.class, args);
  }

  @Benchmark
  void timeStringBuilder(int reps) {
    StringBuilder sb = new StringBuilder();
    for (int i = 0; i < reps; i++) {
      sb.setLength(0);
      sb.append("hello world");
    }
  }
}

Run that file, and Caliper will do the benchmark for you.

Answer 3

I have tried all the suggested solutions on Windows, however without success.

I always encountered the following error:

This selection yields 4 experiments.
ERROR: Trial failed to complete (its results will not be included in the run):
  The worker exited without producing data. It has likely crashed. Inspect C:\Users\Piotr\AppData\Local\Temp\1583760443321-0\trial-1.log to see any worker output.

I was able to solve it by adding the @VmOptions annotation to a custom benchmark class.

Here is the full configuration:

@VmOptions("-XX:-TieredCompilation")
public class CaliperBenchmark {

    @BeforeExperiment
    void setUp() {
        // set up
    }

    @Benchmark
    void boxed() {
       // test
    }
}

Maven: com.google.caliper caliper 1.0-beta-2 compile

Main class:

public class Main {

    public static void main(String[] args) {
        CaliperMain.main(CaliperBenchmark.class, args);
    }
}

Command line: mvn exec:java -Dexec.mainClass="com.google.caliper.runner.CaliperMain" -Dexec.args="org.CaliperBenchmark"