2

I have a maven project with test execution by the maven-surefire-plugin. An odd phenomenon I've observed and been dealing with is that running locally

mvn clean install

which executes my tests, results in a successful build with 0 Failures and 0 Errors.

Now when I deploy this application to our remote repo that Jenkins attempts to build, I get all sorts of random EasyMock errors, typically of the sort:

java.lang.IllegalStateException: 3 matchers expected, 4 recorded. at org.easymock.internal.ExpectedInvocation.createMissingMatchers

This is a legacy application being inherited, and we are aware that many of these tests are flawed if not plainly using EasyMock incorrectly, but I'm in a state where with test execution I get a successful build locally but not in Jenkins.

I know that the order of execution of these tests is not guaranteed, but I am wondering how I can introspect what is different in the Jenkins build pipeline vs. local to help identify the issue?

Is there anything I can do to force execute the tests in the way they're done locally? At this point, I have simply excluded many troublesome test classes but it seems that no matter how many times I see a Jenkins failure, I either fix the problem or exclude the test class, I'm only to find it complain about some other test class it didn't mention before.

Any ideas how to approach a situation like this?

Adam Bronfin
  • 1,069
  • 20
  • 39

2 Answers2

2

I have experimented quite a similar situation, and the cause of mine was obviously some concurrency problems with the tests implementations.

And, after reading your comment:

What I actually did that fixed it (like magic am I right?) is for the maven-surefire plugin, I set the property reuseForks=false, and forkCount=1C, which is just 1*(number of CPU's of machine).

... I get more convinced that you have concurrency problems with your tests. Concurrency is not easy to diagnose, specially when your experiment runs OK on one CPU. But race conditions might arise when you run it on another system (which usually is faster or slower).

I recommend you strongly to review your tests one by one and ensure that each one of them is logically isolated:

  • They should not rely upon an expected previous state (files, database, etc). Instead, they should prepare the proper setup before each execution.
  • If they modify concurrently a common resource which might interfere other test's execution (files, database, singletons, etc), every assert must be done synchronizing as much as needed, and taking in account that its initial state is unknown:

Wrong test:

MySingleton.getInstance().put(myObject);
assertEquals(1, MySingleton.getInstance().size());

Right test:

synchronized(MySingleton.getInstance())
{
    MySingleton.getInstance().put(myObject);
    assertTrue(MySingleton.getInstance().contains(myObject));
}

A good start point for the reviewing is checking one of the failing tests and track the execution backwards to find the root cause of the fail.

Setting explicitly the tests' order is not a good practice, and I wouldn't recommend it to you even if I knew it was possible, because it only would hide the actual cause of the problem. Think that, in a real production environment, the executions' order is not usually guranteed.

Little Santi
  • 7,940
  • 2
  • 14
  • 40
  • 1
    Totally agree with everything said here, and the issues you identified are indeed the problem with many if not the majority of the tests. However there are hundreds and hundreds of them and rewriting or refactoring that many tests is not something we have time to do now. Also unfortunate that it's also simply not easy to fix them, as for one many of the tests that fail in build succeed locally, and the errors typically come from EasyMock and are often not descriptive enough to easily diagnose. – Adam Bronfin Mar 18 '16 at 16:56
  • Yes, I see... Well, your try with `reuseForks` and `forkCount` has been clever enough. Take also a look at `threadCount*`and `parallel` parameters: http://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html – Little Santi Mar 18 '16 at 22:01
1

JUnit test run order is non-deterministic.

Are the versions of Java and Maven the same on the 2 machines? If yes, make sure you're using the most recent maven-surefire-plugin version. Also, make sure to use a Freestyle Jenkins job with a Maven build step instead of the Maven project type. Using the proper Jenkins build type can either fix build problems outright or give you a better error so you can diagnose the actual issue.

You can turn on Maven debug logging to see the order tests are being run in. Each test should set up (and perhaps tear down) its own test data to make sure the tests may run independently. Perhaps seeing the test order will give you some clues as to which classes depend on others inappropriately. And - if the app uses caching, ensure the cache is cleaned out between tests (or explicitly populated depending on what the test needs to do). Also consider running the tests one package at a time to isolate the culprits - multiple surefile plugin executions might be useful.

Also check the app for classpath problems. This answer has some suggestions for cleaning the classpath.

And another possibility: Switching to a later version of JUnit might help - unless the app is using Spring 2.5.6.x. If the app is using Spring 2.5.6.x and cannot upgrade, the highest possible version of JUnit 4.x that may be used is 4.4. Later versions of JUnit are not compatible with Spring Test 2.5.6 and may lead to hard-to-diagnose test errors.

Community
  • 1
  • 1
user944849
  • 13,003
  • 2
  • 55
  • 76
  • What I actually did that fixed it (like magic am I right?) is for the maven-surefire plugin, I set the property reuseForks=false, and forkCount=1C, which is just 1*(number of CPU's of machine). – Adam Bronfin Mar 18 '16 at 05:58