Jenkins + Play 1.2.4 : problems with cobertura lock files / report

Question

We have a Play 1.2.4 application and we got Jenkins (on Ubuntu) for the application. We're having problems with Cobertura.

After running the tests (succesfully), every now and then, we get the following error:

---------------------------------------
java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at net.sourceforge.cobertura.util.FileLocker.lock(FileLocker.java:124)
        at play.modules.cobertura.CoberturaPlugin$CoberturaPluginShutdownThread.run(Unknown Source)
Caused by: java.nio.channels.OverlappingFileLockException
        at sun.nio.ch.FileChannelImpl$SharedFileLockTable.checkList(FileChannelImpl.java:1166)
        at sun.nio.ch.FileChannelImpl$SharedFileLockTable.add(FileChannelImpl.java:1068)
        at sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:824)
        at java.nio.channels.FileChannel.lock(FileChannel.java:860)
        ... 6 more
---------------------------------------
Unable to get lock on /var/lib/jenkins/jobs/project/workspace/cobertura.ser.lock: null
This is known to happen on Linux kernel 2.6.20.
Make sure cobertura.jar is in the root classpath of the jvm 
process running the instrumented code.  If the instrumented code 
is running in a web server, this means cobertura.jar should be in 
the web server's lib directory.
Don't put multiple copies of cobertura.jar in different WEB-INF/lib directories.
Only one classloader should load cobertura.  It should be the root classloader.
---------------------------------------
lock file could not be deleted

This doesn't seem to "break the build" but further down the build, we get the following (which causes the cobertura reports to fail)

Publishing Cobertura coverage report...
No coverage results were found using the pattern 'test-result/code-coverage/coverage.xml' relative to '/var/lib/jenkins/jobs/project/workspace'.  Did you enter a pattern relative to the correct directory?  Did you generate the XML report(s) for Cobertura?
Build step 'Publish Cobertura Coverage Report' changed build result to FAILURE

Running a subsequent build manually usually passes.

According to Zero code coverage with cobertura 1.9.2 but tests are working , I tried setting -Dcobertura.use.java.nio=false after play auto-test -command.

As this error was happening only now and then, I'm not totally sure if this helped. But after that, we got a problem with play auto-test hanging:

  ...
  Executing /opt/play-1.2.4/play auto-test "/var/lib/jenkins/jobs/project/workspace"  -Dcobertura.use.java.nio=false
  [workspace] $ /opt/play-1.2.4/play auto-test "/var/lib/jenkins/jobs/project/workspace" -Dcobertura.use.java.nio=false
  <build stuck here for a couple of days>

Since nothing has been totally deterministic, it's a bit difficult to say about causalities here. (This seems to happen after one or two builds after jenkins/server restart)

Currently I'm considering disabling Cobertura in our project, but if somebody has other ideas, that would be great =)

We have exactly the same problem! I also tried cobertura.use.java.nio in the same way and it also hangs... — valanto, Oct 08 '12 at 08:30
same setup as you I think. Running jenkins on an Ubuntu machine. Using play1.2.4, cobertura module 2.4, latest jenkins. we run play auto-test also on jenkins and when i tried the -Dcobertura.use.java.nio=false it also hanged. our failures with cobertura are also on and off... — valanto, Oct 08 '12 at 08:54
I managed to overcome this by adding a VM as slave to Jenkins. Master is running Ubuntu and not getting any build jobs, slave is running Centos and hasn't shown the error yet (around 300 builds). I have no idea what started causing the error and I occasionally get it on my Win7 workstation (giving the same message about the Linux kernel :P) — Leonidas K, Oct 15 '12 at 22:37
@LeonidasK : So Centos doesn't seem to have this problem that occurs with Ubuntu (at least so far) — Touko, Oct 24 '12 at 08:58
I can also confirm this issue, and I have seen this happening on Mac OS X. — Somatik, Dec 17 '12 at 19:06
I have the impression cobertura is running twice. I see "...Loaded information..." twice for each invocation of the plugin. — Somatik, Dec 17 '12 at 19:09
Try playing with permissions. I was having a similar problem that had to do with the permissions of a file. Jenkins couldn't read/write the file from one command in a build script, but could in another. We ended up `chmod 777 -R ~jenkins` on our machine, haha. — reagan, Jun 23 '13 at 03:58

score 2 · Answer 1 · answered Jul 07 '13 at 15:32

Clearly this is due JVM locking issues either in your JVM implementation, or rather, in the way you are deploying your cobertura JAR.

Jenkins can spawn up alot of JVM threads, and if cobetura is on your global classpath, its possible that some weird collisions are happening.

I assume, ultimately, that this should be attributed to a minor bug in cobertura (unless the complex corbertura file locking is solving some other more important problem).

According to the source code for Cobertura's FileLock (cobertura/src/main/java/net/sourceforge/cobertura/util/FileLocker.java), there are some issues around multiple JVM's loading up the Cobertura jar.

To solve, make sure there is only one copy and one app launching and using Corbetura.

The reason your VM implementation fixed it, most likely, is that you've decreased the amount of variability in the way cobetrura can be loaded. Also maybe you are restarting your VM with higher frequency then your jenkins server.

In our jenkins corbertura builds, we just use the maven plugin and this seems to work fine without issue (but then again, we are not using java 1.7, nor are we using Play).

score 2 · Answer 2 · answered Jul 29 '13 at 21:14

This has been bothering us for a while (play 1.2.4/Jenkins). There is some problem due to overlapping sequences between jenkins cobertura plugin (report publish) and play framework cobertura module. I believe its purely timing coincidence and hence intermittent. We have the following work around for the lack of a better resolution.

Removed jenkins cobertura report publish action from the main build job. We created a new jenkins job that is setup with publish cobertura coverage report action. In the new job, we have the shell action to copy the coverage.xml from the main build job workspace to the workspace of the new job for cobertura coverage report publish action to run. The copy (for obvious reasons) is done to avoid running both play cobertura and jenkins cobertura in the same job.

Its not the best but happy to see the coverage report/graphs :-)

score 2 · Answer 3 · answered Mar 28 '14 at 18:20

The trick is to use one datafile (cobertura.ser) per module to avoid locks from parallel tasks.

With ant:

<cobertura-instrument todir="${build.dir}" datafile="cobertura.ser.${modulename}">
    ...

At end merge the many cobertura files into one cobertura file:

<target name="merge-coverage">
    <cobertura-merge datafile="cobertura.ser">
        <fileset dir="${build.dir}">
            <include name="cobertura.ser.*" />
        </fileset>
    </cobertura-merge>
</target>

score 1 · Answer 4 · edited Jul 25 '13 at 13:20

-Dcobertura.use.java.nio=false the previous appears to require changing to true to be able to use file locking as your error message explained.

Also, somewhere the application probably requires adding the full folder classpath for cobertura.

It appears you are using something similar to a COF(constantly open file) , the error message is referring to a file that exists, but regions of the file are locked n the drive, not the file itself simply.

score 0 · Answer 5 · answered Jun 28 '13 at 07:17

0

Did you set

%test.play.tmp=none

in your application.conf file ?

answered Jun 28 '13 at 07:17

Seb Cesbron

3,759
13
17

2

This should be phrased as an answer, not as a question. – jayunit100 Jul 07 '13 at 15:21

Jenkins + Play 1.2.4 : problems with cobertura lock files / report

5 Answers5