Properly implementing Java modules in a Maven build with inter-module test dependencies

Question

I have a multi-module project using Maven and Java. I am now trying to migrate to Java 9/10/11 and implement modules (as in JSR 376: Java Platform Module System, JPMS). As the project was already consisting of Maven modules, and the dependencies were straight, creating module descriptors for the project was quite straight forward.

Each Maven module now has their own module descriptor (module-info.java), in the src/main/java folder. There is no module descriptor for the test classes.

However, I stumbled upon a problem I have not been able to solve, and not found any descriptions on how to solve:

How can I have inter-module test dependencies with Maven and Java modules?

In my case, I have a "common" Maven module, which contains some interfaces and/or abstract classes (but no concrete implementation). In the same Maven module, I have abstract tests to ensure proper behavior for the implementation of these interfaces/abstract classes. Then, there are one or more sub modules, with implementations of the interface/abstract class and tests extending the abstract test.

However, when trying to execute the test phase of the Maven build, the sub module will fail with:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.0:testCompile (default-testCompile) on project my-impl-module: Compilation failure: Compilation failure:
[ERROR] C:\projects\com.example\my-module-test\my-impl-module\src\test\java\com\example\impl\FooImplTest.java:[4,25] error: cannot find symbol
[ERROR]   symbol:   class FooAbstractTest
[ERROR]   location: package com.example.common

I suspect that this happens because the tests are not part of the module. And even if Maven does some "magic" to get the tests executed within the scope of the module, it doesn't work for the tests in the module I depend on (for some reason). How do I fix this?

The structure of the project looks like this (full demo project files available here):

├───my-common-module
│   ├───pom.xml
│   └───src
│       ├───main
│       │   └───java
│       │       ├───com
│       │       │   └───example
│       │       │       └───common
│       │       │           ├───AbstractFoo.java (abstract, implements Foo)
│       │       │           └───Foo.java (interface)
│       │       └───module-info.java (my.common.module: exports com.example.common)
│       └───test
│           └───java
│               └───com
│                   └───example
│                       └───common
│                           └───FooAbstractTest.java (abstract class, tests Foo)
├───my-impl-module
│   ├───pom.xml
│   └───src
│       ├───main
│       │   └───java
│       │       ├───com
│       │       │   └───example
│       │       │       └───impl
│       │       │           └───FooImpl.java (extends AbstractFoo)
│       │       └───module-info.java (my.impl.module: requires my.common.module)
│       └───test
│           └───java
│               └───com
│                   └───example
│                       └───impl
│                           └───FooImplTest.java (extends FooAbstractTest)
└───pom.xml

Dependencies in the my-impl-module/pom.xml is as follows:

<dependencies>
    <dependency>
        <groupId>com.example</groupId>
        <artifactId>my-common-module</artifactId>
        <scope>compile</scope>
    </dependency>
    <dependency>
        <groupId>com.example</groupId>
        <artifactId>my-common-module</artifactId>
        <classifier>tests</classifier> <!-- tried type:test-jar instead, same error -->
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>junit</groupId>
        <artifactId>junit</artifactId>
        <scope>test</scope>
    </dependency>
</dependencies>

Note: The above is just a project I created to demonstrate the problem. The real project is a lot more complex, and found here (master branch is not modularized yet), but the principle is the same.

PS: I don't think there's anything wrong with the code itself, as everything compiles and runs using normal class path (ie. in IntelliJ, or Maven without the Java module descriptors). The problem is introduced with Java modules and the module path.

Answer 1

Based on your demo project, I was able to duplicate your error. That said, here are the revised changes I made, after my first failed attempt, to be able to build the project:

1. I added the maven-compiler-plugin version 3.8.0 to all the modules. You need a version of 3.7 or higher to compile modules with Maven - at least that's the warning NetBeans showed. Since there is no harm, I added the pluging to both the common and implementation modules' POM files:

   <plugin>
       <groupId>org.apache.maven.plugins</groupId>
       <artifactId>maven-compiler-plugin</artifactId>
       <version>3.8.0</version>
       <executions>
           <execution>
               <goals>
                   <goal>compile</goal>
               </goals>
               <id>compile</id>
           </execution>
       </executions>
   </plugin> 
   

2. I exported the test classes into their own jar file so they will be available to your implementation module or anyone for that matter. To do that, you need to add the following to your my-common-module/pom.xml file:

   <plugin>
       <groupId>org.apache.maven.plugins</groupId>
       <artifactId>maven-jar-plugin</artifactId>
       <version>3.1.0</version>
       <executions>
           <execution>
               <id>test-jar</id>
               <phase>package</phase>
               <goals>
                   <goal>test-jar</goal>
               </goals>
           </execution>
       </executions>
   </plugin>
   

   This will export my-common-module test classes into -tests.jar file - i.e.my-common-module-1.0-SNAPSHOT-tests.jar. Notice there is no need to add an execution for the regular jar file as noted in this post. This will, however, introduced error that I will address next.

3. Rename your test package in my-common-module to com.example.common.test in order for the test classes to be loaded when compiling the implementation test class(es). This corrects the class load issue introduced when we exported the test classes with the same package name as in the module where the first jar, in this case the module, is loaded and the second jar, the test jar file, is ignored. Interesting enough, I'm concluding, based on observation, that the module path has higher precedence than the class path since the Maven compile parameters shows the tests.jar is specified first in the class path. Running mvn clean validate test -X, we see compile parameters:

   -d /home/testenv/NetBeansProjects/MavenProject/Implementation/target/test-classes -classpath /home/testenv/NetBeansProjects/MavenProject/Implementation/target/test-classes:/home/testenv/.m2/repository/com/example/Declaration/1.0-SNAPSHOT/Declaration-1.0-SNAPSHOT-tests.jar:/home/testenv/.m2/repository/junit/junit/4.12/junit-4.12.jar:/home/testenv/.m2/repository/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar: --module-path /home/testenv/NetBeansProjects/MavenProject/Implementation/target/classes:/home/testenv/.m2/repository/com/example/Declaration/1.0-SNAPSHOT/Declaration-1.0-SNAPSHOT.jar: -sourcepath /home/testenv/NetBeansProjects/MavenProject/Implementation/src/test/java:/home/testenv/NetBeansProjects/MavenProject/Implementation/target/generated-test-sources/test-annotations: -s /home/testenv/NetBeansProjects/MavenProject/Implementation/target/generated-test-sources/test-annotations -g -nowarn -target 11 -source 11 -encoding UTF-8 --patch-module example.implementation=/home/testenv/NetBeansProjects/MavenProject/Implementation/target/classes:/home/testenv/NetBeansProjects/MavenProject/Implementation/src/test/java:/home/testenv/NetBeansProjects/MavenProject/Implementation/target/generated-test-sources/test-annotations: --add-reads example.implementation=ALL-UNNAMED
   

4. We need to make the exported test classes available to the implementation module. Add this dependency to your my-impl-module/pom.xml:

   <dependency>
       <groupId>com.example</groupId>
       <artifactId>Declaration</artifactId>
       <version>1.0-SNAPSHOT</version>
       <type>test-jar</type>
       <scope>test</scope>
   </dependency>
   

5. Lastly in the my-impl-module test class, update the import to specify the new test package, com.example.common.text, to access the my-common-module test classes:

   import com.example.declaration.test.AbstractFooTest;
   import com.example.declaration.Foo;
   import org.junit.Test;
   import static org.junit.Assert.*;
   
   /**
    * Test class inheriting from common module...
    */
   public class FooImplementationTest extends AbstractFooTest { ... }
   

Here is the test results from my mvn clean package of the new changes:

I updated my sample code in my java-cross-module-testing GitHub repo. The only lingering question I have, and I'm sure you do as well, is why did it worked when I defined the implementation module as a regular jar project instead of a module. But that, I'll play with some other day. Hopefully what I provided solves your problem.

Answer 2

I tried to do exactly the same, it's not possible to have both whitebox tests and module test-dependencies with your project structure, but I think I found an alternative structure that does 90% of what you want to do:

1/ The problem with whitebox testing is that it works with module patching, because JPMS has not notion of test VS main unlike Maven. So this provokes problems like not working with test-dependencies, or having to pollute your module-info with test dependencies.

2/ So then, why not keep doing whitebox testing, but with the maven structure of blackbox testing, that is: split each module X into X and X-test. Only X has a module-info.java, tests run in the classpath so you skip all these issues.

The only drawbacks I can think of are (in order of increasing importance):

1. That the test jar will not be modularized, but I think that is acceptable (at least for now).
2. That you have twice the number of maven modules, and you might not like separating tests in another maven module.
3. Tests will run in the classpath, and it's always bad to run tests in a different environment (the tests will run with less restrictions than the main code). This could maybe be mitigated by smoke tests or by a dedicated integration tests module? The "official patterns" have not emerged yet.

By the way (if that's what you're doing) I doubt it's worth it modularizing eg. a spring boot app as spring itself is not modularized yet, so you will pay the cost but reap few benefits (but if you're writing a lib that's a different story).

You can find an example here:

a/ Get the example:

git clone https://github.com/vandekeiser/ddd-metamodel.git

git checkout stackoverflow

b/ Look at the example:

1. fr.cla.ddd.metamodel depends on fr.cla.ddd.oo (but no module-info for the tests maven modules)
2. I do whitebox testing in PackagePrivateOoTest and in PackagePrivateMetamodelTest
3. I have a test-depency OoTestDependency