Maven - multi-module Java project code coverage . . .

In addition to developing, building & deploying modern Java applications is also a developers' concern. Build tools come with a promise to save developers' time. But often times they suck developers in. Maven is known for that ;)

Choosing the right building tool before you even start an application/project is a standard practice these days. In modern Java world, the two popular build system choices are: Maven and Gradle. Though Gradle started as the "Next Generation Build Tool" with groovy programming language and well designed DSL to write build scripts, and by addressing pitfalls of Maven, maven still rules modern Java world with the legacy XML (Extensible Markup Language), which is not a programming language and is only good for structured data management. The modern software principle "as code" applied to everything these days, even to infrastructure, still doesn't apply to Maven build scripts.

A multi-module project is inevitable if you build any application with modularity and reusability. Maven poses great many challenges in this use-case. There are solutions available for every issue, but you end up spending too much time reading the poor documentation again and again scratching your head, doing more of the same with plugins documentations, and even more of the same in the form of question & answers on the stackoverflow. Clearly, this is not the way, but unfortunately is the way to find solutions, these days.

This post is the result of a 3-day fight with Maven in getting the multi-module code coverage working in a Maven multi-module project. The following 3 maven plugins are in main focus in this are(n)a:

• JaCoCo for code coverage report generation
• Surefire for unit tess
• Failsafe for integration tests

Environment: Java 16, Spring Boot 2.5.3 on macOS Catalina 10.15.7

The Problem Scenario - code in one module, test-cases in another module

It is quite common in multi-module project to have code in one module, and some test-cases if not all in other module(s). For instance, a multi-module maven project with a sharable domain module, a sharable services module and an API micro-service application module (spring-boot based) is a best example of this scenario.

In this scenario, for example, the domain model code can get it's code coverage from unit test-cases as both source code and test-cases reside in the same module. The Maven Jacoco code coverage plugin works quite well in this case. But, it could be bit hard to write integration test-cases for the services module as it requires spring application context and spring-boot configurations. So, the API application module will definitely have a set of integration test-cases as it is a spring boot application with spring context and configurations available. This is the case that requires a better solution for generating code coverage reports by covering the multi-module distributed application code with distributed test-cases.

The two key-points in this scenario are:

1. Test-cases in one module (API application) covering code in other module (services/domain) in addition to the other module's own code coverage.
2. An individual module-wise code coverage report for all the modules with their own code coverage by their own test-cases and coverage threshold checks.
3. A overall consolidated/aggregated but module-wise code coverage report for the entire application code and a code coverage threshold check for the overall code in all modules.

The Solution

The JaCoCo maven plugin from version 0.7.7 onwards offers a new report-aggregate goal. This is the goal that can be leveraged to get an aggregated code coverage report generated. However, it is not straight forward getting this done.

The following is an example multi-module project structure, my-app is the root project, my-app-api is a Spring Boot application with my-app-domain and my-app-services modules that it depends on:

. └── my-app ├── my-app-api │ └── pom.xml ├── my-app-domain │ └── pom.xml ├── my-app-services │ └── pom.xml ├── my-app-code-coverage │ └── pom.xml └── pom.xml

The project's root module my-app's pom.xml file looks something as shown below:

... <modules> <module>my-app-api</module> <module>my-app-domain</module> <module>my-app-services</module> <module>my-app-code-coverage</module> </modules> ... <properties> <jacoco.plugin.version>0.8.7</jacoco.plugin.version> <surefire.plugin.version>2.22.2</surefire.plugin.version> <failsafe.plugin.version>2.22.2</failsafe.plugin.version> </properties> ... <build> <plugins> <!-- jacoco for code coverage --> <plugin> <groupId>org.jacoco</groupId> <artifactId>jacoco-maven-plugin</artifactId> <version>${jacoco.plugin.version}</version> <executions> <!-- jacoco agent for unit-tests code coverage --> <execution> <id>initialize-coverage-before-unit-test-execution</id> <goals> <goal>prepare-agent</goal> </goals> </execution> <!-- jacoco agent for integration-tests code coverage --> <execution> <id>initialize-coverage-before-integration-test-execution</id> <goals> <goal>prepare-agent</goal> </goals> <phase>pre-integration-test</phase> <configuration> <propertyName>integrationTestCoverageAgent</propertyName> </configuration> </execution> </executions> </plugin> <!-- UNIT tests--> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-surefire-plugin</artifactId> <version>${surefire.plugin.version}</version> <configuration> <excludes> <exclude>**/*IT.java</exclude> </excludes> <!-- NOTE: In case if you need to pass in special JVM argumengts for instance say, to enable Java language preview features, the following is how it MUST be done. The @{argLine} goes through late evaluattion that points to Jacoco agent JVM argument followed by any additional JVM arguments of your choice each separated by space. The expression @{argLine} retains Jacoco JVM agent argument. Without this any additional JVM arguments that you add will be taken but you lose Jacoco's argument causing to lose code coverage report files and hence the coverage report. --> <argLine>@{argLine} --enable-preview</argLine> </configuration> </plugin> <!-- INTEGRATION tests --> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-failsafe-plugin</artifactId> <version>${failsafe.plugin.version}</version> <executions> <execution> <id>integration-tests</id> <goals> <goal>integration-test</goal> <goal>verify</goal> </goals> <configuration> <additionalClasspathElements> <additionalClasspathElement>${basedir}/target/classes</additionalClasspathElement> </additionalClasspathElements> <includes> <include>**/*IT.java</include> </includes> <excludes> <exclude>com.my.api.service.MyNotUsedServiceIT</exclude> </excludes> <!-- NOTE: When running as a Maven plugin, the JaCoCo agent configuration is prepared by invoking the prepare-agent or prepare-agent-integration goals, before the actual tests are run. This sets a property named argLine which points to the JaCoCo agent, later passed as a JVM argument to the test runner. --> <argLine>${integrationTestCoverageAgent}</argLine> <!-- NOTE: In case if you need to pass in special JVM argumengts for instance say, to enable Java language preview features, the following is how it MUST be done. The @{argLine} goes through late evaluattion that points to Jacoco agent JVM argument followed by any additional JVM arguments of your choice each separated by space. The expression @{argLine} retains Jacoco JVM agent argument. Without this any additional JVM arguments that you add will be taken but you lose Jacoco's argument causing to lose code coverage report files and hence the coverage report. --> <argLine>@{argLine} --enable-preview</argLine> </configuration> </execution> </executions> </plugin> </plugins> </build> ...

Notable points from the above build file around code coverage are:

• The JaCoCo plugin configuration for code coverage with two execution configurations for unit and integration tests coverage. Make a note of the configuration <propertyName>integrationTestCoverageAgent</propertyName >, it can be any string. The same name should be passed as an argument (argLine) for failsafe configuration. Also, make sure that you retain Jacoco JVM agent argument pointed to by Jacoco added argLine property to be evaluated late in the game as the recommended expression @{argLine} in case if you have additional JVM arguments to be passed for test executions.
• The surefire plugin configuration for unit tests.
• The failsafe plugin configuration for integration tests.

The domain my-app-domain module's pom.xml file is something like shown below:

... <build> <plugins> <plugin> <groupId>org.jacoco</groupId> <artifactId>jacoco-maven-plugin</artifactId> <version>${jacoco.plugin.version}</version> <executions> <execution> <id>generate-code-coverage-report</id> <phase>test</phase> <goals> <goal>report</goal> </goals> </execution> <execution> <id>perform-code-coverage-threshold-check</id> <goals> <goal>check</goal> </goals> <configuration> <!-- Set Rule to fail build if code coverage is below certain threshold --> <rules> <rule implementation="org.jacoco.maven.RuleConfiguration"> <element>BUNDLE</element> <limits> <limit implementation="org.jacoco.report.check.Limit"> <counter>INSTRUCTION</counter> <value>COVEREDRATIO</value> <minimum>0.60</minimum> </limit> </limits> </rule> </rules> </configuration> </execution> </executions> </plugin> ... </plugins> ... </build> ...

The api my-app-api module's pom.xml file is something like shown below, very similar to my-app-domain module:

... <build> <plugins> <plugin> <groupId>org.jacoco</groupId> <artifactId>jacoco-maven-plugin</artifactId> <version>${jacoco.plugin.version}</version> <executions> <execution> <id>generate-code-coverage-report</id> <phase>test</phase> <goals> <goal>report</goal> </goals> </execution> <execution> <id>perform-code-coverage-threshold-check</id> <goals> <goal>check</goal> </goals> <configuration> <!-- Set Rule to fail build if code coverage is below certain threshold --> <rules> <rule implementation="org.jacoco.maven.RuleConfiguration"> <element>BUNDLE</element> <limits> <limit implementation="org.jacoco.report.check.Limit"> <counter>INSTRUCTION</counter> <value>COVEREDRATIO</value> <minimum>0.80</minimum> </limit> </limits> </rule> </rules> </configuration> </execution> </executions> </plugin> ... </plugins> ... </build> ...

Notable points from the above two modules' build files around code coverage are:

• The JaCoCo plugin's additional configuration for code coverage with execution configurations for code coverage report (goal: report) and code coverage threshold check (goal: check) with a threshold ration number (0.60 for domain and 0.80 for api).
• No surefire and failsafe configurations are needed as they are available in sub-modules from the root/main module's build configuration.

The new module, my-app-code-coverage module's pom.xml file for consolidated code coverage is like shown below:

<?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <parent> <groupId>com.giri</groupId> <artifactId>my-app</artifactId> <version>1.0.0-SNAPSHOT</version> </parent> <modelVersion>4.0.0</modelVersion> <artifactId>my-app-code-coverage</artifactId> <packaging>pom</packaging> <name>My Api App Service multi-module code coverage</name> <description>Module for My Api App multi-module code coverage across all modules</description> <properties> <code.coverage.project.dir>${basedir}/../</code.coverage.project.dir> <code.coverage.overall.data.dir>${basedir}/target/</code.coverage.overall.data.dir> <maven-resources-plugin.version>3.2.0</maven-resources-plugin.version> </properties> <dependencies> <dependency> <groupId>com.giri</groupId> <artifactId>my-app-domain</artifactId> <version>${project.version}</version> <scope>compile</scope> </dependency> <dependency> <groupId>com.giri</groupId> <artifactId>my-app-services</artifactId> <version>${project.version}</version> <scope>compile</scope> </dependency> <dependency> <groupId>com.giri</groupId> <artifactId>my-app-api</artifactId> <version>${project.version}</version> <scope>compile</scope> </dependency> </dependencies> <build> <plugins> <!-- required by jacoco for the goal: check to work --> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-resources-plugin</artifactId> <version>${maven-resources-plugin.version}</version> <executions> <execution> <id>copy-class-files</id> <phase>generate-resources</phase> <goals> <goal>copy-resources</goal> </goals> <configuration> <overwrite>false</overwrite> <resources> <resource> <directory>../my-app-domain/target/classes</directory> </resource> <resource> <directory>../my-app-services/target/classes</directory> </resource> <resource> <directory>../my-app-api/target/classes</directory> </resource> </resources> <outputDirectory>${project.build.directory}/classes</outputDirectory> </configuration> </execution> </executions> </plugin> <plugin> <groupId>org.jacoco</groupId> <artifactId>jacoco-maven-plugin</artifactId> <version>${jacoco.plugin.version}</version> <executions> <execution> <id>report-aggregate</id> <phase>verify</phase> <goals> <goal>report-aggregate</goal> </goals> </execution> <execution> <id>merge-results-data</id> <phase>verify</phase> <goals> <goal>merge</goal> </goals> <configuration> <fileSets> <fileSet> <directory>${code.coverage.project.dir}</directory> <includes> <include>**/target/jacoco.exec</include> </includes> </fileSet> </fileSets> <destFile>${code.coverage.overall.data.dir}/aggregate.exec</destFile> </configuration> </execution> <execution> <id>perform-code-coverage-threshold-check</id> <phase>verify</phase> <goals> <goal>check</goal> </goals> <configuration> <dataFile>${code.coverage.overall.data.dir}/aggregate.exec</dataFile> <rules> <rule> <element>BUNDLE</element> <limits> <limit> <counter>INSTRUCTION</counter> <value>COVEREDRATIO</value> <minimum>0.90</minimum> </limit> </limits> </rule> </rules> </configuration> </execution> </executions> </plugin> </plugins> </build> </project>

Notable points from the above build file around code coverage are:

• List all modules that the code lives in as dependencies for this module to get the consolidated report generated.
• Collect all compiled class files from all modules under this module's build directory. This requires maven-resource-plugin and is required for JaCoCo goal: check for coverage threshold check.
• JaCoCo plugin configuration with three executions with goals: report-aggregate, merge, and check for code coverage threshold check.
• The JaCoCo execution goal: report-aggregate is the one that gets aggregate reports generated.
• The JaCoCo goal: merge is needed to merge all modules' jacoco.exec files to be merged into one file.
• And, of course, the JaCoCo goal: check is needed for the overall code coverage threshold check and a threshold ratio number (0.90) which is different than any of the individual module's threshold ration number.

With the above maven module build files, from the root project just run: mvn clean install or mvn clean verify. It cleans, compiles code, runs all test-cases, and generates code coverage reports in each module's build directory: target/site/jacoco. Each module's coverage report shows code coverage attained from test-cases existing within that module. This number could be different (less or equal) for the same module in the overall code coverage report. It also, generates an overall aggregated code coverage report in the newly added module's build directory: my-app-code-coverage/target/site/jacoco-aggregate. The overall code-coverage generates module-wise code coverage with the overall coverage threshold level checked.

TIPS

• Have plugin configurations in the main/root project pom.xml file so that they are available to sub-modules. Only overwrite or add things that are necessary for the sub-module. For example the report goal configuration for JaCoCo in each sub-module and report-aggregate goal configuration for the overall code-coverage module.
• The main/root project can define common code coverage configurations & executions for JaCoCo, surefire and failsafe plugins for all modules with coverage threshold check value set to 0.0 in the root with sub-modules overriding that property with their specific values. That way build scripts can follow the DRY principle.
• It is good to have each module report generated in it's build taget to see the code coverage of the module by it's own test-cases though the special overall code coverage module generates reports for the overall coverage for all modules.
• If there is any module that contains code but not test-cases due to any limitations like not having needed spring application context, and boot configurations, then that module's build file (pom.xml) doesn't need JaCoCo additional configuration for goal: report.
• Another helpful maven goal is help:effective-pom. By running this command: mvn help:effective-pom for any sub-module, you can see and verify the effective pom (XML) with all parent inherited plugins and properties resolved, so that you don't need to do any guess work. This is very useful in multi-module builds to investigate any issues.

GOTCHAS

• If surefire or failsafe plugins do not run unit & integration test-cases and do not leave a clue even when run in debug mode with mvn -X option, just try adding junit dependency surefire-junit47 as described in the plugin documentation to specify the test-framework provider.
• For passing additional JVM arguments like --enable-preview or anything else, make sure to use the expression ${argLine} to retain Jacoco JVM agent argument along with your additional arguments. Missing that expression will miss code coverage and it's hard figuring out.
• With Java 16, you might run into IllegalClassFormatException if your integration test-cases hit any code that uses reflection. The test-cases pass and they get the correct code coverage. This exception in the build output can just be treated as a misleading noise and can be filtered by excluding all those classes that are involved in the reflection. For instance an exception like: java.lang.instrument.IllegalClassFormatException: Error while instrumenting com/giri/app/util/MyClassOneMethodAccess. can be filtered by adding <excludes> to the JaCoCo plugin configuration as shown below:

<build> <plugins> <!-- jacoco for code coverage --> <plugin> <groupId>org.jacoco</groupId> <artifactId>jacoco-maven-plugin</artifactId> <version>${jacoco-maven-plugin.version}</version> <configuration> <!-- Filter the misleading Exception noise by IllegalClassFormatException from JaCoCo instrumentation --> <excludes> <exclude>*MyClassOneMethodAccess*</exclude> <exclude>*MyClassTwoMethodAccess*</exclude> ... </excludes> </configuration> ...

• If code coverage threshold ratio number doesn't match the coverage report total coverage percentage number (less than the threshold number), this could be due to a silent failure in appending the integration tests coverage report to the JaCoCo generated binary coverage report file: jacoco.exec which is used for generating the HTML code coverage reports for both unit and integration tests combined. This file typically gets generated with results after running the unit tests and get appended with results of integration tests. To fix this issue, the combined binary report file can be separated and then merged as shown below. This also gives greater control on code coverage reporting.

<build> <plugins> <!-- jacoco for code coverage --> <plugin> <groupId>org.jacoco</groupId> <artifactId>jacoco-maven-plugin</artifactId> <version>${jacoco-maven-plugin.version}</version> <configuration> <excludes> <exclude>*MyClassOneMethodAccess*</exclude> <exclude>*MyClassTwoMethodAccess*</exclude> </excludes> </configuration> <executions> <!-- jacoco unit test agent for code coverage --> <execution> <id>initialize-coverage-before-unit-test-execution</id> <goals> <goal>prepare-agent</goal> </goals> <configuration> <destFile>${project.build.directory}/jacoco-unit.exec</destFile> </configuration> </execution> <!-- jacoco integration test agent for code coverage --> <execution> <id>initialize-coverage-before-integration-test-execution</id> <goals> <goal>prepare-agent</goal> </goals> <phase>pre-integration-test</phase> <configuration> <propertyName>integrationTestCoverageAgent</propertyName> <destFile>${project.build.directory}/jacoco-integration.exec</destFile> </configuration> </execution> <execution> <id>generate-merged-code-coverage-report</id> <phase>post-integration-test</phase> <goals> <goal>merge</goal> <goal>report</goal> </goals> <configuration> <!-- merge config --> <destFile>${project.build.directory}/jacoco-merged.exec</destFile> <fileSets> <fileSet> <directory>${project.build.directory}</directory> <includes> <include>*.exec</include> </includes> </fileSet> </fileSets> <!-- report config --> <dataFile>${project.build.directory}/jacoco-merged.exec</dataFile> </configuration> </execution> <!-- Threshold check --> <execution> <id>coverage-check</id> <goals> <goal>check</goal> </goals> <configuration> <dataFile>${project.build.directory}/jacoco-merged.exec</dataFile> <!-- Set Rule to fail build if code coverage is below certain threshold --> <rules> <rule implementation="org.jacoco.maven.RuleConfiguration"> <element>BUNDLE</element> <limits> <limit implementation="org.jacoco.report.check.Limit"> <counter>INSTRUCTION</counter> <value>COVEREDRATIO</value> <minimum>${jacoco.percentage.instruction}</minimum> </limit> </limits> </rule> </rules> </configuration> </execution> </executions> </plugin> ...

• Skipping unit/integration tests - both surefire and failsafe plugins offer a default pre-defined property skipTests which is false by default and when set to true skips both unit and integration tests. Unless until needed, no special configuration is needed to get a good hold on running and skipping tests. However, this is bit tricky. From the project-root/main-module run the following to control running tests of my-app-api application module.

Skip all tests:

  ./mvnw -pl my-app-api clean install -DskipTests

Run only unit tests (Skip integration tests): 

  ./mvnw -pl my-app-api surefire:test

Run specific unit test: 

  ./mvnw -pl my-app-api surefire:test -Dtest=MyUtilTest

Run specific set of unit tests, matching pattern:

  ./mvnw -pl my-app-api surefire:test -Dtest=MyU*

Run only integration tests (Skip unit tests):

  ./mvnw -pl my-app-api failsafe:integration-test

Run specific integration test:

  ./mvnw -pl my-app-api failsafe:integration-test -Dit.test=MyAppIT

Run specific set of integration tests, matching pattern:

  ./mvnw -pl my-app-api failsafe:integration-test -Dit.test=MyApp*

Summary

Maven eats up your time. You often get puzzled with many things mixed up in XML files. It's always confusingly challenging to deal with XML as specification for driving application builds.

"Making simple things super-complex" is what Software Engineering is all about. Of course, new concepts, languages, frameworks keep coming in attempts to make complex simple, but in reality only making complex more-complex. Anyways, have FUN with solving build issues/problems, and finding/inventing/re-inventing solutions in Maven & it's plugins.

References

• Jacoco report-aggregate goal
• Reporting aggregated unit and integration test coverage with jacoco
• How to fix a drop of code coverage with a multi module Maven project
• Merging Integration, Unit and functional test reports with JaCoCo
• Jacoco and Surefire argline
• Jacoco prepare-agent goal
• Surefire plugin argLine property expression
• Failsafe plugin argLine property expression
• Selecting test-framework provider
• Why Your JUnit 5 Tests Are Not Running Under Maven