Modularize Spring Boot micro-service with Spring Modulith - Notes from my exploration . . .

The dictionary meaning of Modularity is - the use of individually distinct functional units, as in assembling an electronic or mechanical system. In other words, it is the degree to which components of a system can be separated. 

In Software Development, achieving modularity involves structuring various components involved into distinct modules or packages. The separation of components of various concerns is where Software Developers often fall into the traps of architectural layers. In other words, structure code that aligns with familiar technology layers like controller, service, persistence, domain etc by grouping similar kind of classes into a package with the name that mainly describes the logical layer. Structuring code by architectural layers that gives an architectural layered overview was once a good practice, but is quite common and a norm in modern application development. With this kind of code packaging structure, code within one package or layer gets interwoven with many functional/business use-cases. This also achieves modularity, but gets driven by the technical architecture, and not driven by business domain use-cases. The top-level domain or functional visibility is lost and pushed underneath architectural layers.

Spring Modulith

Version: 1.2.5

Spring Modulith is fairly new addition to Spring Projects. It not only helps bring in well-structured application modules that can be driven by domain, but also helps in verifying their arrangement, and even facilitates creating documentation.

It comes with few key fundamental arrangement/accessibility principles/rules/conventions with some limitations. The arrangement rules are considered violated only if you have a verification test in place. Otherwise, with your own arrangement, you will be able to generate documentation like PlantUML diagram showing Modules, their dependencies and boundaries.

Some points learned and noted

• Modulith modules are analogous to Java packages.
• Without making any modular/structural changes, if you add Modulith dependency and a unit test-case to verify, it will detect and report circular references which is very useful by itself.
• By default each direct sub-package of the application main package is considered an application module, a.k.a module's base package.
• Each module's base package is treated as an API package with all public classes under a module's base package made available for access and dependency injection from other modules.
• Sub-packages under a module's base package are treated as internal to that and are not accessible by any other module, though Java allows if classes under that sub-package are public.
• In order to expose or make a sub-package under a module's base package accessible to other packages, you need to provide a package-info.java file under that sub-package by annotating that package with @NamedInetrface in that file. With this the annotated sub-package which otherwise is treated as internal to the module that is under becomes available and is accessible from other application modules.
• The file package-info.java is Java's standard way of adding package documentation which was introduced in Java 5. This file can contain Javadoc comment with Javadoc tags for the package along with package declaration and package annotations. Spring framework's null safety annotations like @NonNullFields, @NonNullApi can be specified in this file with which those specified annotations can be applied to all classes under that package.
• By default the name of a module in UML generated is nothing but the module's base package name with the uppercase first letter. This name can be customized by annotating the package with @ApplicationModule and specifying the value for displayName property. This applies to only base package of the module. Sub-packages cannot be shown in the generated UML diagram anyway.
• Additional customizations are possible. But I wouldn't overuse as it defeats simplicity.  

An example of package-info.java in a sub-package made accessible to other modules is described below:

Application main package: com.giri.myapp

Application module base package: event

Sub-package of module events exposed to other modules: publishers

The file package-info.java under com.giri.myapp.event.publisher looks like below: 

package-info.java

@org.springframework.modulith.NamedInterface package com.giri.myapp.event.publisher;

An example of package-info.java added to a module (base package) with different module name (EventConsumers) than the default name (Consumer) is shown below:

@org.springframework.modulith.ApplicationModule(displayName = "EventConsumers") package com.giri.myapp.consumer;

Limitations

• Sub-package exposed as package cannot be shown in generated diagram.

Documentation can be generated by having a simple test-case as outlined in the documentation. UML diagram is particularly useful to get to see Architectural overview of the application modules, their dependencies and boundaries. However, if you organized any of the related components into sub-packages of a module base package and exposed those by annotating the package by adding package-info.java as shown above, it's quite natural to expect that the sub-package is shown in the generated UML as a module since it's treated as a module from exposure point of view for other modules. But the UML diagram seems only restricted and limited to showing the default application modules, the direct sub-packages under the project main package.

I did explore the API little bit to find out if there is a way to override this rule by any means, I couldn't find a way to do so. Hope the future version will consider this kind of expectation and provide an option for specifying for documentation as the sub-package exposed as a module by annotating the package with @NamedInterface is visible and used/depended on by other modules.

• Generated visible diagram files should be treated as code to be checked in.
  

The clickable modules and visual diagram generated as .puml files are only useful to developers with IDE plugins. A mechanism to integrate into code documentation files like README.md or wiki would be more useful to let the Visible Architecture up-to-date with the codebase.

 

TIP

IntelliJ IDEA has a PlantUML Integration plugin available and can be used to view generated .puml files as UML diagram in the IDE. Follow these steps in order to generate Spring Modulith modules diagram and view the UML diagram in IntelliJ IDEA.

• Make sure you have graphviz installed.
• Install IntelliJ IDEA's PlantUML Integration plugin
• Run test-case: ModularityTest which generates modulith PlantUML files (.puml) under application's target/spring-modulith-docs directory.
• When you open any .puml file generated in IntelliJ, the plugin shows it as PlantUML diagram.

A sample JUnit test-case (ModularityTest.java) is shown below:

package com.giri.myapp.modularity; import com.giri.myapp.MyApplication; import org.junit.jupiter.api.Test; import org.springframework.modulith.core.ApplicationModules; import org.springframework.modulith.docs.Documenter; class ModularityTest { ApplicationModules modules = ApplicationModules.of(MyApplication.class); /** * Test to verify application structure. Rejects cyclic dependencies and access to internal types. */ @Test void verifyModularity() { System.out.println(modules); modules.verify(); } /** * Test to generate Application Module Component diagrams under target/spring-modulith-docs. */ @SuppressWarnings("squid:S2699") @Test void writeDocumentationSnippets() { new Documenter(modules) .writeModulesAsPlantUml() .writeIndividualModulesAsPlantUml(); } }

Summary

Structuring by domain use-cases vs architectural layers cannot be a personal preference. Viewing an application from domain aspect with underneath familiar technological layers gives a view that aligns better with business than viewing an application from technological layers. Spring Modulith helps achieve domain-driven modularity by following simple package level modular conventions where modules align with domain concepts.

Achieving better modularity doesn't necessarily need to be showing only business domain-based modules. Certain aspects of technologies like GraphQL let's say to indicate that the application provides GraphQL API for its client can as well get depicted in the generated UML module diagram by structuring controllers into a module with name graphql. With a right balanced mix of modularity, code can be restructured to show all main business/domain use-cases along with some technology related modules like Rest, GraphQL, Events etc. added to the mix. This kind of balanced approach gives good architectural and structural overview of the application showing how modules interact with each other, in some cases event showing high-level flow.

Spring Modulith comes with added support for structural validation, visual documentation of the modular arrangement, modular testability and observability.

Domain-driven modularity brings in more maintainable and understandable structure. However, keep things simple and do not overuse modularity, it defeats simplicity that Software Development is badly in need of. ;)

References

• Spring Modulith GitHub Repo
• Reference Documentation
• API - Javadoc
• Java - package comment file
• The package-info.java File

Spring Boot - Check database connectivity after the application starts up . . .

Database Integration is much simpler with Spring Boot's non-invasive Auto Configuration feature. A typical Spring Boot application is configured to run in multiple environments, a.k.a profiles. However, there are multiple options available when it comes to configuring Database, like Docker Compose, Testcontainers, explicit DataSource profile based properties/yaml, externalized DataSource properties through Vault etc. In any case, it is good to have a database connection check in place to make sure that the database connection looks good once the application boots up and starts to run.

Environment: Java 21, Spring Boot 3.2.1, PostgreSQL 16, maven 3.9.6 on macOS Catalina 10.15.7

The Scenario

The Database is PostgreSQL and we want to run a simple query to make sure that the database connection looks good once the application starts up.

One way to achieve this

One way to achieve this is to execute a simple query after the application starts up. Spring Boot's CommandLineRunner or ApplicationRunner can be leveraged to do this. This is a good place to run specific code after the application has started.

Here is a code snippet for this:

import lombok.extern.slf4j.Slf4j; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.boot.CommandLineRunner; import org.springframework.boot.SpringApplication; import org.springframework.boot.autoconfigure.SpringBootApplication; import org.springframework.context.annotation.Bean; import org.springframework.jdbc.core.simple.JdbcClient; @SpringBootApplication @Slf4j public class MyApplication { public static void main(String[] args) { SpringApplication.run(MyApplication.class, args); } @Autowired(required = false) JdbcClient jdbcClient; @Bean public CommandLineRunner commandLineRunner() { return args -> { if (jdbcClient != null) { log.info("Database check: {}", jdbcClient.sql("SELECT version()").query(String.class).single()); } }; } }

The above highlighted is the code snippet that gets executed after the application gets started. It just logs the executed query result, nothing but the database version. 

An integration test case can also be put in place as shown below, which makes sure that the database connection and version look good. This kind of testcase is good to have to make sure that the code is tested against the same db version as the production.

import org.junit.jupiter.api.Test; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.boot.test.autoconfigure.jdbc.AutoConfigureTestDatabase; import org.springframework.boot.test.autoconfigure.orm.jpa.DataJpaTest; import org.springframework.context.annotation.Import; import org.springframework.jdbc.core.simple.JdbcClient; import org.springframework.test.context.ActiveProfiles; import static org.assertj.core.api.Assertions.*; /** * An integration test to check Database connectivity. */ @ActiveProfiles("test") // We don't want the H2 in-memory database. // We will provide a custom 'test container' as DataSource, so don't replace it. @AutoConfigureTestDatabase(replace = AutoConfigureTestDatabase.Replace.NONE) @DataJpaTest @Import(TestContainersConfiguration.class) public class DatabaseCheckIT { @Autowired JdbcClient jdbcClient; @Test void database_connection_works_and_version_looks_good() { assertThat(jdbcClient.sql("SELECT version()").query(String.class).single()) .contains("16.0"); } }

The above test case uses Testcontainers and a test configuration as shown below for unit/integration tests:

import lombok.extern.slf4j.Slf4j; import org.springframework.boot.test.context.TestConfiguration; import org.springframework.boot.testcontainers.service.connection.ServiceConnection; import org.springframework.context.annotation.Bean; import org.testcontainers.containers.PostgreSQLContainer; /** * Test Configuration for testcontainers. */ @TestConfiguration(proxyBeanMethods = false) @Slf4j public class TestContainersConfiguration { private static final String POSTGRES_IMAGE_TAG = "postgres:16.0"; @Bean @ServiceConnection PostgreSQLContainer postgreSQLContainer() { return new PostgreSQLContainer<>(POSTGRES_IMAGE_TAG) .withDatabaseName("my-application") .withUsername("my-application") .withPassword("s3cr3t") .withReuse(true); } }

⚠ Gotcha

Note that in the main application class, for JdbcClient @Autowired annotation, the optional property required is explicitly set to false.  The reason for this is if there are any integration test cases to test specific layers (test slices like @GraphQlTest) that do not auto configure datasource, when the application class is run as part of starting Spring Boot run, the testcase runs into exception as JdbcClient bean is not available for auto wiring. So, in those cases jdbcClient property would be null. So, a non null check is required to safely run the SQL statement.

💡 TIPS

The CommandLineRunner bean in the application class can conditionally be defined on some DataSource related bean/class by annotation it with @ConditionalOnBean or @ConditionalOnClass. I couldn't find a way to get it conditionally defined and be working for all scenarios.

Resources

• CommadLineRunner
• Spring Boot support for Testcontainers
• Conditional Beans

Spot-check Spotless . . .

Sometimes, overwhelming details are underwhelming. I happened to quickly explore Spotless for code formatting and checks. Spotless is the plugin to spot check. It's good that the README of Spotless upfront has separated Gradle and Maven documentation. I was looking to add maven support for a new Java Spring Boot 3.0 GraphQL project. But the Spotless Maven README documentation was overwhelming.

Quickly read through for few minutes to get an essence of it. Though there are so many details, what I wanted to get done seemed simple. Of course, practicality is always different. That's where experience is gained or comes from ;)

Environment: Java 20, Spring Boot 3.2.0, maven 3.9.6 on macOS Catalina 10.15.7

I quickly applied it to the project, and immediately ran into a couple of Gotchas. It took me couple of hours to explore and set it up finding issues and fixing settings. This post is just about those issues that I ran into.

Use Spaces, not Tabs in code

I prefer SPACES to TABS in code. It is good practice to use SPACE instead of TAB in code formatting. This cannot be a personal preference. But most of IDEs still come with TAB as default setting and many developers don't even pay attention to it. In code reviews, the formatting goes off due to SPACESs vs. TABs and is always annoying.

I use IntelliJ IDEA for my development. Whenever I install IntelliJ, the very first thing I would setup is to use spaces instead of tabs (Go to Preferences > Editor > Code Style > Java, and uncheck Use tab character). With that all code that I write will only use spaces and not tabs. The next thing is to setup to show whitespaces (Go to Preferences > Editor > Appearance, and check Show whitespaces), to easily distinguish Tabs and Spaces by this setting on.

Gotcha-1

The Spotless maven plugin documentation describes many details. The setting I wanted for code check was found at a couple of places in there under <indent> tag. At only one place the setting <spaces>true</spaces> is specified. Though I tried few different things like false for <tabs>, and true for <spaces> etc., the spotless:check maven goal was still suggesting to change code with spaces to tabs.

We also have to use google code style settings: eclipse-formatter.xml, that I took from the other project to use. This is described under Java > eclipse jdt in the spotless documentation. It's a google code style Java settings guide in xml format ;)

The setting id that is used for what I wanted was like: <setting id="org.eclipse.jdt.core.formatter.tabulation.char" value="tab"/>. Changing that "tab" to "space" made spotless happy with spaces in code. ;)

Gotcha-2

Binding to maven phase : I should have read it carefully. At the minimum the <executions><execution><goal>check</goal></execution></execution> is required. Without this ./mvnw spotless:check works, but ./mvnw clean install which also runs verify will not run spotless check. So, for spotless check goal to be bound to maven verify goal, the above setting is required.

A sample spotless settings looks like this:

... <plugins> <!-- spotless https://github.com/diffplug/spotless --> <plugin> <groupId>com.diffplug.spotless</groupId> <artifactId>spotless-maven-plugin</artifactId> <version>${spotless-plugin.version}</version> <executions> <execution> <goals> <goal>check</goal> </goals> </execution> </executions> <configuration> <java> <endWithNewline /> <trimTrailingWhitespace /> <cleanthat> <version>2.18</version> <mutators> <mutator>AvoidInlineConditionals</mutator> <mutator>LiteralsFirstInComparisons</mutator> <mutator>UnnecessaryImport</mutator> <mutator>UnnecessaryModifier</mutator> <mutator>UseUnderscoresInNumericLiterals</mutator> <mutator>UseDiamondOperator</mutator> </mutators> </cleanthat> <eclipse> <version>4.26</version> <file>${project.basedir}/eclipse-formatter.xml</file> </eclipse> </java> </configuration> </plugin> ... </plugins>

💡 TIPS

Spotless handy maven goals

Check: ./mvnw spotless:check

Apply: ./mvnw spotless:apply

Conclusion

Any new exploration in Maven world never goes smooth for me. There are always bumps along the way. I do not like copy and paste at all. After all, if everything works in the first place when quick-copy-paste-tool is used, there is not much left to learn ;)  With the Generative AI getting trained to write code, learning will soon become a rare thing!

References

• Spotless
• Spotless Maven Plugin
• Binding Spotless goal to Maven Phase
• Eclipse jdt
• Google style guides
• Cleanthat Mutators 

Docker maven plugin - Spring Boot, Redis : Gotcha

The docker-maven-plugin comes in handy for managing Docker images and containers in integration tests. It can be used to build images or run. Multiple images can be configured to be built or run depending on the need of your application. This post focuses on run aspect of this plugin; specifically running Redis image in Docker container and a potential issue that one might run into.

Environment: Java 20, Spring Boot 3.1.5, maven 3.8.6 on macOS Catalina 10.15.7

The Scenario

After going through some painful hoops, I upgraded a Spring Boot application from 2.6.3 to 3.1.5 with the two-step recommended approach 2.6.x to 2.7.x and then to 3.1.x. The application uses Redis for caching needs. All worked well at the end. The app was built through concourse CI/CD pipeline and successfully deployed as Kubernetes workload to int and cert environments and has been successfully running for few weeks.

I started to work on fixing some critical and high Snyk reported open-source dependency vulnerabilities. Suddenly, integration tests around Redis started to fail with the following exception:

[ERROR] com.hmhco.api.assessmentservice.service.AssessmentServiceCacheIT.testCache -- Time elapsed: 0.478 s <<< ERROR! org.springframework.data.redis.RedisConnectionFailureException: Unable to connect to Redis at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory$ExceptionTranslatingConnectionProvider.translateException(LettuceConnectionFactory.java:1604) at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory$ExceptionTranslatingConnectionProvider.getConnection(LettuceConnectionFactory.java:1535) at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory$SharedConnection.getNativeConnection(LettuceConnectionFactory.java:1360) at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory$SharedConnection.getConnection(LettuceConnectionFactory.java:1343) at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory.getSharedConnection(LettuceConnectionFactory.java:1061) at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory.getConnection(LettuceConnectionFactory.java:400) at org.springframework.data.redis.cache.DefaultRedisCacheWriter.execute(DefaultRedisCacheWriter.java:272) at org.springframework.data.redis.cache.DefaultRedisCacheWriter.clean(DefaultRedisCacheWriter.java:189) at org.springframework.data.redis.cache.RedisCache.clear(RedisCache.java:220)

The above exception stack-trace didn't give much clue. After looking into few things including redis test configurations, dependencies including transitive dependencies etc. it was still puzzling as it was actually working few weeks ago, and all of sudden only locally the integration test-cases around redis started to fail.

The docker-maven-plugin actually logs that redis:latest started, before running integration tests like below:

... [INFO] DOCKER> [redis:latest] "redis-test": Start container 395afa51913b ...

The last thing I looked at was the docker containers:

$ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 395afa51913b redis "docker-entrypoint.s…" About a minute ago Exited (1) About a minute ago redis-1 b03c67ae628e postgres:12.4 "docker-entrypoint.s…" About a minute ago Up About a minute 0.0.0.0:5433->5432/tcp postgres-1

That showed like the container started but exited for some reason. I also looked at the for Docker Dashboard for any additional details.

Clicking redis-1 container that exited shows logs and the reason: # Fatal: Can't initialize Background Jobs. Error message: Operation not permitted

Also gives Redis version detail: 7.2.3

That's the issue. Redis started but exited and hence the test failed with the exception - Unable to connect to Redis 

After googling about that initialization issue (# Fatal: Can't initialize Background Jobs. Error message: Operation not permitted), people recommended to try few specific Redis versions. After some trail and error, I had luck with version 6.2.6.

The logs can also be fetched from command-line by executing for the CONTAINER ID (395afa51913b ) or the NAME (redis-1):

$ docker logs 395afa51913b

The Fix

In the docker-maven-plugin configuration, the image tag can be specified for specific version 6.2.6 as shown below:

<properties> <redis.test.port>6381</redis.test.port> </properties> ... <plugin> <groupId>io.fabric8</groupId> <artifactId>docker-maven-plugin</artifactId> <version>0.43.4</version> <executions> <execution> <id>start</id> <phase>pre-integration-test</phase> <goals> <goal>start</goal> </goals> </execution> <execution> <id>stop</id> <phase>post-integration-test</phase> <goals> <goal>stop</goal> </goals> </execution> </executions> <configuration> <images> <image> <build> <cleanup>true</cleanup> <tags> <tag>latest</tag> </tags> </build> <external> <type>properties</type> <prefix>postgres.docker</prefix> </external> </image> <image> <name>redis:6.2.6</name> <alias>redis-test</alias> <run> <ports> <port>${redis.test.port}:6379</port> </ports> </run> </image> </images> </configuration> </plugin>

TIP

Testcontainers is another choice to look into for integration tests.

GOTCHA

It's Gotcha in a Gotcha post ;) I always run into these kinds of issues whenever I explore anything anytime in maven world.

The docker-maven-plugin docker goals start and stop are bound to maven failsafe plugin pre-integration-test and post-integration-test phases of integration-test goal. But if you skip tests by passing argument -DskipTests which actually would skip running all unit and integration tests, I expected docker images not to be run. For instance ./mvnw clean install -DskipTests command to compile, build, package and install all modules by skipping tests. However, in this case, the images do get run: started and stopped. I couldn't find a way to get a hold on this, could be a bug or issue in the plugin by design, not sure.

That was happening right after spring-boot-maven-plugin's package goal. That plugin also gets bound to pre-integration-test and post-integration-test phases when specified. We don't have these executions specified for this plugin though.

Couldn't figure out a workaround for not getting docker images run when integration test cases are skipped. If anyone has a solution for this, please post a comment, I would appreciate it.

The word trivial in software development is actually complex. Things are unnecessarily made super-complex. Software developers get paid for dealing with it anyways ;)

References

• Docker Maven Plugin - Github Repo
• Docker Maven Plugin - Documentation
• Docker Hub - Redis Images
• Maven Failsafe skipping tests
• Spring Boot Maven Plugin

Java bytecode - compiler version options and compatibilities . . .

One of many strengths of Java platform is its backward compatibility with the language. As language keeps evolving and moving forward, the good old syntax is still supported for backward compatibility. However, the compiler adds certain indicative options for specifying version details. The --source, --target are two such compiler (javac) options. From Java 9 onwards a third option --release got added to this mix. Getting a good understanding of these options is not trivial without actually experiencing all three. When compiling source code of a single class you may not need to specify these options. But in Java project when building with maven like build system, one needs to understand these options and their implications.

Environment: Java 20, Spring Boot 2.7.15, maven 3.9.3 on macOS Catalina 10.15.7

The maven-compiler-plugin

Maven build system uses maven-compiler-plugin for compiling source code. This plugin documentation upfront talks about source and target options and highly recommends to change these in plugin configuration. In order to change these per application/module needs, one needs to look under the hood for understanding.

Various extra Java compiler options can be specified in the maven-compiler-plugin configuration. The actual Java compiler options related to version are: --source, --target and --release that can be specified and passed to the compiler during code compilation through maven-compiler-plugin configuration. This can be done in two different ways in pom.xml:

1. Through maven properties: maven.compiler.source,  maven.compiler.target and maven.compiler.release as highlighted below:

<properties> <maven.compiler.source>20</maven.compiler.source> <maven.compiler.target>20</maven.compiler.target> <maven.compiler.target>20</maven.compiler.target> </properties>

If these properties are not explicitly defined, maven compiler plugin uses 1.8 for source and target.

2. Through the plugin configuration settings as highlighted below. Note - For convenience defined extra properties and used for source, target and release but straight version numbers can be used.

... <properties> <java.version>20</java.version> <javac.source.version>${java.version}</javac.source.version> <javac.target.version>${java.version}</javac.target.version> <javac.release.version>${java.version}</javac.release.version> <!-- Maven plugins --> <maven-compiler-plugin.version>3.11.0</maven-compiler-plugin.version> </properties> <build> <pluginManagement> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-compiler-plugin</artifactId> <version>${maven-compiler-plugin.version}</version> <configuration> <source>${javac.source.version}</source> <target>${javac.target.version}</target> <release>${javac.release.version}</release> <compilerArgs> <arg>-Xlint:all</arg> </compilerArgs> </configuration> </plugin> </plugins> </pluginManagement> ...

If no special configuration is required, it doesn't require even to specify maven-compiler-plugin. If specified, the above are two ways to control/change default 1.8 set by the plugin for these options which eventually get passed to the Java compiler (javac) during code compilation of sources (both under src and test. 

Note from Java 9 onwards, the values to these options are not like 1.7, 1.8 but must be 7 and 8.

Java 20

Java 20 compiler doesn't support version 7 for source, target anymore. The supported releases are 8 through 20. So, for any reason if maven compiler plugin is set explicitly with 1.7, build fails with ERRORS saying: Source option 7 is no longer supported. Use 8 or later. , and Target option 7 is no longer supported. Use 8 or later. 

Now it's time to understand what these options actually tell the compiler, javac. The compiler's help option (javac -help) lists all available options and a brief description about each option. The -source, -target, -release options descriptions are helpful to some extent.

--source <release>, -source <release> Provide source compatibility with the specified Java SE release. Supported releases: 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20 --target <release>, -target <release> Generate class files suitable for the specified Java SE release. Supported releases: 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20 --release <release> Compile for the specified Java SE release. Supported releases: 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20

To understand these compiler options better, we can compile a simple Java application class with just main method.

HelloJava.java

import java.util.Properties; public class HelloJava { public static void main(String[] args) { Properties systemProperties = System.getProperties(); System.out.println(String.format("Hello Java %s!", systemProperties.getProperty("java.vm.specification.version"))); systemProperties.entrySet().stream() .filter(entry -> entry.getKey().toString().startsWith("java")) .toList().stream() .forEach(entry -> System.out.println(entry.getKey() + "=" + systemProperties.getProperty(entry.getKey().toString())) ); } }

Note - The above class prints system properties that start with "java" with their values. It uses toList() method that Java 16 added to Stream class. With this the expectation is- the code should not be compiled for Java/JVM version less than 16.

Java compiler version options

Let's compile the class with different Java versions and compiler options.

 

// compile with Java 20: no options specified $ sdk use java 20.0.2-amzn // check: major version $ javap -verbose HelloJava.class | grep major major version: 64 // run on Java 20: works $ javac HelloJava.java "Hello Java 20!" // switch to Java 17 and run: fails with LinkageError $ sdk use java 17.0.1.12.1-amzn $ java HelloJava Error: LinkageError occurred while loading main class HelloJava java.lang.UnsupportedClassVersionError: HelloJava has been compiled by a more recent version of the Java Runtime (class file version 64.0), this version of the Java Runtime only recognizes class file versions up to 61.0 // switch to Java 15 and run: fails with LinkageError $ sdk use java 15.0.2.7.1-amzn $ java HelloJava Error: LinkageError occurred while loading main class HelloJava java.lang.UnsupportedClassVersionError: HelloJava has been compiled by a more recent version of the Java Runtime (class file version 64.0), this version of the Java Runtime only recognizes class file versions up to 59.0 // switch to Java 8 and run: fails with UnsupportedClassVersionError Exception $ sdk use java 8.0.352-amzn $ java HelloJava Error: A JNI error has occurred, please check your installation and try again Exception in thread "main" java.lang.UnsupportedClassVersionError: HelloJava has been compiled by a more recent version of the Java Runtime (class file version 64.0), this version of the Java Runtime only recognizes class file versions up to 52.0 at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:756) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:473) at java.net.URLClassLoader.access$100(URLClassLoader.java:74) at java.net.URLClassLoader$1.run(URLClassLoader.java:369) at java.net.URLClassLoader$1.run(URLClassLoader.java:363) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:362) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:601)

So. when compiled with a specific version of java compiler (in this case Java 20 and no version options are specified), it gets compiled with default target of the Java compiler which is 20. The class generated cannot be run on prior JVM versions (prior to 20). To find the target of JVM code the byte-code is generated for, use javap -verbose HelloJava.class | grep major.

Now, let's try compiling for target 17 and try to run on different JVM versions.

// compile with Java 20 for target 17: --source and --target options specified $ sdk use java 20.0.2-amzn $ javac --source=17 --target=17 HelloJava.java warning: [options] system modules path not set in conjunction with -source 17 1 warning // check: major version $ javap -verbose HelloJava.class | grep major major version: 61 // run on Java 20: works $ java HelloJava Hello Java 20! // switch to Java 17 and run: works $ sdk use java 17.0.1.12.1-amzn // run on Java 17: works $ java HelloJava Hello Java 17! // swicth to Java 15 and run: fails with LinkageError $ sdk use java 15.0.2.7.1-amzn $ java HelloJava Error: LinkageError occurred while loading main class HelloJava java.lang.UnsupportedClassVersionError: HelloJava has been compiled by a more recent version of the Java Runtime (class file version 61.0), this version of the Java Runtime only recognizes class file versions up to 59.0 // compile with Java 20 for target 17: --source and --target options specified $ sdk use java 20.0.2-amzn $ javac --source=15 --target=15 HelloJava.java warning: [options] system modules path not set in conjunction with -source 15 1 warning // check: major version $ javap -verbose HelloJava.class | grep major major version: 59 // switch to Java 17 and run: works $ sdk use java 17.0.1.12.1-amzn $ java HelloJava Hello Java 17! // swicth to Java 15 and run: fails with NoSuchMethodError $ sdk use java 15.0.2.7.1-amzn $ java HelloJava Hello Java 15! Exception in thread "main" java.lang.NoSuchMethodError: 'java.util.List java.util.stream.Stream.toList()' at HelloJava.main(HelloJava.java:15) // switch to Java 8 and run: fais with UnsupportedClassVersionError $ sdk use java 8.0.352-amzn $ java HelloJava Error: A JNI error has occurred, please check your installation and try again Exception in thread "main" java.lang.UnsupportedClassVersionError: HelloJava has been compiled by a more recent version of the Java Runtime (class file version 59.0), this version of the Java Runtime only recognizes class file versions up to 52.0 at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:756) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:473) at java.net.URLClassLoader.access$100(URLClassLoader.java:74) at java.net.URLClassLoader$1.run(URLClassLoader.java:369) at java.net.URLClassLoader$1.run(URLClassLoader.java:363) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:362) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:601) // compile with Java 20 for target 8: --source and --target options specified $ sdk use java 20.0.2-amzn $ javac --source=8 --target=8 HelloJava.java javac --source=8 --target=8 HelloJava.java warning: [options] bootstrap class path not set in conjunction with -source 8 warning: [options] source value 8 is obsolete and will be removed in a future release warning: [options] target value 8 is obsolete and will be removed in a future release warning: [options] To suppress warnings about obsolete options, use -Xlint:-options. 4 warnings // check: major version $ javap -verbose HelloJava.class | grep major major version: 52 // switch to Java 20 and run: works $ sdk use java 20.0.2-amzn $ java HelloJava Hello Java 20! // switch to Java 17 and run: works $ sdk use java 17.0.1.12.1-amzn $ java HelloJava Hello Java 17! // swicth to Java 15 and run: fails with NoSuchMethodError $ sdk use java 15.0.2.7.1-amzn $ java HelloJava Hello Java 15! Exception in thread "main" java.lang.NoSuchMethodError: 'java.util.List java.util.stream.Stream.toList()' at HelloJava.main(HelloJava.java:15)

So, when compiled for a target version, it cannot be run on prior JVM versions.

Let's try target 7.

$ sdk use java 20.0.2-amzn $ javac --source=7 --target=7 HelloJava.java warning: [options] bootstrap class path not set in conjunction with -source 7 error: Source option 7 is no longer supported. Use 8 or later. error: Target option 7 is no longer supported. Use 8 or later.

Java version 7 is not supported anymore.

Let's experience --release option.

// compile with Java 20 using options --source and --target options specified $ sdk use java 20.0.2-amzn $ javac --source=17 --target=17 --release=17 HelloJava.java error: option --source cannot be used together with --release error: option --target cannot be used together with --release Usage: javac <options> <source files> use --help for a list of possible options // compile with Java 20 for target 17: --release options specified $ sdk use java 20.0.2-amzn $ javac --release=17 HelloJava.java // check: major version $ javap -verbose HelloJava.class | grep major major version: 61 // switch to Java 20 and run: works $ sdk use java 20.0.2-amzn $ java HelloJava Hello Java 20! // switch to Java 17 and run: works $ sdk use java 17.0.1.12.1-amzn $ java HelloJava Hello Java 17! // switch to Java 15 and run: fails with NoSuchMethodError $ sdk use java 15.0.2.7.1-amzn $ java HelloJava Hello Java Exception in thread "main" java.lang.NoSuchMethodError: 'java.util.List java.util.stream.Stream.toList()' at HelloJava.main(HelloJava.java:14) // compile with Java 20 for target 15: --release options specified $ sdk use java 20.0.2-amzn $ javac --release=15 HelloJava.java HelloJava.java:14: error: cannot find symbol .toList().stream() ^ symbol: method toList() location: interface Stream<Entry<Object,Object>> 1 error // compile with Java 20 for target 15: --source --target options specified $ sdk use java 20.0.2-amzn $ javac --source=15 --target=15 HelloJava.java warning: [options] system modules path not set in conjunction with -source 15 1 warning // switch to Java 15 and run: fails with NoSuchMethodError $ sdk use java 15.0.2.7.1-amzn $ java HelloJava Exception in thread "main" java.lang.NoSuchMethodError: 'java.util.List java.util.stream.Stream.toList()' at HelloJava.main(HelloJava.java:14)

So, --release option does a strict compilation time checks to see if the code is compliant with the release target and fails to compile if a method not supported in target version is used in the code. This makes sure the compiled class works on target release (the target JVM version that the code is released to run on). Whereas, the --target option doesn't do code compliance checks during compilation time, it simply compiles code but fails during runtime. So, --release seems like better option to leverage when specifying.

Implications of version options

• No compiler options specified: The code gets compiled with default target as the version of the Java compiler.
• All 3 options  --source, --target and --release specified:  Not allowed.
• Options --source, --target specified: 1) Both can be same (e.g. 17, 17). 2) The option: --source can be lower version (e.g. 15) and --target can be higher version (e.g. 17), but not the other way. If --source is higher version (e.g.17) and --target is lower version (e.g.15), compiler fails with a warning: warning: source release 17 requires target release 17, it doesn't get compiled.
• Only option --source: The option --source can be any version but default target would be the version of Java compiler being used.
• Only option --target: The option: --target cannot be lower than the version of Java compiler being used because default source would be the version of the compiler being used. A lower target version gets into compiler option source higher, target lower and fails compilation. E.g javac --target=19 HelloJava.java with Java 20 compiler fails with warning: target release 19 conflicts with default source release 20 and code doesn't get compiled.
•  Only option --release: Strict code check during compilation to make sure that compiled code gets compiled and works on the target version. Also, the byte-code generated runs only on the release specified and higher, but doesn't run on any lower versions.
• Compiling using Java 20, and no --release option or --release=20 results with major version 64 (Java 20).
• Compiling using Java 20 with --release=19 results with major version 63 (Java 19).
• Compiling using Java 20 with --release=17 results with major version 61 (Java 17) and would result with LinkageError when run on lower version other than 17. Works on 17 and higher.

The maven-compiler-plugin variations with these options

Maven, out of the box with no maven-compiler-plugin specified in pom.xml, has the following variations with it's compiler version option properties (maven.compiler.source, maven.compiler.target and maven.compiler.release).

<properties> <maven.compiler.source>20</maven.compiler.source> <maven.compiler.target>20</maven.compiler.target> <maven.compiler.release>20</maven.compiler.release> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> </properties>

• No version properties are specified: (No maven.compiler.source, maven.compiler.target and maven.compiler.release properties): Build fails with compilation ERRORS: Source option 5 is no longer supported. Use 8 or later. and Target option 5 is no longer supported. Use 8 or later.
• All 3 version properties are specified: The option: maven.compiler.release is ignored, it can be any junk. Only source and target properties matter. The value for target option cannot be less than the source. For instance, source 20, target 19 fails build with warning: source release 20 requires target release 20.
• Only source version property is specified: When only source is specified, target must also be specified. Otherwise, maven build fails with: Fatal error compiling: warning: source release 20 requires target release 20
• Only target version property is specified: When only target is specified, source must also be specified. Otherwise, maven build fails with: Source option 5 is no longer supported. Use 8 or later.
• The release property specified: When release is specified and is 8 or later, this takes the precedence. Make a special NOTE of it. When release is specified, it takes the precedence and source, target options are ignored. In this case, any non-sense value will make the build work and code gets compiled for the release version specified. But this must be 8 or later.

With maven-compiler-plugin specified in pom.xml, has the following variations with these options specified in the plugin configuration (<configuration>): 

<properties> <maven.compiler.source>20</maven.compiler.source> <maven.compiler.target>20</maven.compiler.target> <maven.compiler.release>20</maven.compiler.release> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> </properties> ... <build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-compiler-plugin</artifactId> <version>3.11.0</version> <configuration> <source>20</source> <target>20</target> <release>20</release> </configuration> </plugin> </plugins> </build>

With the above way of having both defined set of properties, and maven-compiler-plugin configuration, the configuration values override the property values defined. If no configuration is specified for the maven-compiler-plugin, it uses the set of properties defined. With configuration <source>, <target>, and <release> taking the precedence, the variations are as follows:

• No <configuration> specified for the plugin, but set of properties are defined: It uses defined set of properties if exist, release takes the precedence over target and source.
• No properties are defined, and no <configuration> is specified for the plugin : It defaults to target 1.8.
• All 3 configuration options are specified: The release configuration takes the precedence and is built for the release version.
• No properties are set, and only source configuration option is specified: When only source is specified, target must also be specified. Otherwise, maven build fails with: Fatal error compiling: warning: source release 20 requires target release 20
• No properties are set, and only target configuration option is specified: Code gets compiled for the target specified.
• No properties are set, and both target and release configuration options are specified: The release option takes the precedence and code gets compiled for the release specified.

Summary

For Java 9 and after, use --release option.

For older versions prior to Java 9 use --source and --target options.

TIPS

• Use SDKMAN to install multiple Java versions and easily switch between different versions.
• If You see noisy warning: Using deprecated '-debug' fallback for parameter name resolution. Compile the affected code with '-parameters' instead or avoid its introspection:, then add compiler argument <arg>-parameters</arg> to the maven-compiler-plugin configuration as shown below:

<plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-compiler-plugin</artifactId> <version>${maven-compiler-plugin.version}</version> <configuration> <source>${javac.source.version}</source> <target>${javac.target.version}</target> <release>${javac.release.version}</release> <compilerArgs> <arg>-Xlint:all</arg> <arg>-parameters</arg> </compilerArgs> </configuration> </plugin>

References

• Maven Compiler Plugin
• Maven - setting compiler source target versions
• Maven - setting compiler release target version
• Java bytecode versions
• The --release option
• What Is the –release Option in the Java 9 Compiler?
• Java class File Format
• Java 20 Home

Spring Boot - WebTestClient Gotcha . . .

The Scenario

Recently I had a scenario to write an integration test in a Spring Boot micro-service (say MyService) which hosts it's own data in it's own PostgreSQL database in order to verify it's data against another Spring Boot micro-service which hosted master data in it's own database. The database contains a specific entity data set (say Item), a fixed set of items, used for a specific purpose. Another micro-service (say OtherService) uses one of the end-points of MyService by passing entity ids (Item ids) for some processing that MyService is capable of. The OtherService hosts master data of those Entities (Items) in its own database. But, the data hosted for each entity by both services is entirely different, and each has its own business with it's own data set. Only ids are common. OtherService data is considered master data. So, a MyService integration test requires to make sure that there are no entities missing in it's database. OtherService, offers an end-point to get it's hosted Item entities.

Environment: Java 17, Spring Boot 3.0.2, maven 3.8.5 on macOS Catalina 10.15.7

So, WebTestClient seemed like better choice to leverage to make an API request and get the master list of Items and check it's database to see if the count of Items and list of Item ids match.

In MyService, there was an integration test already in place written to check most of it's lookup like database entities; one test method for testing each entity's data set. This was a natural integration test to extend by adding another integration test method for testing Item entity set. But this one goes beyond it's database, out to OtherService, making an API request to to get the master list of Items to verify against.

The Issue

Simply Auto-wiring WebTestClient like:

@Autowired privateWebTestClient webTestClient;

broke the auto-configuration by the following exception:

org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'com.hmhco.sgm.scoring.api.persistence.LookupRepositoriesDataIT': Unsatisfied dependency expressed through field 'webTestClient'; nested exception is org.springframework.beans.factory.NoSuchBeanDefinitionException: No qualifying bean of type 'org.springframework.test.web.reactive.server.WebTestClient' available: expected at least 1 bean which qualifies as autowire candidate. Dependency annotations: {@org.springframework.beans.factory.annotation.Autowired(required=true)} Caused by: org.springframework.beans.factory.NoSuchBeanDefinitionException: No qualifying bean of type 'org.springframework.test.web.reactive.server.WebTestClient' available: expected at least 1 bean which qualifies as autowire candidate. Dependency annotations: {@org.springframework.beans.factory.annotation.Autowired(required=true)}

The Solution

A quick Googling suggested to try the following, which actually worked (NOTE: WebTestClient requires org.springframework.boot:spring-boot-starter-webflux dependency which we already had in MyService app pom.xml:

@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)

But this solution brings up the embedded web server (Tomcat) starting it at a random port which is actually unnecessary for this test. All I need is an instance of WebTestClient to make a HTTP Get request to OtherService end-point.

Here is the code snippet for this solution:

@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT) @ActiveProfiles("test") @RunWith(SpringRunner.class) public class MyServiceDataIT { @Value("${spring.other-service-api.get-Items-end-point}") private String apiEndPoint; @Value("${spring.other-service-api.trusted_token}") private String trustedToken; @Autowired private WebTestClient webTestClient; ... @Test public void itemLookup_has_no_missing_items() throws Exception { // given: request spec var requestSpec = webTestClient.get() .uri(apiEndPoint) .header(HttpHeaders.ACCEPT, MediaType.APPLICATION_JSON_VALUE) .header(HttpHeaders.AUTHORIZATION, trustedToken); // expect: end-point used to get items, succeeds and get JSON response body var response = requestSpec.exchange() .expectStatus().isOk() .expectBody(String.class) .returnResult() .getResponseBody(); // and: convery JSON response to List List<Map<String, String>> items = new ObjectMapper().readValue(response, List.class); // verify: items size matches with what we have in database ... // verify: item ids match with what we have in database ... }

Little more Research

But a follow-up reading of Spring Boot Documentation made me dig bit deeper. Actually, I did not need a full web environment to be up with embedded servers started and listening on random port that the above change brings in, which also satisfied auto-configuration need for WebTestClient.

Spring Boot auto configuration has a set of annotations with which you can pick and chose the ones that you really need. Apparently there is one for WebTestClient,  @AutoConfigureWebTestClient. That sounded like the way to go instead of making having the complete test web environment up and running.

So, thought the following would work, but it also failed with the above, same exception:

@SpringBootTest @AutoConfigureWebTestClient @AutoConfigureWebFlux @ActiveProfiles("test") @RunWith(SpringRunner.class) public class MyServiceDataIT { @Autowired private WebTestClient webTestClient; ... }

After little more investigation, I found that we do have dependency: spring-boot-starter-hateoas that Spring Boot Documentation clearly warns on this saying it is meant to be specifically for Spring MVC and should not be used with WebFlux. The alternative in this case is to use: org.springframework.hateoas:spring-hateoas instead. Well, tried that too with the above annotations, but ended up with the same exception.

Better and cleaner solution

The better and cleaner solution is not to have the embedded web server started, but WebTestClient instance created to make a HTTP Get request. So the obvious solution is to ditch dependency injection for WebTestClient and create an instance.

The following is the code snippet:

@SpringBootTest @ActiveProfiles("test") @RunWith(SpringRunner.class) public class MyServiceDataIT { @Value("${spring.other-service-api.get-Items-end-point}") private String apiEndPoint; @Value("${spring.other-service-api.trusted_token}") private String trustedToken; ... @Test public void itemLookup_has_no_missing_items() throws Exception { // given: web test client WebTestClient webTestClient = WebTestClient .bindToServer() .baseUrl(apiEndPoint) .build(); // and: request spec var requestSpec = webTestClient .get() .header(HttpHeaders.ACCEPT, MediaType.APPLICATION_JSON_VALUE) .header(HttpHeaders.AUTHORIZATION, trustedToken); // expect: end-point used to get items, succeeds and get JSON response body var response = requestSpec.exchange() .expectStatus().isOk() .expectBody(String.class) .returnResult() .getResponseBody(); // and: convery JSON response to List List<Map<String, String>> items = new ObjectMapper().readValue(response, List.class); // verify: items size matches with what we have in database ... // verify: item ids match with what we have in database ... }

TIP

Sometimes, WebTestClient times out if the response takes more time than the default value 5000 milliseconds (5 seconds) by throwing the following exception:

java.lang.IllegalStateException: Timeout on blocking read for 5000 MILLISECONDS at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:123) at reactor.core.publisher.Mono.block(Mono.java:1734)

This timeout can be configured in two ways.

1. If WebTestClient is @Autowired with web server starting on a random port, by using annotation @AutoConfigureWebTestClient as shown below:

@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT) @AutoConfigureWebTestClient(timeout = "10000") // millis, 10 seconds @ActiveProfiles("test") @RunWith(SpringRunner.class) public class MyServiceDataIT { @Value("${spring.other-service-api.get-Items-end-point}") private String apiEndPoint; @Value("${spring.other-service-api.trusted_token}") private String trustedToken; @Autowired private WebTestClient webTestClient; ... }

2. If WebTestClient is NOT @AutoWired and the embedded web server is not started, then by setting response timeout while building WebTestClient instance as shown below:

@Test public void itemLookup_has_no_missing_items() throws Exception { // given: web test client WebTestClient webTestClient = WebTestClient .bindToServer() .baseUrl(apiEndPoint) .build() .mutate() .responseTimeout(Duration.ofSeconds(10)) .build(); ... }

Conclusion

These days, most of the times, solutions can be found by googling or on stackoverflow. But, sometimes a clea(ne)r solution takes time to explore and try out.

A software developer's life is never easy, it's always challenging by simple things that look simple on the surface, yet complex under the hoods. Spring Boot framework is no exception in this regard ;)

Here is the link to GitHub repo that contains example code for two different integration test-cases of the two solutions mentioned in this post to try out: spring-boot-gotchas GitHub repo

References

• Testing Spring Boot Applications

Maven - multi-module Java project code coverage . . .

In addition to developing, building & deploying modern Java applications is also a developers' concern. Build tools come with a promise to save developers' time. But often times they suck developers in. Maven is known for that ;)

Choosing the right building tool before you even start an application/project is a standard practice these days. In modern Java world, the two popular build system choices are: Maven and Gradle. Though Gradle started as the "Next Generation Build Tool" with groovy programming language and well designed DSL to write build scripts, and by addressing pitfalls of Maven, maven still rules modern Java world with the legacy XML (Extensible Markup Language), which is not a programming language and is only good for structured data management. The modern software principle "as code" applied to everything these days, even to infrastructure, still doesn't apply to Maven build scripts.

A multi-module project is inevitable if you build any application with modularity and reusability. Maven poses great many challenges in this use-case. There are solutions available for every issue, but you end up spending too much time reading the poor documentation again and again scratching your head, doing more of the same with plugins documentations, and even more of the same in the form of question & answers on the stackoverflow. Clearly, this is not the way, but unfortunately is the way to find solutions, these days.

This post is the result of a 3-day fight with Maven in getting the multi-module code coverage working in a Maven multi-module project. The following 3 maven plugins are in main focus in this are(n)a:

• JaCoCo for code coverage report generation
• Surefire for unit tess
• Failsafe for integration tests

Environment: Java 16, Spring Boot 2.5.3 on macOS Catalina 10.15.7

The Problem Scenario - code in one module, test-cases in another module

It is quite common in multi-module project to have code in one module, and some test-cases if not all in other module(s). For instance, a multi-module maven project with a sharable domain module, a sharable services module and an API micro-service application module (spring-boot based) is a best example of this scenario.

In this scenario, for example, the domain model code can get it's code coverage from unit test-cases as both source code and test-cases reside in the same module. The Maven Jacoco code coverage plugin works quite well in this case. But, it could be bit hard to write integration test-cases for the services module as it requires spring application context and spring-boot configurations. So, the API application module will definitely have a set of integration test-cases as it is a spring boot application with spring context and configurations available. This is the case that requires a better solution for generating code coverage reports by covering the multi-module distributed application code with distributed test-cases.

The two key-points in this scenario are:

1. Test-cases in one module (API application) covering code in other module (services/domain) in addition to the other module's own code coverage.
2. An individual module-wise code coverage report for all the modules with their own code coverage by their own test-cases and coverage threshold checks.
3. A overall consolidated/aggregated but module-wise code coverage report for the entire application code and a code coverage threshold check for the overall code in all modules.

The Solution

The JaCoCo maven plugin from version 0.7.7 onwards offers a new report-aggregate goal. This is the goal that can be leveraged to get an aggregated code coverage report generated. However, it is not straight forward getting this done.

The following is an example multi-module project structure, my-app is the root project, my-app-api is a Spring Boot application with my-app-domain and my-app-services modules that it depends on:

. └── my-app ├── my-app-api │ └── pom.xml ├── my-app-domain │ └── pom.xml ├── my-app-services │ └── pom.xml ├── my-app-code-coverage │ └── pom.xml └── pom.xml

The project's root module my-app's pom.xml file looks something as shown below:

... <modules> <module>my-app-api</module> <module>my-app-domain</module> <module>my-app-services</module> <module>my-app-code-coverage</module> </modules> ... <properties> <jacoco.plugin.version>0.8.7</jacoco.plugin.version> <surefire.plugin.version>2.22.2</surefire.plugin.version> <failsafe.plugin.version>2.22.2</failsafe.plugin.version> </properties> ... <build> <plugins> <!-- jacoco for code coverage --> <plugin> <groupId>org.jacoco</groupId> <artifactId>jacoco-maven-plugin</artifactId> <version>${jacoco.plugin.version}</version> <executions> <!-- jacoco agent for unit-tests code coverage --> <execution> <id>initialize-coverage-before-unit-test-execution</id> <goals> <goal>prepare-agent</goal> </goals> </execution> <!-- jacoco agent for integration-tests code coverage --> <execution> <id>initialize-coverage-before-integration-test-execution</id> <goals> <goal>prepare-agent</goal> </goals> <phase>pre-integration-test</phase> <configuration> <propertyName>integrationTestCoverageAgent</propertyName> </configuration> </execution> </executions> </plugin> <!-- UNIT tests--> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-surefire-plugin</artifactId> <version>${surefire.plugin.version}</version> <configuration> <excludes> <exclude>**/*IT.java</exclude> </excludes> <!-- NOTE: In case if you need to pass in special JVM argumengts for instance say, to enable Java language preview features, the following is how it MUST be done. The @{argLine} goes through late evaluattion that points to Jacoco agent JVM argument followed by any additional JVM arguments of your choice each separated by space. The expression @{argLine} retains Jacoco JVM agent argument. Without this any additional JVM arguments that you add will be taken but you lose Jacoco's argument causing to lose code coverage report files and hence the coverage report. --> <argLine>@{argLine} --enable-preview</argLine> </configuration> </plugin> <!-- INTEGRATION tests --> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-failsafe-plugin</artifactId> <version>${failsafe.plugin.version}</version> <executions> <execution> <id>integration-tests</id> <goals> <goal>integration-test</goal> <goal>verify</goal> </goals> <configuration> <additionalClasspathElements> <additionalClasspathElement>${basedir}/target/classes</additionalClasspathElement> </additionalClasspathElements> <includes> <include>**/*IT.java</include> </includes> <excludes> <exclude>com.my.api.service.MyNotUsedServiceIT</exclude> </excludes> <!-- NOTE: When running as a Maven plugin, the JaCoCo agent configuration is prepared by invoking the prepare-agent or prepare-agent-integration goals, before the actual tests are run. This sets a property named argLine which points to the JaCoCo agent, later passed as a JVM argument to the test runner. --> <argLine>${integrationTestCoverageAgent}</argLine> <!-- NOTE: In case if you need to pass in special JVM argumengts for instance say, to enable Java language preview features, the following is how it MUST be done. The @{argLine} goes through late evaluattion that points to Jacoco agent JVM argument followed by any additional JVM arguments of your choice each separated by space. The expression @{argLine} retains Jacoco JVM agent argument. Without this any additional JVM arguments that you add will be taken but you lose Jacoco's argument causing to lose code coverage report files and hence the coverage report. --> <argLine>@{argLine} --enable-preview</argLine> </configuration> </execution> </executions> </plugin> </plugins> </build> ...

Notable points from the above build file around code coverage are:

• The JaCoCo plugin configuration for code coverage with two execution configurations for unit and integration tests coverage. Make a note of the configuration <propertyName>integrationTestCoverageAgent</propertyName >, it can be any string. The same name should be passed as an argument (argLine) for failsafe configuration. Also, make sure that you retain Jacoco JVM agent argument pointed to by Jacoco added argLine property to be evaluated late in the game as the recommended expression @{argLine} in case if you have additional JVM arguments to be passed for test executions.
• The surefire plugin configuration for unit tests.
• The failsafe plugin configuration for integration tests.

The domain my-app-domain module's pom.xml file is something like shown below:

... <build> <plugins> <plugin> <groupId>org.jacoco</groupId> <artifactId>jacoco-maven-plugin</artifactId> <version>${jacoco.plugin.version}</version> <executions> <execution> <id>generate-code-coverage-report</id> <phase>test</phase> <goals> <goal>report</goal> </goals> </execution> <execution> <id>perform-code-coverage-threshold-check</id> <goals> <goal>check</goal> </goals> <configuration> <!-- Set Rule to fail build if code coverage is below certain threshold --> <rules> <rule implementation="org.jacoco.maven.RuleConfiguration"> <element>BUNDLE</element> <limits> <limit implementation="org.jacoco.report.check.Limit"> <counter>INSTRUCTION</counter> <value>COVEREDRATIO</value> <minimum>0.60</minimum> </limit> </limits> </rule> </rules> </configuration> </execution> </executions> </plugin> ... </plugins> ... </build> ...

The api my-app-api module's pom.xml file is something like shown below, very similar to my-app-domain module:

... <build> <plugins> <plugin> <groupId>org.jacoco</groupId> <artifactId>jacoco-maven-plugin</artifactId> <version>${jacoco.plugin.version}</version> <executions> <execution> <id>generate-code-coverage-report</id> <phase>test</phase> <goals> <goal>report</goal> </goals> </execution> <execution> <id>perform-code-coverage-threshold-check</id> <goals> <goal>check</goal> </goals> <configuration> <!-- Set Rule to fail build if code coverage is below certain threshold --> <rules> <rule implementation="org.jacoco.maven.RuleConfiguration"> <element>BUNDLE</element> <limits> <limit implementation="org.jacoco.report.check.Limit"> <counter>INSTRUCTION</counter> <value>COVEREDRATIO</value> <minimum>0.80</minimum> </limit> </limits> </rule> </rules> </configuration> </execution> </executions> </plugin> ... </plugins> ... </build> ...

Notable points from the above two modules' build files around code coverage are:

• The JaCoCo plugin's additional configuration for code coverage with execution configurations for code coverage report (goal: report) and code coverage threshold check (goal: check) with a threshold ration number (0.60 for domain and 0.80 for api).
• No surefire and failsafe configurations are needed as they are available in sub-modules from the root/main module's build configuration.

The new module, my-app-code-coverage module's pom.xml file for consolidated code coverage is like shown below:

<?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <parent> <groupId>com.giri</groupId> <artifactId>my-app</artifactId> <version>1.0.0-SNAPSHOT</version> </parent> <modelVersion>4.0.0</modelVersion> <artifactId>my-app-code-coverage</artifactId> <packaging>pom</packaging> <name>My Api App Service multi-module code coverage</name> <description>Module for My Api App multi-module code coverage across all modules</description> <properties> <code.coverage.project.dir>${basedir}/../</code.coverage.project.dir> <code.coverage.overall.data.dir>${basedir}/target/</code.coverage.overall.data.dir> <maven-resources-plugin.version>3.2.0</maven-resources-plugin.version> </properties> <dependencies> <dependency> <groupId>com.giri</groupId> <artifactId>my-app-domain</artifactId> <version>${project.version}</version> <scope>compile</scope> </dependency> <dependency> <groupId>com.giri</groupId> <artifactId>my-app-services</artifactId> <version>${project.version}</version> <scope>compile</scope> </dependency> <dependency> <groupId>com.giri</groupId> <artifactId>my-app-api</artifactId> <version>${project.version}</version> <scope>compile</scope> </dependency> </dependencies> <build> <plugins> <!-- required by jacoco for the goal: check to work --> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-resources-plugin</artifactId> <version>${maven-resources-plugin.version}</version> <executions> <execution> <id>copy-class-files</id> <phase>generate-resources</phase> <goals> <goal>copy-resources</goal> </goals> <configuration> <overwrite>false</overwrite> <resources> <resource> <directory>../my-app-domain/target/classes</directory> </resource> <resource> <directory>../my-app-services/target/classes</directory> </resource> <resource> <directory>../my-app-api/target/classes</directory> </resource> </resources> <outputDirectory>${project.build.directory}/classes</outputDirectory> </configuration> </execution> </executions> </plugin> <plugin> <groupId>org.jacoco</groupId> <artifactId>jacoco-maven-plugin</artifactId> <version>${jacoco.plugin.version}</version> <executions> <execution> <id>report-aggregate</id> <phase>verify</phase> <goals> <goal>report-aggregate</goal> </goals> </execution> <execution> <id>merge-results-data</id> <phase>verify</phase> <goals> <goal>merge</goal> </goals> <configuration> <fileSets> <fileSet> <directory>${code.coverage.project.dir}</directory> <includes> <include>**/target/jacoco.exec</include> </includes> </fileSet> </fileSets> <destFile>${code.coverage.overall.data.dir}/aggregate.exec</destFile> </configuration> </execution> <execution> <id>perform-code-coverage-threshold-check</id> <phase>verify</phase> <goals> <goal>check</goal> </goals> <configuration> <dataFile>${code.coverage.overall.data.dir}/aggregate.exec</dataFile> <rules> <rule> <element>BUNDLE</element> <limits> <limit> <counter>INSTRUCTION</counter> <value>COVEREDRATIO</value> <minimum>0.90</minimum> </limit> </limits> </rule> </rules> </configuration> </execution> </executions> </plugin> </plugins> </build> </project>

Notable points from the above build file around code coverage are:

• List all modules that the code lives in as dependencies for this module to get the consolidated report generated.
• Collect all compiled class files from all modules under this module's build directory. This requires maven-resource-plugin and is required for JaCoCo goal: check for coverage threshold check.
• JaCoCo plugin configuration with three executions with goals: report-aggregate, merge, and check for code coverage threshold check.
• The JaCoCo execution goal: report-aggregate is the one that gets aggregate reports generated.
• The JaCoCo goal: merge is needed to merge all modules' jacoco.exec files to be merged into one file.
• And, of course, the JaCoCo goal: check is needed for the overall code coverage threshold check and a threshold ratio number (0.90) which is different than any of the individual module's threshold ration number.

With the above maven module build files, from the root project just run: mvn clean install or mvn clean verify. It cleans, compiles code, runs all test-cases, and generates code coverage reports in each module's build directory: target/site/jacoco. Each module's coverage report shows code coverage attained from test-cases existing within that module. This number could be different (less or equal) for the same module in the overall code coverage report. It also, generates an overall aggregated code coverage report in the newly added module's build directory: my-app-code-coverage/target/site/jacoco-aggregate. The overall code-coverage generates module-wise code coverage with the overall coverage threshold level checked.

TIPS

• Have plugin configurations in the main/root project pom.xml file so that they are available to sub-modules. Only overwrite or add things that are necessary for the sub-module. For example the report goal configuration for JaCoCo in each sub-module and report-aggregate goal configuration for the overall code-coverage module.
• The main/root project can define common code coverage configurations & executions for JaCoCo, surefire and failsafe plugins for all modules with coverage threshold check value set to 0.0 in the root with sub-modules overriding that property with their specific values. That way build scripts can follow the DRY principle.
• It is good to have each module report generated in it's build taget to see the code coverage of the module by it's own test-cases though the special overall code coverage module generates reports for the overall coverage for all modules.
• If there is any module that contains code but not test-cases due to any limitations like not having needed spring application context, and boot configurations, then that module's build file (pom.xml) doesn't need JaCoCo additional configuration for goal: report.
• Another helpful maven goal is help:effective-pom. By running this command: mvn help:effective-pom for any sub-module, you can see and verify the effective pom (XML) with all parent inherited plugins and properties resolved, so that you don't need to do any guess work. This is very useful in multi-module builds to investigate any issues.

GOTCHAS

• If surefire or failsafe plugins do not run unit & integration test-cases and do not leave a clue even when run in debug mode with mvn -X option, just try adding junit dependency surefire-junit47 as described in the plugin documentation to specify the test-framework provider.
• For passing additional JVM arguments like --enable-preview or anything else, make sure to use the expression ${argLine} to retain Jacoco JVM agent argument along with your additional arguments. Missing that expression will miss code coverage and it's hard figuring out.
• With Java 16, you might run into IllegalClassFormatException if your integration test-cases hit any code that uses reflection. The test-cases pass and they get the correct code coverage. This exception in the build output can just be treated as a misleading noise and can be filtered by excluding all those classes that are involved in the reflection. For instance an exception like: java.lang.instrument.IllegalClassFormatException: Error while instrumenting com/giri/app/util/MyClassOneMethodAccess. can be filtered by adding <excludes> to the JaCoCo plugin configuration as shown below:

<build> <plugins> <!-- jacoco for code coverage --> <plugin> <groupId>org.jacoco</groupId> <artifactId>jacoco-maven-plugin</artifactId> <version>${jacoco-maven-plugin.version}</version> <configuration> <!-- Filter the misleading Exception noise by IllegalClassFormatException from JaCoCo instrumentation --> <excludes> <exclude>*MyClassOneMethodAccess*</exclude> <exclude>*MyClassTwoMethodAccess*</exclude> ... </excludes> </configuration> ...

• If code coverage threshold ratio number doesn't match the coverage report total coverage percentage number (less than the threshold number), this could be due to a silent failure in appending the integration tests coverage report to the JaCoCo generated binary coverage report file: jacoco.exec which is used for generating the HTML code coverage reports for both unit and integration tests combined. This file typically gets generated with results after running the unit tests and get appended with results of integration tests. To fix this issue, the combined binary report file can be separated and then merged as shown below. This also gives greater control on code coverage reporting.

<build> <plugins> <!-- jacoco for code coverage --> <plugin> <groupId>org.jacoco</groupId> <artifactId>jacoco-maven-plugin</artifactId> <version>${jacoco-maven-plugin.version}</version> <configuration> <excludes> <exclude>*MyClassOneMethodAccess*</exclude> <exclude>*MyClassTwoMethodAccess*</exclude> </excludes> </configuration> <executions> <!-- jacoco unit test agent for code coverage --> <execution> <id>initialize-coverage-before-unit-test-execution</id> <goals> <goal>prepare-agent</goal> </goals> <configuration> <destFile>${project.build.directory}/jacoco-unit.exec</destFile> </configuration> </execution> <!-- jacoco integration test agent for code coverage --> <execution> <id>initialize-coverage-before-integration-test-execution</id> <goals> <goal>prepare-agent</goal> </goals> <phase>pre-integration-test</phase> <configuration> <propertyName>integrationTestCoverageAgent</propertyName> <destFile>${project.build.directory}/jacoco-integration.exec</destFile> </configuration> </execution> <execution> <id>generate-merged-code-coverage-report</id> <phase>post-integration-test</phase> <goals> <goal>merge</goal> <goal>report</goal> </goals> <configuration> <!-- merge config --> <destFile>${project.build.directory}/jacoco-merged.exec</destFile> <fileSets> <fileSet> <directory>${project.build.directory}</directory> <includes> <include>*.exec</include> </includes> </fileSet> </fileSets> <!-- report config --> <dataFile>${project.build.directory}/jacoco-merged.exec</dataFile> </configuration> </execution> <!-- Threshold check --> <execution> <id>coverage-check</id> <goals> <goal>check</goal> </goals> <configuration> <dataFile>${project.build.directory}/jacoco-merged.exec</dataFile> <!-- Set Rule to fail build if code coverage is below certain threshold --> <rules> <rule implementation="org.jacoco.maven.RuleConfiguration"> <element>BUNDLE</element> <limits> <limit implementation="org.jacoco.report.check.Limit"> <counter>INSTRUCTION</counter> <value>COVEREDRATIO</value> <minimum>${jacoco.percentage.instruction}</minimum> </limit> </limits> </rule> </rules> </configuration> </execution> </executions> </plugin> ...

• Skipping unit/integration tests - both surefire and failsafe plugins offer a default pre-defined property skipTests which is false by default and when set to true skips both unit and integration tests. Unless until needed, no special configuration is needed to get a good hold on running and skipping tests. However, this is bit tricky. From the project-root/main-module run the following to control running tests of my-app-api application module.

Skip all tests:

  ./mvnw -pl my-app-api clean install -DskipTests

Run only unit tests (Skip integration tests): 

  ./mvnw -pl my-app-api surefire:test

Run specific unit test: 

  ./mvnw -pl my-app-api surefire:test -Dtest=MyUtilTest

Run specific set of unit tests, matching pattern:

  ./mvnw -pl my-app-api surefire:test -Dtest=MyU*

Run only integration tests (Skip unit tests):

  ./mvnw -pl my-app-api failsafe:integration-test

Run specific integration test:

  ./mvnw -pl my-app-api failsafe:integration-test -Dit.test=MyAppIT

Run specific set of integration tests, matching pattern:

  ./mvnw -pl my-app-api failsafe:integration-test -Dit.test=MyApp*

Summary

Maven eats up your time. You often get puzzled with many things mixed up in XML files. It's always confusingly challenging to deal with XML as specification for driving application builds.

"Making simple things super-complex" is what Software Engineering is all about. Of course, new concepts, languages, frameworks keep coming in attempts to make complex simple, but in reality only making complex more-complex. Anyways, have FUN with solving build issues/problems, and finding/inventing/re-inventing solutions in Maven & it's plugins.

References

• Jacoco report-aggregate goal
• Reporting aggregated unit and integration test coverage with jacoco
• How to fix a drop of code coverage with a multi module Maven project
• Merging Integration, Unit and functional test reports with JaCoCo
• Jacoco and Surefire argline
• Jacoco prepare-agent goal
• Surefire plugin argLine property expression
• Failsafe plugin argLine property expression
• Selecting test-framework provider
• Why Your JUnit 5 Tests Are Not Running Under Maven