Friday, December 29, 2023

Testable Spring Boot Application main method with Docker Compose added - passing required arguments . . .

Java application's entry point is its main method which takes String array as argument. The command line arguments are passed by Java runtime system to the main method as an array of Strings.

Spring Boot application is a command line runnable Java application with a main method. The SpringApplication class is used to bootstrap and launch Spring app from Java's main method. Spring framework promotes POJO development and hence makes it easy to write testable code. A simple JUnit test can be written to test the main method.

This post is about writing a simple JUnit test case for Spring Boot application's main method in the context of Docker Compose dependency, and passing required arguments.

Environment: Java 21, Spring Boot 3.2.1, maven 3.9.6 on macOS Catalina 10.15.7

A typical, simple Spring Boot application class with a main method looks like:
import org.springframework.boot.SpringApplication; import org.springframework.boot.autoconfigure.SpringBootApplication; @SpringBootApplication public class GraalVmApplication { public static void main(String[] args) { SpringApplication.run(GraalVmApplication.class, args); } }

A simple JUnit test case to test the application looks like:
import org.junit.jupiter.api.Test; public class ApplicationStartsTest { @Test void application_starts() { GraalVmApplication.main(new String[]{}); } }

The test case is simple. It just invokes application's main method which in turn launches the Spring boot application and exits. Note that there are no arguments passed, so it's launched with default profile, which is ok to make sure the application runs. 

The Scenario

Now, let's says that we want to add JPA and Postgres DB support to our application. Also want to try Spring support for Docker Compose by adding Postgres database service docker container for database. Spring docker support takes care of starting up the Postgres docker container and bringing it down when the app exists. All it requires is to to add spring-boot-docker-compose dependency and compose.yaml or docker-compose.yaml under the root project. Refer to this Sample Git project.

With the JPA, spring-boot-docker-compose dependencies and docker-compose.yaml added, the above JUnit test case fails to run with the following exception:
*************************** APPLICATION FAILED TO START *************************** Description: Failed to configure a DataSource: 'url' attribute is not specified and no embedded datasource could be configured. Reason: Failed to determine a suitable driver class Action: Consider the following: If you want an embedded database (H2, HSQL or Derby), please put it on the classpath. If you have database settings to be loaded from a particular profile you may need to activate it (no profiles are currently active). [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 3.731 s <<< FAILURE! -- in com.giri.boot.graalvm.ApplicationStartsTest [ERROR] com.giri.boot.graalvm.ApplicationStartsTest.applicationStarts -- Time elapsed: 3.705 s <<< ERROR! org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'dataSourceScriptDatabaseInitializer' defined in class path resource [org/springframework/boot/autoconfigure/sql/init/DataSourceInitializationConfiguration.class]: Unsatisfied dependency expressed through method 'dataSourceScriptDatabaseInitializer' parameter 0: Error creating bean with name 'dataSource' defined in class path resource [org/springframework/boot/autoconfigure/jdbc/DataSourceConfiguration$Hikari.class]: Failed to instantiate [com.zaxxer.hikari.HikariDataSource]: Factory method 'dataSource' threw exception with message: Failed to determine a suitable driver class at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:802) at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:546) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1334) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1164) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:561) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:521) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:325) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:323) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:312) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199) at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1232) at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:950) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:625) at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:146) at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:762) at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:464) at org.springframework.boot.SpringApplication.run(SpringApplication.java:334) at org.springframework.boot.SpringApplication.run(SpringApplication.java:1358) at org.springframework.boot.SpringApplication.run(SpringApplication.java:1347) at com.giri.boot.graalvm.GraalVmApplication.main(GraalVmApplication.java:12) at com.giri.boot.graalvm.ApplicationStartsTest.applicationStarts(ApplicationStartsTest.java:17) at java.base/java.lang.reflect.Method.invoke(Method.java:580) at java.base/java.util.ArrayList.forEach(ArrayList.java:1596) at java.base/java.util.ArrayList.forEach(ArrayList.java:1596) Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dataSource' defined in class path resource [org/springframework/boot/autoconfigure/jdbc/DataSourceConfiguration$Hikari.class]: Failed to instantiate [com.zaxxer.hikari.HikariDataSource]: Factory method 'dataSource' threw exception with message: Failed to determine a suitable driver class at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:655) at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:643) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1334) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1164) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:561) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:521) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:325) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:323) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199) at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:254) at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1443) at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1353) at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:911) at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:789)

The reason for the failure is -  by default docker compose support is disabled when running tests.

The Fix

The fix is to pass additional arguments like - enable docker compose support for this test, profile (local) for which datasource properties are configured etc. With these, when the unit test is run, docker compose is up, which brings up Postgres container and the application connects to the database through datasource configuration properties specified in application-local.yml file for local profile. With the fix, the test case looks like:

public class ApplicationStartsTest { @Test void applicationStarts() { GraalVmApplication.main( new String[]{ "--spring.profiles.active=local", "--spring.docker.compose.skip.in-tests=false", "--spring.docker.compose.file=docker/docker-compose.yaml" } ); } }

Now, the test case passes. Below is the output with docker compose highlighted:
[INFO] ------------------------------------------------------- [INFO] T E S T S [INFO] ------------------------------------------------------- [INFO] Running com.giri.boot.graalvm.ApplicationStartsTest arg: --spring.profiles.active=local arg: --spring.docker.compose.skip.in-tests=false arg: --spring.docker.compose.file=docker/docker-compose.yaml ____ _ ____ ___ __ | __ ) ___ ___ | |_ / ___|_ __ __ _ __ _| \ \ / / __ ___ | _ \ / _ \ / _ \| __| | | _| '__/ _` |/ _` | |\ \ / / '_ ` _ \ | |_) | (_) | (_) | |_ | |_| | | | (_| | (_| | | \ V /| | | | | | |____/ \___/ \___/ \__| \____|_| \__,_|\__,_|_| \_/ |_| |_| |_| :: Spring Boot :: 3.2.1 :: Running on Java :: 21 :: Application :: 0.0.1-SNAPSHOT 2023-12-28T20:09:11.059-05:00 INFO 65561 --- [boot-graalvm] [ main] c.giri.boot.graalvm.GraalVmApplication : Starting GraalVmApplication using Java 21 with PID 65561 (started by pottepalemg in /Users/pottepalemg/dev/boot-graalvm) 2023-12-28T20:09:11.065-05:00 INFO 65561 --- [boot-graalvm] [ main] c.giri.boot.graalvm.GraalVmApplication : The following 1 profile is active: "local" 2023-12-28T20:09:11.523-05:00 INFO 65561 --- [boot-graalvm] [ main] .s.b.d.c.l.DockerComposeLifecycleManager : Using Docker Compose file '/Users/pottepalemg/dev/boot-graalvm/docker/docker-compose.yaml' 2023-12-28T20:09:12.547-05:00 INFO 65561 --- [boot-graalvm] [utReader-stderr] o.s.boot.docker.compose.core.DockerCli : Container docker-postgres-1 Creating 2023-12-28T20:09:12.614-05:00 INFO 65561 --- [boot-graalvm] [utReader-stderr] o.s.boot.docker.compose.core.DockerCli : Container docker-postgres-1 Created 2023-12-28T20:09:12.618-05:00 INFO 65561 --- [boot-graalvm] [utReader-stderr] o.s.boot.docker.compose.core.DockerCli : Container docker-postgres-1 Starting 2023-12-28T20:09:13.169-05:00 INFO 65561 --- [boot-graalvm] [utReader-stderr] o.s.boot.docker.compose.core.DockerCli : Container docker-postgres-1 Started 2023-12-28T20:09:13.174-05:00 INFO 65561 --- [boot-graalvm] [utReader-stderr] o.s.boot.docker.compose.core.DockerCli : Container docker-postgres-1 Waiting 2023-12-28T20:09:13.682-05:00 INFO 65561 --- [boot-graalvm] [utReader-stderr] o.s.boot.docker.compose.core.DockerCli : Container docker-postgres-1 Healthy 2023-12-28T20:09:16.211-05:00 INFO 65561 --- [boot-graalvm] [ main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode. 2023-12-28T20:09:16.237-05:00 INFO 65561 --- [boot-graalvm] [ main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 17 ms. Found 0 JPA repository interfaces. 2023-12-28T20:09:17.133-05:00 INFO 65561 --- [boot-graalvm] [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port 8080 (http) 2023-12-28T20:09:17.157-05:00 INFO 65561 --- [boot-graalvm] [ main] o.apache.catalina.core.StandardService : Starting service [Tomcat] 2023-12-28T20:09:17.157-05:00 INFO 65561 --- [boot-graalvm] [ main] o.apache.catalina.core.StandardEngine : Starting Servlet engine: [Apache Tomcat/10.1.17] 2023-12-28T20:09:17.233-05:00 INFO 65561 --- [boot-graalvm] [ main] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext 2023-12-28T20:09:17.234-05:00 INFO 65561 --- [boot-graalvm] [ main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 2082 ms 2023-12-28T20:09:17.648-05:00 INFO 65561 --- [boot-graalvm] [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Starting... 2023-12-28T20:09:17.895-05:00 INFO 65561 --- [boot-graalvm] [ main] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Added connection org.postgresql.jdbc.PgConnection@24a99b1c 2023-12-28T20:09:17.897-05:00 INFO 65561 --- [boot-graalvm] [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Start completed. 2023-12-28T20:09:17.970-05:00 INFO 65561 --- [boot-graalvm] [ main] o.hibernate.jpa.internal.util.LogHelper : HHH000204: Processing PersistenceUnitInfo [name: default] 2023-12-28T20:09:18.102-05:00 INFO 65561 --- [boot-graalvm] [ main] org.hibernate.Version : HHH000412: Hibernate ORM core version 6.4.1.Final 2023-12-28T20:09:18.163-05:00 INFO 65561 --- [boot-graalvm] [ main] o.h.c.internal.RegionFactoryInitiator : HHH000026: Second-level cache disabled 2023-12-28T20:09:18.557-05:00 INFO 65561 --- [boot-graalvm] [ main] o.s.o.j.p.SpringPersistenceUnitInfo : No LoadTimeWeaver setup: ignoring JPA class transformer 2023-12-28T20:09:19.123-05:00 INFO 65561 --- [boot-graalvm] [ main] o.h.e.t.j.p.i.JtaPlatformInitiator : HHH000489: No JTA platform available (set 'hibernate.transaction.jta.platform' to enable JTA platform integration) 2023-12-28T20:09:19.129-05:00 INFO 65561 --- [boot-graalvm] [ main] j.LocalContainerEntityManagerFactoryBean : Initialized JPA EntityManagerFactory for persistence unit 'default' 2023-12-28T20:09:19.398-05:00 WARN 65561 --- [boot-graalvm] [ main] JpaBaseConfiguration$JpaWebConfiguration : spring.jpa.open-in-view is enabled by default. Therefore, database queries may be performed during view rendering. Explicitly configure spring.jpa.open-in-view to disable this warning 2023-12-28T20:09:20.474-05:00 INFO 65561 --- [boot-graalvm] [ main] o.s.b.a.e.web.EndpointLinksResolver : Exposing 13 endpoint(s) beneath base path '/actuator' 2023-12-28T20:09:20.559-05:00 INFO 65561 --- [boot-graalvm] [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port 8080 (http) with context path '' 2023-12-28T20:09:20.576-05:00 INFO 65561 --- [boot-graalvm] [ main] c.giri.boot.graalvm.GraalVmApplication : Started GraalVmApplication in 10.156 seconds (process running for 11.266) [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.57 s -- in com.giri.boot.graalvm.ApplicationStartsTest [INFO] [INFO] Results: [INFO] [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0 [INFO] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------

💡 TIPS

It is also helpful to print arguments in the Spring boot application:
@SpringBootApplication public class GraalVmApplication { public static void main(String[] args) { for (String arg: args) { // let's examine arguments passed System.out.println("arg: " + arg); } SpringApplication.run(GraalVmApplication.class, args); } }

Conclusion

Much of the learning comes, not from working code, but from failing code. If everything works on the first go, there isn't much learned ;)
Keep exploring!
Happy Holidays!!

References

Thursday, December 28, 2023

Spring Boot, Docker compose on Mac OS Catalina - Gotcha . . .

Spring Boot 3.0 enhanced docker compose support with which container startup and destroy is taken care by spring boot and is quite convenient.

Anything I try first time, I hit road blocks. It's both good and bad. The good part is - it makes me explore things little deeper, which otherwise is not possible. The bad part is - it is frustrating and needs patience to get it to work.

Environment: Java 21, Spring Boot 3.2.0, maven 3.9.6 on macOS Catalina 10.15.7

Scenario

I wanted to add Postgres db service through docker compose. So, just added docker compose support to an existing simple Spring Boot 3.2.0 application. All that I had to do was simple. Add the dependency in pom.xml and a docker-compose.yml file in the root directory of the project. This is customizable through the property: spring.docker.compose.file.

pom.xml
... <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-docker-compose</artifactId> <scope>runtime</scope> <optional>true</optional> </dependency> ...

docker-compose.yaml
services: postgres: image: 'postgres16:16.0' environment: - 'POSTGRES_DB=boot-graalvm' - 'POSTGRES_USER=postgres' - 'POSTGRES_PASSWORD=s3cr3t' ports: - '5222:5432'

Application fails to run with the following error:
2023-12-27T08:58:48.534-05:00 ERROR 229 --- [ main] o.s.boot.SpringApplication : Application run failed org.springframework.boot.docker.compose.core.ProcessExitException: 'docker-compose version --format json' failed with exit code 1. Stdout: Stderr: Show version information and quit. Usage: version [--short] Options: --short Shows only Compose's version number. at org.springframework.boot.docker.compose.core.ProcessRunner.run(ProcessRunner.java:96) ~[spring-boot-docker-compose-3.2.0.jar:3.2.0] at org.springframework.boot.docker.compose.core.ProcessRunner.run(ProcessRunner.java:74) ~[spring-boot-docker-compose-3.2.0.jar:3.2.0] at org.springframework.boot.docker.compose.core.DockerCli$DockerCommands.getDockerComposeCommand(DockerCli.java:165) ~[spring-boot-docker-compose-3.2.0.jar:3.2.0] at org.springframework.boot.docker.compose.core.DockerCli$DockerCommands.(DockerCli.java:130) ~[spring-boot-docker-compose-3.2.0.jar:3.2.0] at org.springframework.boot.docker.compose.core.DockerCli.lambda$new$0(DockerCli.java:65) ~[spring-boot-docker-compose-3.2.0.jar:3.2.0] at java.base/java.util.HashMap.computeIfAbsent(HashMap.java:1228) ~[na:na] at org.springframework.boot.docker.compose.core.DockerCli.(DockerCli.java:64) ~[spring-boot-docker-compose-3.2.0.jar:3.2.0] at org.springframework.boot.docker.compose.core.DockerCompose.get(DockerCompose.java:92) ~[spring-boot-docker-compose-3.2.0.jar:3.2.0] at org.springframework.boot.docker.compose.lifecycle.DockerComposeLifecycleManager.getDockerCompose(DockerComposeLifecycleManager.java:149) ~[spring-boot-docker-compose-3.2.0.jar:3.2.0] at org.springframework.boot.docker.compose.lifecycle.DockerComposeLifecycleManager.start(DockerComposeLifecycleManager.java:110) ~[spring-boot-docker-compose-3.2.0.jar:3.2.0] at org.springframework.boot.docker.compose.lifecycle.DockerComposeListener.onApplicationEvent(DockerComposeListener.java:53) ~[spring-boot-docker-compose-3.2.0.jar:3.2.0] at org.springframework.boot.docker.compose.lifecycle.DockerComposeListener.onApplicationEvent(DockerComposeListener.java:35) ~[spring-boot-docker-compose-3.2.0.jar:3.2.0] at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:178) ~[spring-context-6.1.1.jar:6.1.1] at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:171) ~[spring-context-6.1.1.jar:6.1.1] at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:149) ~[spring-context-6.1.1.jar:6.1.1] at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:137) ~[spring-context-6.1.1.jar:6.1.1] at org.springframework.boot.context.event.EventPublishingRunListener.multicastInitialEvent(EventPublishingRunListener.java:136) ~[spring-boot-3.2.0.jar:3.2.0] at org.springframework.boot.context.event.EventPublishingRunListener.contextLoaded(EventPublishingRunListener.java:98) ~[spring-boot-3.2.0.jar:3.2.0] at org.springframework.boot.SpringApplicationRunListeners.lambda$contextLoaded$4(SpringApplicationRunListeners.java:72) ~[spring-boot-3.2.0.jar:3.2.0] at java.base/java.lang.Iterable.forEach(Iterable.java:75) ~[na:na] at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:118) ~[spring-boot-3.2.0.jar:3.2.0] at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:112) ~[spring-boot-3.2.0.jar:3.2.0] at org.springframework.boot.SpringApplicationRunListeners.contextLoaded(SpringApplicationRunListeners.java:72) ~[spring-boot-3.2.0.jar:3.2.0] at org.springframework.boot.SpringApplication.prepareContext(SpringApplication.java:431) ~[spring-boot-3.2.0.jar:3.2.0] at org.springframework.boot.SpringApplication.run(SpringApplication.java:322) ~[spring-boot-3.2.0.jar:3.2.0] at org.springframework.boot.SpringApplication.run(SpringApplication.java:1342) ~[spring-boot-3.2.0.jar:3.2.0] at org.springframework.boot.SpringApplication.run(SpringApplication.java:1331) ~[spring-boot-3.2.0.jar:3.2.0] [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------


The Fix

The issue was the older version of Docker Desktop 2.5.0.1 running on my laptop. The menu item Check for Updates... wouldn't even work. It seemed like I was stuck with very old version of Docker Desktop. The fix was to upgrade to a version that works. Yes, you have to find out the version that works for Mac OS Catalina. I tried and learned the fact that Docker Desktop latest version (4.26.1) doesn't run on Mac OS Catalina and Docker Desktop silently removed the support for Mac OS Catalina in 4.16.0. So the only version to upgrade for Mac OS Catalina is: 4.15.0.

Jeez, what a mess of higher elevated software development. It's just ridiculous!

There is no easy way even to find the version 4.15.0 download link. Finally found the links for both the versions: 2.5.0.1 that I had and 4.15.0 that I want to upgrade to. 

To downgrade to 2.5.0.1, here is the link: https://desktop.docker.com/mac/stable/49550/Docker.dmg
To upgrade to 4.15.0, here is the link: https://desktop.docker.com/mac/main/amd64/93002/Docker.dmg

TIPS

1. Spring Boot docker compose out-of-the-box looks for docker compose file named compose.yaml or docker-compose.yaml in the project root directory. If you want to place the file anywhere else under the project root folder, for instance like a docker sub-dir under project root, here is how the customization property should look like in the appropriate application.yml or application-<env>.yml related yml file:

spring: docker: compose: file: docker/docker-compose.yaml

2. Docker compose is enabled by default. When the dependent jar is found, it looks for compose.yaml/docker-compose.yaml. It can be disabled by setting explicitly the property spring.docker.enabled to false. Typically, docker compose is good for local development. If there are multiple profiles (environments) that the application is configured to run in, the application.properties or application.yml file should have docker compose disabled and it can be enabled in the appropriate profile that it's used, e.g local.

3. In the compose.yaml or docker-compose.yaml file, note that we specify the HOST_POST and CONTAINER_PORT for PostgreSQL in the format HOST_PORT:CONTAINER_PORT. The container port is 5432 (postgres default). The host port also can be 5432. But in case if you want to have multiple postgres databases managed by Docker for multiple applications/versions, you need to use different HOST_PORT for each image for both to be up and running at the same time. The HOST_PORT is the port you use to connect to the database.

4. Give a unique name for each postgres image if you need to have multiple versions of postgres instances managed by Docker. Otherwise, you will end up having just one. You can use version number in the image name to distinguish.

5. Check docker and docker-compose versions:
$ docker -v or docker --version
$ docker compose version

6. If the application fails with the following exception:

org.springframework.boot.docker.compose.lifecycle.ReadinessTimeoutException: Readiness timeout of PT2M reached while waiting for services

Add the following to docker-compose.yaml file 
services: postgres: image: 'postgres16:16.0' environment: - 'POSTGRES_DB=boot-graalvm' - 'POSTGRES_USER=postgres' - 'POSTGRES_PASSWORD=s3cr3t' ports: - '5222:5432' labels: org.springframework.boot.readiness-check.tcp.disable: true

References

Thursday, December 21, 2023

Looking into Testcontainers . . .

Everything is easy and stays easy only up until experienced. In software development the "Aha-moment" of someone would be a "Huh-moment" of some other. We hear the phrase "It worked for me" almost daily that whispers aloud "It didn't work for someone else".

This blog post is not about Testcontainers but an issue that took awhile for me to find a workaround for. Testcontainers is a framework that provides lightweight throwaway instance for anything that run in a docker container. It's just appropriate for using in integration tests that require resources like Database server, Redis instance etc.

With the enhanced support in Spring Boot 3, Testcontainers got much simpler and better. Much of this enhanced support comes in as a new annotation: @ServieConnection. A bean method annotated with this annotation is enough and Spring Boot takes care of the rest - managing the life cycle of the container like staring at the application startup and stopping, establishing connection to the service running in the container etc. No other configuration like datasource connection is needed. Check this detailed Spring team blogpost on this.

The enthusiasm of "Lets give it a try", I got stuck for a couple of hours with a problem that I couldn't find a solution by tirelessly by trying many possible including googling, reading and what not.

Environment: Java 20, Spring Boot 3.2.1, maven 3.9.6 on macOS Catalina 10.15.7

The Scenario

The scenario is simple. I wanted to quickly try the newly enhanced and improved Spring Boot support for Testcontainers with PostgreSQL database.

However, I got stuck with getting the container started successfully for PostgreSQL database image. Started with the latest Postgres image tag, tried specific older tags including 16, 16.1. Downgraded testcontainer library versions from the latest 1.19.3 to 1.18.3 and few other as I hit for people who faced similar issue said it worked with other versions.

For everything that I tried, the issue was at least consistent with the following stacktrace:

2023-12-20T12:07:56.443-05:00 INFO 60313 --- [boot-graalvm] [ main] tc.postgres:16 : Container postgres:16 is starting: 995cb534ef6982062dcc98d9841cdf7962c0e1d37c809f3e479d06667cc567f7 2023-12-20T12:08:56.871-05:00 ERROR 60313 --- [boot-graalvm] [ main] tc.postgres:16 : Could not start container java.lang.IllegalStateException: Failed to load ApplicationContext for [WebMergedContextConfiguration@2df9b4f3 testClass = com.giri.boot.graalvm.GraalVmApplicationIT, locations = [], classes = [com.giri.boot.graalvm.GraalVmApplication], contextInitializerClasses = [], activeProfiles = ["test"], propertySourceDescriptors = [], propertySourceProperties = ["org.springframework.boot.test.context.SpringBootTestContextBootstrapper=true"], contextCustomizers = [[ImportsContextCustomizer@6112390a key = [com.giri.boot.graalvm.config.TestContainersConfiguration]], org.springframework.boot.test.context.filter.ExcludeFilterContextCustomizer@bdc8014, org.springframework.boot.test.json.DuplicateJsonObjectContextCustomizerFactory$DuplicateJsonObjectContextCustomizer@13047d7d, org.springframework.boot.test.mock.mockito.MockitoContextCustomizer@0, org.springframework.boot.test.web.client.TestRestTemplateContextCustomizer@675ffd1d, org.springframework.boot.test.autoconfigure.actuate.observability.ObservabilityContextCustomizerFactory$DisableObservabilityContextCustomizer@1f, org.springframework.boot.test.autoconfigure.properties.PropertyMappingContextCustomizer@0, org.springframework.boot.test.autoconfigure.web.servlet.WebDriverContextCustomizer@3104351d, org.spockframework.spring.mock.SpockContextCustomizer@0, org.springframework.boot.testcontainers.service.connection.ServiceConnectionContextCustomizer@0, org.springframework.boot.test.context.SpringBootTestAnnotation@156baa3c], resourceBasePath = "src/main/webapp", contextLoader = org.springframework.boot.test.context.SpringBootContextLoader, parent = null] at org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate.loadContext(DefaultCacheAwareContextLoaderDelegate.java:180) at org.springframework.test.context.support.DefaultTestContext.getApplicationContext(DefaultTestContext.java:130) at org.springframework.test.context.web.ServletTestExecutionListener.setUpRequestContextIfNecessary(ServletTestExecutionListener.java:191) at org.springframework.test.context.web.ServletTestExecutionListener.prepareTestInstance(ServletTestExecutionListener.java:130) at org.springframework.test.context.TestContextManager.prepareTestInstance(TestContextManager.java:247) at org.springframework.test.context.junit.jupiter.SpringExtension.postProcessTestInstance(SpringExtension.java:163) at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179) at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1708) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) at java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.forEachRemaining(StreamSpliterators.java:310) at java.base/java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:735) at java.base/java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:734) at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:762) at java.base/java.util.Optional.orElseGet(Optional.java:364) at java.base/java.util.ArrayList.forEach(ArrayList.java:1596) at java.base/java.util.ArrayList.forEach(ArrayList.java:1596) Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'graalVmApplication': Error creating bean with name 'postgreSQLContainer' defined in class path resource [com/giri/boot/graalvm/config/TestContainersConfiguration.class]: Container startup failed for image postgres:16 at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:608) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:523) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:325) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:323) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199) at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:973) at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:946) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:616) at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:753) at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:455) at org.springframework.boot.SpringApplication.run(SpringApplication.java:323) at org.springframework.boot.test.context.SpringBootContextLoader.lambda$loadContext$3(SpringBootContextLoader.java:137) at org.springframework.util.function.ThrowingSupplier.get(ThrowingSupplier.java:58) at org.springframework.util.function.ThrowingSupplier.get(ThrowingSupplier.java:46) at org.springframework.boot.SpringApplication.withHook(SpringApplication.java:1442) at org.springframework.boot.test.context.SpringBootContextLoader$ContextLoaderHook.run(SpringBootContextLoader.java:552) at org.springframework.boot.test.context.SpringBootContextLoader.loadContext(SpringBootContextLoader.java:137) at org.springframework.boot.test.context.SpringBootContextLoader.loadContext(SpringBootContextLoader.java:108) at org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate.loadContextInternal(DefaultCacheAwareContextLoaderDelegate.java:225) at org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate.loadContext(DefaultCacheAwareContextLoaderDelegate.java:152) ... 17 more Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'postgreSQLContainer' defined in class path resource [com/giri/boot/graalvm/config/TestContainersConfiguration.class]: Container startup failed for image postgres:16 at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:608) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:523) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:325) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:323) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199) at org.springframework.boot.testcontainers.lifecycle.TestcontainersLifecycleBeanPostProcessor.getBeans(TestcontainersLifecycleBeanPostProcessor.java:128) at org.springframework.boot.testcontainers.lifecycle.TestcontainersLifecycleBeanPostProcessor.initializeContainers(TestcontainersLifecycleBeanPostProcessor.java:119) at org.springframework.boot.testcontainers.lifecycle.TestcontainersLifecycleBeanPostProcessor.postProcessAfterInitialization(TestcontainersLifecycleBeanPostProcessor.java:79) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsAfterInitialization(AbstractAutowireCapableBeanFactory.java:437) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1778) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:601) ... 37 more Caused by: org.testcontainers.containers.ContainerLaunchException: Container startup failed for image postgres:16 at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:362) at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:333) at java.base/java.lang.Iterable.forEach(Iterable.java:75) at org.springframework.boot.testcontainers.lifecycle.TestcontainersStartup$1.start(TestcontainersStartup.java:43) at org.springframework.boot.testcontainers.lifecycle.TestcontainersLifecycleBeanPostProcessor.start(TestcontainersLifecycleBeanPostProcessor.java:114) at org.springframework.boot.testcontainers.lifecycle.TestcontainersLifecycleBeanPostProcessor.initializeStartables(TestcontainersLifecycleBeanPostProcessor.java:103) at org.springframework.boot.testcontainers.lifecycle.TestcontainersLifecycleBeanPostProcessor.postProcessAfterInitialization(TestcontainersLifecycleBeanPostProcessor.java:83) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsAfterInitialization(AbstractAutowireCapableBeanFactory.java:437) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1778) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:601) ... 48 more Caused by: org.rnorth.ducttape.RetryCountExceededException: Retry limit hit with exception at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:88) at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:347) ... 57 more Caused by: org.testcontainers.containers.ContainerLaunchException: Could not create/start container at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:566) at org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:357) at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81) ... 58 more Caused by: java.lang.IllegalStateException: Wait strategy failed. Container exited with code 1 at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:536) ... 60 more Caused by: org.testcontainers.containers.ContainerLaunchException: Timed out waiting for log output matching '.*database system is ready to accept connections.*\s' at org.testcontainers.containers.wait.strategy.LogMessageWaitStrategy.waitUntilReady(LogMessageWaitStrategy.java:47) at org.testcontainers.containers.wait.strategy.AbstractWaitStrategy.waitUntilReady(AbstractWaitStrategy.java:52) at org.testcontainers.containers.PostgreSQLContainer.waitUntilContainerStarted(PostgreSQLContainer.java:147) at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:503) ... 60 more

Finally the logs from the container Docker dashboard for postgres:16 container that failed with the message was the last resort looked into:

popen failure: Cannot allocate memory

initdb: error: program "postgres" is needed by initdb but was not found in the same directory as "/usr/lib/postgresql/16/bin/initdb"


That wasn't that helpful. By googling for the above error message, I landed at a blog post. The solution that was described there worked for me as well.

The solution

Use Alpine Postgres. Diving further deeper is not worth my time at this time. I had to move on with what worked for me ;). Couldn't resist that saying, "it (also) worked for me".

The following is the code snippet:
@TestConfiguration(proxyBeanMethods = false) @Slf4j public class TestContainersConfiguration { private static final String POSTGRES_IMAGE_TAG = "postgres:16-alpine"; // Issue: postgres:latest or postgres:16 @Bean @ServiceConnection PostgreSQLContainer<?> postgreSQLContainer() { log.info("PostgreSQLContainer bean..."); return new PostgreSQLContainer<>(DockerImageName.parse(POSTGRES_IMAGE_TAG)) .withDatabaseName("boot-graalvm") .withUsername("postgres") .withPassword("s3cr3t"); } }


What is Alpine Linux?

Alpine Linux is a lightweight Linux distribution with the focus on SSS distribution - Small.Simple.Secure. It is famous for its small size with the fastest boot time, hence heavily used in containers.

💡 TIPS

Check your docker desktop version. If you have old version like what I have: 2.5.0.1, it is time to upgrade to newer version that works. For Mac OS Catalina, the newer version that works is 4.15.0.
Check this blog post on this.

Conclusion

The elevated development experience with underpinned layers of languages, frameworks, and systems increase the depth of things to know and understand for a Software Engineer, keeping the self-training capabilities and building knowledge, the real and not artificial, a daily challenge.

References

Tuesday, December 12, 2023

Spot-check Spotless . . .

Sometimes, overwhelming details are underwhelming. I happened to quickly explore Spotless for code formatting and checks. Spotless is the plugin to spot check. It's good that the README of Spotless upfront has separated Gradle and Maven documentation. I was looking to add maven support for a new Java Spring Boot 3.0 GraphQL project. But the Spotless Maven README documentation was overwhelming.

Quickly read through for few minutes to get an essence of it. Though there are so many details, what I wanted to get done seemed simple. Of course, practicality is always different. That's where experience is gained or comes from ;)

Environment: Java 20, Spring Boot 3.2.0, maven 3.9.6 on macOS Catalina 10.15.7

I quickly applied it to the project, and immediately ran into a couple of Gotchas. It took me couple of hours to explore and set it up finding issues and fixing settings. This post is just about those issues that I ran into.

Use Spaces, not Tabs in code

I prefer SPACES to TABS in code. It is good practice to use SPACE instead of TAB in code formatting. This cannot be a personal preference. But most of IDEs still come with TAB as default setting and many developers don't even pay attention to it. In code reviews, the formatting goes off due to SPACESs vs. TABs and is always annoying.

I use IntelliJ IDEA for my development. Whenever I install IntelliJ, the very first thing I would setup is to use spaces instead of tabs (Go to Preferences > Editor > Code Style > Java, and uncheck Use tab character). With that all code that I write will only use spaces and not tabs. The next thing is to setup to show whitespaces (Go to Preferences > Editor > Appearance, and check Show whitespaces), to easily distinguish Tabs and Spaces by this setting on.

Gotcha-1

The Spotless maven plugin documentation describes many details. The setting I wanted for code check was found at a couple of places in there under <indent> tag. At only one place the setting <spaces>true</spaces> is specified. Though I tried few different things like false for <tabs>, and true for <spaces> etc., the spotless:check maven goal was still suggesting to change code with spaces to tabs.

We also have to use google code style settings: eclipse-formatter.xml, that I took from the other project to use. This is described under Java > eclipse jdt in the spotless documentation. It's a google code style Java settings guide in xml format ;)

The setting id that is used for what I wanted was like: <setting id="org.eclipse.jdt.core.formatter.tabulation.char" value="tab"/>. Changing that "tab" to "space" made spotless happy with spaces in code. ;)

Gotcha-2

Binding to maven phase : I should have read it carefully. At the minimum the <executions><execution><goal>check</goal></execution></execution> is required. Without this ./mvnw spotless:check works, but ./mvnw clean install which also runs verify will not run spotless check. So, for spotless check goal to be bound to maven verify goal, the above setting is required.

A sample spotless settings looks like this:

... <plugins> <!-- spotless https://github.com/diffplug/spotless --> <plugin> <groupId>com.diffplug.spotless</groupId> <artifactId>spotless-maven-plugin</artifactId> <version>${spotless-plugin.version}</version> <executions> <execution> <goals> <goal>check</goal> </goals> </execution> </executions> <configuration> <java> <endWithNewline /> <trimTrailingWhitespace /> <cleanthat> <version>2.18</version> <mutators> <mutator>AvoidInlineConditionals</mutator> <mutator>LiteralsFirstInComparisons</mutator> <mutator>UnnecessaryImport</mutator> <mutator>UnnecessaryModifier</mutator> <mutator>UseUnderscoresInNumericLiterals</mutator> <mutator>UseDiamondOperator</mutator> </mutators> </cleanthat> <eclipse> <version>4.26</version> <file>${project.basedir}/eclipse-formatter.xml</file> </eclipse> </java> </configuration> </plugin> ... </plugins>

💡 TIPS

Spotless handy maven goals
Check: ./mvnw spotless:check
Apply: ./mvnw spotless:apply

Conclusion

Any new exploration in Maven world never goes smooth for me. There are always bumps along the way. I do not like copy and paste at all. After all, if everything works in the first place when quick-copy-paste-tool is used, there is not much left to learn ;)  With the Generative AI getting trained to write code, learning will soon become a rare thing!

References

Thursday, November 30, 2023

Docker maven plugin - Spring Boot, Redis : Gotcha

The docker-maven-plugin comes in handy for managing Docker images and containers in integration tests. It can be used to build images or run. Multiple images can be configured to be built or run depending on the need of your application. This post focuses on run aspect of this plugin; specifically running Redis image in Docker container and a potential issue that one might run into.

Environment: Java 20, Spring Boot 3.1.5, maven 3.8.6 on macOS Catalina 10.15.7

The Scenario

After going through some painful hoops, I upgraded a Spring Boot application from 2.6.3 to 3.1.5 with the two-step recommended approach 2.6.x to 2.7.x and then to 3.1.x. The application uses Redis for caching needs. All worked well at the end. The app was built through concourse CI/CD pipeline and successfully deployed as Kubernetes workload to int and cert environments and has been successfully running for few weeks.

I started to work on fixing some critical and high Snyk reported open-source dependency vulnerabilities. Suddenly, integration tests around Redis started to fail with the following exception:

[ERROR] com.hmhco.api.assessmentservice.service.AssessmentServiceCacheIT.testCache -- Time elapsed: 0.478 s <<< ERROR! org.springframework.data.redis.RedisConnectionFailureException: Unable to connect to Redis at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory$ExceptionTranslatingConnectionProvider.translateException(LettuceConnectionFactory.java:1604) at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory$ExceptionTranslatingConnectionProvider.getConnection(LettuceConnectionFactory.java:1535) at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory$SharedConnection.getNativeConnection(LettuceConnectionFactory.java:1360) at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory$SharedConnection.getConnection(LettuceConnectionFactory.java:1343) at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory.getSharedConnection(LettuceConnectionFactory.java:1061) at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory.getConnection(LettuceConnectionFactory.java:400) at org.springframework.data.redis.cache.DefaultRedisCacheWriter.execute(DefaultRedisCacheWriter.java:272) at org.springframework.data.redis.cache.DefaultRedisCacheWriter.clean(DefaultRedisCacheWriter.java:189) at org.springframework.data.redis.cache.RedisCache.clear(RedisCache.java:220)

The above exception stack-trace didn't give much clue. After looking into few things including redis test configurations, dependencies including transitive dependencies etc. it was still puzzling as it was actually working few weeks ago, and all of sudden only locally the integration test-cases around redis started to fail.

The docker-maven-plugin actually logs that redis:latest started, before running integration tests like below:

... [INFO] DOCKER> [redis:latest] "redis-test": Start container 395afa51913b ...

The last thing I looked at was the docker containers:

$ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 395afa51913b redis "docker-entrypoint.s…" About a minute ago Exited (1) About a minute ago redis-1 b03c67ae628e postgres:12.4 "docker-entrypoint.s…" About a minute ago Up About a minute 0.0.0.0:5433->5432/tcp postgres-1

That showed like the container started but exited for some reason. I also looked at the for Docker Dashboard for any additional details.


Clicking redis-1 container that exited shows logs and the reason: # Fatal: Can't initialize Background Jobs. Error message: Operation not permitted
Also gives Redis version detail: 7.2.3


That's the issue. Redis started but exited and hence the test failed with the exception - Unable to connect to Redis 

After googling about that initialization issue (# Fatal: Can't initialize Background Jobs. Error message: Operation not permitted), people recommended to try few specific Redis versions. After some trail and error, I had luck with version 6.2.6.

The logs can also be fetched from command-line by executing for the CONTAINER ID (395afa51913b ) or the NAME (redis-1):
$ docker logs 395afa51913b

The Fix

In the docker-maven-plugin configuration, the image tag can be specified for specific version 6.2.6 as shown below:

<properties> <redis.test.port>6381</redis.test.port> </properties> ... <plugin> <groupId>io.fabric8</groupId> <artifactId>docker-maven-plugin</artifactId> <version>0.43.4</version> <executions> <execution> <id>start</id> <phase>pre-integration-test</phase> <goals> <goal>start</goal> </goals> </execution> <execution> <id>stop</id> <phase>post-integration-test</phase> <goals> <goal>stop</goal> </goals> </execution> </executions> <configuration> <images> <image> <build> <cleanup>true</cleanup> <tags> <tag>latest</tag> </tags> </build> <external> <type>properties</type> <prefix>postgres.docker</prefix> </external> </image> <image> <name>redis:6.2.6</name> <alias>redis-test</alias> <run> <ports> <port>${redis.test.port}:6379</port> </ports> </run> </image> </images> </configuration> </plugin>

TIP

Testcontainers is another choice to look into for integration tests.

GOTCHA

It's Gotcha in a Gotcha post ;) I always run into these kinds of issues whenever I explore anything anytime in maven world.

The docker-maven-plugin docker goals start and stop are bound to maven failsafe plugin pre-integration-test and post-integration-test phases of integration-test goal. But if you skip tests by passing argument -DskipTests which actually would skip running all unit and integration tests, I expected docker images not to be run. For instance ./mvnw clean install -DskipTests command to compile, build, package and install all modules by skipping tests. However, in this case, the images do get run: started and stopped. I couldn't find a way to get a hold on this, could be a bug or issue in the plugin by design, not sure.

That was happening right after spring-boot-maven-plugin's package goal. That plugin also gets bound to pre-integration-test and post-integration-test phases when specified. We don't have these executions specified for this plugin though.

Couldn't figure out a workaround for not getting docker images run when integration test cases are skipped. If anyone has a solution for this, please post a comment, I would appreciate it.

The word trivial in software development is actually complex. Things are unnecessarily made super-complex. Software developers get paid for dealing with it anyways ;)

References

Thursday, August 31, 2023

Spring Boot 2.6 to 2.7 - Profile related changes go through increased complexity levels . . .

Indirection is directly proportional to complexity. The more the indirection levels in one direction, the more the turns to take in reverse direction to understand. The higher the indirection levels, the deeper the dive in finding details.

A simple feature like Spring Profiles interwoven across several Spring frameworks is not only made it difficult to understand but also is painful to follow through multiple, disconnected sets of details, and multiple documents. Pulling necessary details together, across different versions of each framework, related to one specific feature is even more painful.

Environment: Java 20, Spring Boot 2.6.3, Spring Boot 2.7.14, maven 3.9.3 on macOS Catalina 10.15.7

Spring Profiles is a pretty simple to understand indirection, abstracted out and externalized with several benefits. Spring - as a framework to support all possible ways that this feature can be supported, also started to evolve. Cloud and cloud related frameworks extended it further increasing the levels of indirection and thus its complexity.

In my recent attempt upgrading a Spring Boot application from Spring Boot 2.6.x to 2.7.x, failed to get deployed quietly with no ERRORS or WARNINGS in Kubernetes (K8S) environment. I noticed a log message about multiple active profiles in logs, and started digging deeper into why there were more than one active profile. There was certainly an interwoven behavioral change between these two Spring Boot minor versions around active profiles with different frameworks working together on profile related configurations, mainly coming in from bootstrap and application yaml files.

The scenario

I did migrate a Spring Boot 2.6.3 application successfully to 2.7.14 and tested locally with local profile. The application also depends on Spring framework support for Vault, and Consul. The default active profile (spring.profiles.active) is set to test in application.yml file. When the application is run locally an explicit active profile parameter is passed (-Dspring-boot.run.profiles=local) to the maven goal: spring-boot:run. So, profile: local passed through maven build option gets passed as Java option, takes the precedence, and overrides the default active profile test. The default active profile test is good for the default profile active as there is no explicit profile specified during maven test phase when it runs unit and integration tests. Environment specific configuration files like application-int.ymlapplication-int.yml have spring.config.activate.on-profile set to respective environment. So, environment specific configuration (e.g. application-int.yml) is considered when active profile is set to specific environment (e.g. int). All looked good before and after the upgrade.

When the application gets built through concourse pipeline, after successful build including unit and integration tests run, it automatically gets deployed to both int and cert. The deployment script sets Java option -Dspring.profiles.active=<env>, thus overriding the default active profile test with specific env as command line option takes the precedence over configuration files. So, all worked as expected for local profile after the upgrade and went for deployment with the same expectations.

Also, initially the application had one set of yaml files (application) when Apache Mesos was the deployment platform. Later it got migrated to K8S deployment with added Vault and Consul support, which forced a new set of yaml configuration files (bootstrap). Mainly Vault and Consul required certain properties to be configured through bootstrap yaml set. So after  the application got migrated from Apache Mesos to K8S, it ended up having two different sets of application properties: bootstrap.yml set and application.yml set, along with their counterpart env specific files for localtestintcert and prod. With that there were 2 sets of 7 yaml files for each set. The first set: bootstrap, bootstrap-local, bootstrap-test, bootstrap-dev, bootstrap-int, bootstrap-cert and bootstrap-prod. The second set:  application, application-local, application-test, application-dev, application-int, application-cert and application-prod.

The issue

The upgraded application (upgraded to Spring Boot 2.7.14) failed to come up successfully after it got deployed to K8S cluster through Concourse CI/CD pipeline with logs showing two active profiles during the application startup after deployment onto both int and cert. For instance on int logs, both test and int were logged in as active profiles.

It was bit hard to dig into where this profiles handling of active profile getting overridden by the Java option passed was broken - instead of overriding, it was appended to active profiles. Spring allows more than one profile to be specified to be active separated by comma treating it as a list of active profiles. When multiple profiles are active, each profile specific configuration is read in the order of environments that appear in the list, and configuration properties get consolidated, with the later environment related properties replacing the previous environment related if there are same properties configured in both. In this case, it's like merging multiple environment specific configurations.

In my case, test profile was default active. The active profile passed through Java options, instead of replacing the default list which has just one element test, it was getting prepended to the list. So in int the active profiles were: int, test and similarly in cert: cert, test. The test profile being the last element in the list, was taking the precedence and the application in both int and cert failed to start up as test environment specific configuration was not good for int or cert.

The fix (better one) - separating out test configurations into it's own configuration area under test

All application, and bootstrap yaml files resided under src/main/resources directory.

The fix I did for this issue involved the following changes:
1. Removed default active profile configuration property set in application.yml file.
2. Moved test profile specific configuration files (application-test.yml and bootstrap-test.yml) that contain spring.config.activate.on-profile property set to test along with other test environment related properties under src/test/resources directory.
3. Added new application.yml and bootstrap.yml files under test/resources directory. Both just contain one property spring.profiles.active set to test.

With this separation, all configuration files including the base and environment specific reside under src/main/resources directory. There is no active profile set at all in any of these configuration files under src/main/resources. In other words, active profile is always passed in as an option for all environments: local, dev, int, cert, and prod.

Test environment configuration files: application.ymlbootstrap.ymlapplication-test.yml, and bootstrap-test.yml reside under src/test/resources directory.
Both application.yml, and bootstrap.yml have active profile set to test. So spring testing framework considers these configurations when found under src/test/resources during unit and integration tests. That way test environment active profile and it's associated test specific configurations are totally isolated from all other env configurations into its own configuration area: src/test/resources

This is more cleaner approach. Also isolates test configurations under src/test/resources area which are not even bundled into the application jar.

Summary

Enterprise Java's unnecessary overcomplexity gave birth to Spring Framework and it has quickly become popular and a de-facto Java framework since then. The core of it is based on the simple Dependency Injection (or Inversion of Control - IoC) design pattern. It started to grow into every corner of the technology concept. It is not simple anymore, in my opinion. There are too many layers of details one needs to know. With tons and tons Spring-eco-system frameworks, it has become even more complex. A framework started to simplify Enterprise Java development has grown too big to be(come) complex.

A simple Java framework like JUnit itself is getting complex. So, no wonder Spring being in there already. Software developers absolutely love indirection and complexity. In Software Development simplicity is a rare quality. Even if some simplicity exists, it slowly becomes complex over a period of time. Simple becomes complex, complex only becomes super complex ;)

Happy software development, and enjoy the complexity!

References

Monday, August 28, 2023

IntelliJ - the Community Edition rescued the broken Ultimate Edition . . . ;)

It sound funny to say that - IntelliJ Community Edition (free edition) rescued Ultimate Edition (paid edition). But that's what really put me back on Ultimate Edition after many months of switching to Community Edition for my day-to-day development.

Environment: IntelliJ Community Edition 2023.2.1IntelliJ Ultimate Edition 2023.2.1, maven 3.9.3 on macOS Catalina 10.15.7

The Issue

I stopped using IntelliJ Ultimate Edition as for some weird reason it got stuck with maven dependencies broken paths issue. Specifically, SLF4J and javax libraries were shown in maven dependency broken paths in red. It happened few months ago. One day one of my maven projects that uses Lombok and had classes annotated with @Slf4J suddenly couldn't recognize the log (logger object) statements. When I checked project module dependencies there were broken maven dependency paths.

I tried all recommended and possible ways to recover from this issue like: Invalidate Caches and restart the IDE(A), blowing away specific libraries maven cache ~/.m2/libraries/.../*.* and letting it build again, blowing away entire maven local cache ~/.m2/*.* and letting it build again, downloading dependency jars and explicitly setting dependency paths in the project/module settings, reinstalling the Ultimate Edition, even reinstalling different version of the Ultimate Edition etc. Nothing worked. I spent lot of time few times since then and couldn't get it back to working. Finally gave up and moved to Community Edition. Same projects that were having broken dependency issues in Ultimate Edition, when opened in Community edition, had no issues with those dependencies, same maven local cache paths are used by both. The Ultimate Edition complains, the Community Edition doesn't. Alas!

The Fix

Here is what I did to fix it.

Started IntelliJ Ultimate Edition. At the startup there is a Customize link, click that and click Import Settings... link as shown below:

Selected the Community Edition settings directory (~/Library/Application Support/JetBrains/IdeaIC2023.2) of Community Edition in which my projects were fine with dependencies. It prompted to take a backup of current settings. I did that and imported ommunity Edition settings. It opened projects and the issue of broken dependency paths was gone.

It seemed like, there was some broken dependency path setting saved in the Ultimate Edition settings that was stuck and not getting fixed by any means.

Some Internals

IntelliJ saves all it's settings under the specific version's area. On Mac, by default, this area is under ~/Library/Application Support/JetBrains dir. The community edition directories start with IdeaIC<version> (e.g IdeaIC2023.2) and ultimate edition directories start with IntelliJIdea<version> (e.g. IntelliJIdea2023.2). The options sub-directory is where plugins settings get saved. 

Plugin Settings

Plugin settings are pat of IntelliJ settings and get stored in xml files under options sub-directory of specific IntelliJ version's settings directory. For instance, awesome editor plugin is a simple and pretty neat plugin which lets add image backgrounds. I did setup plugin awesome editor to display different kinds of images for different types of files. To get all the settings from one IntelliJ version to another, just copy the plugin settings file (in this case it is: awesome-editor-3.xml). 

Here is an example to copy plugin settings set in Ultimate edition to Community Edition.
NOTE: Once copied restart IntelliJ Community Edition.

$ cd ~/Library/"Application Support"/JetBrains $ ls -al |grep IdeaIC drwxr-xr-x 17 pottepalemg 163264107 544 Jul 26 2022 IdeaIC2022.1 drwxr-xr-x 20 pottepalemg 163264107 640 Nov 30 2022 IdeaIC2022.2 drwxr-xr-x 20 pottepalemg 163264107 640 Mar 30 15:23 IdeaIC2022.3 drwxr-xr-x 20 pottepalemg 163264107 640 Jul 7 11:31 IdeaIC2023.1 drwxr-xr-x 20 pottepalemg 163264107 640 Aug 28 10:13 IdeaIC2023.2 $ ls -al |grep IntelliJIdea drwxr-xr-x 22 pottepalemg 163264107 704 Aug 24 15:43 IntelliJIdea2020.1 drwxr-xr-x 19 pottepalemg 163264107 608 Nov 19 2020 IntelliJIdea2020.2 drwxr-xr-x 21 pottepalemg 163264107 672 Mar 2 2022 IntelliJIdea2020.3 drwxr-xr-x 20 pottepalemg 163264107 640 Jul 9 2021 IntelliJIdea2021.1 drwxr-xr-x 24 pottepalemg 163264107 768 Feb 9 2022 IntelliJIdea2021.2 drwxr-xr-x 25 pottepalemg 163264107 800 Apr 12 2022 IntelliJIdea2021.3 drwxr-xr-x 26 pottepalemg 163264107 832 Sep 23 2022 IntelliJIdea2022.1 drwxr-xr-x 25 pottepalemg 163264107 800 Nov 30 2022 IntelliJIdea2022.2 drwxr-xr-x 25 pottepalemg 163264107 800 Jun 28 13:26 IntelliJIdea2022.3 drwxr-xr-x 24 pottepalemg 163264107 768 Aug 16 15:40 IntelliJIdea2023.1 drwxr-xr-x 22 pottepalemg 163264107 704 Aug 28 10:36 IntelliJIdea2023.2 drwxr-xr-x 11 pottepalemg 163264107 352 Aug 25 15:35 IntelliJIdea2023.2-backup $ find . -name awesome* ./IntelliJIdea2023.2/options/awesome-editor-3.xml $ ls -ltr ./IdeaIC2023.2/options | grep awesome* $ cp ./IntelliJIdea2023.2/options/awesome-editor-3.xml ./IdeaIC2023.2/options

TIPS

Getting back lost database connection settings into Ultimate Edition

IntelliJ Ultimate Edition comes bundled with Database plugin that supports all features that are available in DataGrip (an SQL IDE which is a product of JetBrains). I had database connections set to connect to various PostgreSQL databases (local, int, cert, prod etc.) which I lost by importing Community Edition settings. But I had a settings backup prompted and done for the Ultimate Edition settings that my broken Ultimate Edition was setup with when I imported Community Edition Settings. The backup directory is also listed in the above list of Ultimate Editions setting directories. To get those settings back onto my new settings imported from Community Edition, I had to repeat the Customize > Import Settings... step two more times. First time pointing it to the backup directory and selecting and copying all database connections settings to the clipboard. Second time pointing it to the Community Edition Directory and pasting those connection settings from the clipboard. This was the only way I could get those database connections copied. I got all connection settings except passwords fro every connection. I had to set password for every single connection individually. This was not possible by simply copying xml files like I did for awesome editor plugin.

Summary

No software application or tool is bug-free. Applications do crash, tools do get corrupted. There is no one solution that works or fixes a similar issue for everybody. Some stupid, nasty, unknown, not very well documented internals of tools do take up lot of time to discover a fix that works for your situation and may help some others who get into similar situation.

Hope this blog post on my discovery saves someone's time sometime when that someone bumps into it.