The docker-maven-plugin comes in handy for managing Docker images and containers in integration tests. It can be used to build images or run. Multiple images can be configured to be built or run depending on the need of your application. This post focuses on run aspect of this plugin; specifically running Redis image in Docker container and a potential issue that one might run into.
Environment: Java 20, Spring Boot 3.1.5, maven 3.8.6 on macOS Catalina 10.15.7
The Scenario
After going through some painful hoops, I upgraded a Spring Boot application from 2.6.3 to 3.1.5 with the two-step recommended approach 2.6.x to 2.7.x and then to 3.1.x. The application uses Redis for caching needs. All worked well at the end. The app was built through concourse CI/CD pipeline and successfully deployed as Kubernetes workload to int and cert environments and has been successfully running for few weeks.
I started to work on fixing some critical and high Snyk reported open-source dependency vulnerabilities. Suddenly, integration tests around Redis started to fail with the following exception:
[ERROR] com.hmhco.api.assessmentservice.service.AssessmentServiceCacheIT.testCache -- Time elapsed: 0.478 s <<< ERROR!
org.springframework.data.redis.RedisConnectionFailureException: Unable to connect to Redis
at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory$ExceptionTranslatingConnectionProvider.translateException(LettuceConnectionFactory.java:1604)
at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory$ExceptionTranslatingConnectionProvider.getConnection(LettuceConnectionFactory.java:1535)
at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory$SharedConnection.getNativeConnection(LettuceConnectionFactory.java:1360)
at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory$SharedConnection.getConnection(LettuceConnectionFactory.java:1343)
at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory.getSharedConnection(LettuceConnectionFactory.java:1061)
at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory.getConnection(LettuceConnectionFactory.java:400)
at org.springframework.data.redis.cache.DefaultRedisCacheWriter.execute(DefaultRedisCacheWriter.java:272)
at org.springframework.data.redis.cache.DefaultRedisCacheWriter.clean(DefaultRedisCacheWriter.java:189)
at org.springframework.data.redis.cache.RedisCache.clear(RedisCache.java:220)
The above exception stack-trace didn't give much clue. After looking into few things including redis test configurations, dependencies including transitive dependencies etc. it was still puzzling as it was actually working few weeks ago, and all of sudden only locally the integration test-cases around redis started to fail.
The docker-maven-plugin actually logs that redis:latest started, before running integration tests like below:
...
[INFO] DOCKER> [redis:latest] "redis-test": Start container 395afa51913b
...
The last thing I looked at was the docker containers:
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
395afa51913b redis "docker-entrypoint.s…" About a minute ago Exited (1) About a minute ago redis-1
b03c67ae628e postgres:12.4 "docker-entrypoint.s…" About a minute ago Up About a minute 0.0.0.0:5433->5432/tcp postgres-1
That showed like the container started but exited for some reason. I also looked at the for Docker Dashboard for any additional details.
Clicking redis-1 container that exited shows logs and the reason: # Fatal: Can't initialize Background Jobs. Error message: Operation not permitted
Also gives Redis version detail: 7.2.3
That's the issue. Redis started but exited and hence the test failed with the exception - Unable to connect to Redis
After googling about that initialization issue (# Fatal: Can't initialize Background Jobs. Error message: Operation not permitted), people recommended to try few specific Redis versions. After some trail and error, I had luck with version 6.2.6.
The logs can also be fetched from command-line by executing for the CONTAINER ID (395afa51913b ) or the NAME (redis-1):
$ docker logs 395afa51913b
The Fix
In the docker-maven-plugin configuration, the image tag can be specified for specific version 6.2.6 as shown below:
<properties>
<redis.test.port>6381</redis.test.port>
</properties>
...
<plugin>
<groupId>io.fabric8</groupId>
<artifactId>docker-maven-plugin</artifactId>
<version>0.43.4</version>
<executions>
<execution>
<id>start</id>
<phase>pre-integration-test</phase>
<goals>
<goal>start</goal>
</goals>
</execution>
<execution>
<id>stop</id>
<phase>post-integration-test</phase>
<goals>
<goal>stop</goal>
</goals>
</execution>
</executions>
<configuration>
<images>
<image>
<build>
<cleanup>true</cleanup>
<tags>
<tag>latest</tag>
</tags>
</build>
<external>
<type>properties</type>
<prefix>postgres.docker</prefix>
</external>
</image>
<image>
<name>redis:6.2.6</name>
<alias>redis-test</alias>
<run>
<ports>
<port>${redis.test.port}:6379</port>
</ports>
</run>
</image>
</images>
</configuration>
</plugin>
TIP
Testcontainers is another choice to look into for integration tests.
GOTCHA
It's Gotcha in a Gotcha post ;) I always run into these kinds of issues whenever I explore anything anytime in maven world.
The docker-maven-plugin docker goals start and stop are bound to maven failsafe plugin pre-integration-test and post-integration-test phases of integration-test goal. But if you skip tests by passing argument -DskipTests which actually would skip running all unit and integration tests, I expected docker images not to be run. For instance ./mvnw clean install -DskipTests command to compile, build, package and install all modules by skipping tests. However, in this case, the images do get run: started and stopped. I couldn't find a way to get a hold on this, could be a bug or issue in the plugin by design, not sure.
That was happening right after spring-boot-maven-plugin's package goal. That plugin also gets bound to pre-integration-test and post-integration-test phases when specified. We don't have these executions specified for this plugin though.
Couldn't figure out a workaround for not getting docker images run when integration test cases are skipped. If anyone has a solution for this, please post a comment, I would appreciate it.
The word trivial in software development is actually complex. Things are unnecessarily made super-complex. Software developers get paid for dealing with it anyways ;)
Posts
