Thursday, November 30, 2023

Docker maven plugin - Spring Boot, Redis : Gotcha

The docker-maven-plugin comes in handy for managing Docker images and containers in integration tests. It can be used to build images or run. Multiple images can be configured to be built or run depending on the need of your application. This post focuses on run aspect of this plugin; specifically running Redis image in Docker container and a potential issue that one might run into.

Environment: Java 20, Spring Boot 3.1.5, maven 3.8.6 on macOS Catalina 10.15.7

The Scenario

After going through some painful hoops, I upgraded a Spring Boot application from 2.6.3 to 3.1.5 with the two-step recommended approach 2.6.x to 2.7.x and then to 3.1.x. The application uses Redis for caching needs. All worked well at the end. The app was built through concourse CI/CD pipeline and successfully deployed as Kubernetes workload to int and cert environments and has been successfully running for few weeks.

I started to work on fixing some critical and high Snyk reported open-source dependency vulnerabilities. Suddenly, integration tests around Redis started to fail with the following exception:

[ERROR] com.hmhco.api.assessmentservice.service.AssessmentServiceCacheIT.testCache -- Time elapsed: 0.478 s <<< ERROR! org.springframework.data.redis.RedisConnectionFailureException: Unable to connect to Redis at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory$ExceptionTranslatingConnectionProvider.translateException(LettuceConnectionFactory.java:1604) at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory$ExceptionTranslatingConnectionProvider.getConnection(LettuceConnectionFactory.java:1535) at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory$SharedConnection.getNativeConnection(LettuceConnectionFactory.java:1360) at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory$SharedConnection.getConnection(LettuceConnectionFactory.java:1343) at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory.getSharedConnection(LettuceConnectionFactory.java:1061) at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory.getConnection(LettuceConnectionFactory.java:400) at org.springframework.data.redis.cache.DefaultRedisCacheWriter.execute(DefaultRedisCacheWriter.java:272) at org.springframework.data.redis.cache.DefaultRedisCacheWriter.clean(DefaultRedisCacheWriter.java:189) at org.springframework.data.redis.cache.RedisCache.clear(RedisCache.java:220)

The above exception stack-trace didn't give much clue. After looking into few things including redis test configurations, dependencies including transitive dependencies etc. it was still puzzling as it was actually working few weeks ago, and all of sudden only locally the integration test-cases around redis started to fail.

The docker-maven-plugin actually logs that redis:latest started, before running integration tests like below:

... [INFO] DOCKER> [redis:latest] "redis-test": Start container 395afa51913b ...

The last thing I looked at was the docker containers:

$ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 395afa51913b redis "docker-entrypoint.s…" About a minute ago Exited (1) About a minute ago redis-1 b03c67ae628e postgres:12.4 "docker-entrypoint.s…" About a minute ago Up About a minute 0.0.0.0:5433->5432/tcp postgres-1

That showed like the container started but exited for some reason. I also looked at the for Docker Dashboard for any additional details.


Clicking redis-1 container that exited shows logs and the reason: # Fatal: Can't initialize Background Jobs. Error message: Operation not permitted
Also gives Redis version detail: 7.2.3


That's the issue. Redis started but exited and hence the test failed with the exception - Unable to connect to Redis 

After googling about that initialization issue (# Fatal: Can't initialize Background Jobs. Error message: Operation not permitted), people recommended to try few specific Redis versions. After some trail and error, I had luck with version 6.2.6.

The logs can also be fetched from command-line by executing for the CONTAINER ID (395afa51913b ) or the NAME (redis-1):
$ docker logs 395afa51913b

The Fix

In the docker-maven-plugin configuration, the image tag can be specified for specific version 6.2.6 as shown below:

<properties> <redis.test.port>6381</redis.test.port> </properties> ... <plugin> <groupId>io.fabric8</groupId> <artifactId>docker-maven-plugin</artifactId> <version>0.43.4</version> <executions> <execution> <id>start</id> <phase>pre-integration-test</phase> <goals> <goal>start</goal> </goals> </execution> <execution> <id>stop</id> <phase>post-integration-test</phase> <goals> <goal>stop</goal> </goals> </execution> </executions> <configuration> <images> <image> <build> <cleanup>true</cleanup> <tags> <tag>latest</tag> </tags> </build> <external> <type>properties</type> <prefix>postgres.docker</prefix> </external> </image> <image> <name>redis:6.2.6</name> <alias>redis-test</alias> <run> <ports> <port>${redis.test.port}:6379</port> </ports> </run> </image> </images> </configuration> </plugin>

TIP

Testcontainers is another choice to look into for integration tests.

GOTCHA

It's Gotcha in a Gotcha post ;) I always run into these kinds of issues whenever I explore anything anytime in maven world.

The docker-maven-plugin docker goals start and stop are bound to maven failsafe plugin pre-integration-test and post-integration-test phases of integration-test goal. But if you skip tests by passing argument -DskipTests which actually would skip running all unit and integration tests, I expected docker images not to be run. For instance ./mvnw clean install -DskipTests command to compile, build, package and install all modules by skipping tests. However, in this case, the images do get run: started and stopped. I couldn't find a way to get a hold on this, could be a bug or issue in the plugin by design, not sure.

That was happening right after spring-boot-maven-plugin's package goal. That plugin also gets bound to pre-integration-test and post-integration-test phases when specified. We don't have these executions specified for this plugin though.

Couldn't figure out a workaround for not getting docker images run when integration test cases are skipped. If anyone has a solution for this, please post a comment, I would appreciate it.

The word trivial in software development is actually complex. Things are unnecessarily made super-complex. Software developers get paid for dealing with it anyways ;)

References

Thursday, August 31, 2023

Spring Boot 2.6 to 2.7 - Profile related changes go through increased complexity levels . . .

Indirection is directly proportional to complexity. The more the indirection levels in one direction, the more the turns to take in reverse direction to understand. The higher the indirection levels, the deeper the dive in finding details.

A simple feature like Spring Profiles interwoven across several Spring frameworks is not only made it difficult to understand but also is painful to follow through multiple, disconnected sets of details, and multiple documents. Pulling necessary details together, across different versions of each framework, related to one specific feature is even more painful.

Environment: Java 20, Spring Boot 2.6.3, Spring Boot 2.7.14, maven 3.9.3 on macOS Catalina 10.15.7

Spring Profiles is a pretty simple to understand indirection, abstracted out and externalized with several benefits. Spring - as a framework to support all possible ways that this feature can be supported, also started to evolve. Cloud and cloud related frameworks extended it further increasing the levels of indirection and thus its complexity.

In my recent attempt upgrading a Spring Boot application from Spring Boot 2.6.x to 2.7.x, failed to get deployed quietly with no ERRORS or WARNINGS in Kubernetes (K8S) environment. I noticed a log message about multiple active profiles in logs, and started digging deeper into why there were more than one active profile. There was certainly an interwoven behavioral change between these two Spring Boot minor versions around active profiles with different frameworks working together on profile related configurations, mainly coming in from bootstrap and application yaml files.

The scenario

I did migrate a Spring Boot 2.6.3 application successfully to 2.7.14 and tested locally with local profile. The application also depends on Spring framework support for Vault, and Consul. The default active profile (spring.profiles.active) is set to test in application.yml file. When the application is run locally an explicit active profile parameter is passed (-Dspring-boot.run.profiles=local) to the maven goal: spring-boot:run. So, profile: local passed through maven build option gets passed as Java option, takes the precedence, and overrides the default active profile test. The default active profile test is good for the default profile active as there is no explicit profile specified during maven test phase when it runs unit and integration tests. Environment specific configuration files like application-int.yml, application-int.yml have spring.config.activate.on-profile set to respective environment. So, environment specific configuration (e.g. application-int.yml) is considered when active profile is set to specific environment (e.g. int). All looked good before and after the upgrade.

When the application gets built through concourse pipeline, after successful build including unit and integration tests run, it automatically gets deployed to both int and cert. The deployment script sets Java option -Dspring.profiles.active=<env>, thus overriding the default active profile test with specific env as command line option takes the precedence over configuration files. So, all worked as expected for local profile after the upgrade and went for deployment with the same expectations.

Also, initially the application had one set of yaml files (application) when Apache Mesos was the deployment platform. Later it got migrated to K8S deployment with added Vault and Consul support, which forced a new set of yaml configuration files (bootstrap). Mainly Vault and Consul required certain properties to be configured through bootstrap yaml set. So after  the application got migrated from Apache Mesos to K8S, it ended up having two different sets of application properties: bootstrap.yml set and application.yml set, along with their counterpart env specific files for local, test, int, cert and prod. With that there were 2 sets of 7 yaml files for each set. The first set: bootstrap, bootstrap-local, bootstrap-test, bootstrap-dev, bootstrap-int, bootstrap-cert and bootstrap-prod. The second set:  application, application-local, application-test, application-dev, application-int, application-cert and application-prod.

The issue

The upgraded application (upgraded to Spring Boot 2.7.14) failed to come up successfully after it got deployed to K8S cluster through Concourse CI/CD pipeline with logs showing two active profiles during the application startup after deployment onto both int and cert. For instance on int logs, both test and int were logged in as active profiles.

It was bit hard to dig into where this profiles handling of active profile getting overridden by the Java option passed was broken - instead of overriding, it was appended to active profiles. Spring allows more than one profile to be specified to be active separated by comma treating it as a list of active profiles. When multiple profiles are active, each profile specific configuration is read in the order of environments that appear in the list, and configuration properties get consolidated, with the later environment related properties replacing the previous environment related if there are same properties configured in both. In this case, it's like merging multiple environment specific configurations.

In my case, test profile was default active. The active profile passed through Java options, instead of replacing the default list which has just one element test, it was getting prepended to the list. So in int the active profiles were: int, test and similarly in cert: cert, test. The test profile being the last element in the list, was taking the precedence and the application in both int and cert failed to start up as test environment specific configuration was not good for int or cert.

The fix (better one) - separating out test configurations into it's own configuration area under test

All application, and bootstrap yaml files resided under src/main/resources directory.

The fix I did for this issue involved the following changes:
1. Removed default active profile configuration property set in application.yml file.
2. Moved test profile specific configuration files (application-test.yml and bootstrap-test.yml) that contain spring.config.activate.on-profile property set to test along with other test environment related properties under src/test/resources directory.
3. Added new application.yml and bootstrap.yml files under test/resources directory. Both just contain one property spring.profiles.active set to test.

With this separation, all configuration files including the base and environment specific reside under src/main/resources directory. There is no active profile set at all in any of these configuration files under src/main/resources. In other words, active profile is always passed in as an option for all environments: local, dev, int, cert, and prod.

Test environment configuration files: application.yml, bootstrap.yml, application-test.yml, and bootstrap-test.yml reside under src/test/resources directory.
Both application.yml, and bootstrap.yml have active profile set to test. So spring testing framework considers these configurations when found under src/test/resources during unit and integration tests. That way test environment active profile and it's associated test specific configurations are totally isolated from all other env configurations into its own configuration area: src/test/resources. 

This is more cleaner approach. Also isolates test configurations under src/test/resources area which are not even bundled into the application jar.

Summary

Enterprise Java's unnecessary overcomplexity gave birth to Spring Framework and it has quickly become popular and a de-facto Java framework since then. The core of it is based on the simple Dependency Injection (or Inversion of Control - IoC) design pattern. It started to grow into every corner of the technology concept. It is not simple anymore, in my opinion. There are too many layers of details one needs to know. With tons and tons Spring-eco-system frameworks, it has become even more complex. A framework started to simplify Enterprise Java development has grown too big to be(come) complex.

A simple Java framework like JUnit itself is getting complex. So, no wonder Spring being in there already. Software developers absolutely love indirection and complexity. In Software Development simplicity is a rare quality. Even if some simplicity exists, it slowly becomes complex over a period of time. Simple becomes complex, complex only becomes super complex ;)

Happy software development, and enjoy the complexity!

References

Monday, August 28, 2023

IntelliJ - the Community Edition rescued the broken Ultimate Edition . . . ;)

It sound funny to say that - IntelliJ Community Edition (free edition) rescued Ultimate Edition (paid edition). But that's what really put me back on Ultimate Edition after many months of switching to Community Edition for my day-to-day development.

Environment: IntelliJ Community Edition 2023.2.1, IntelliJ Ultimate Edition 2023.2.1, maven 3.9.3 on macOS Catalina 10.15.7

The Issue

I stopped using IntelliJ Ultimate Edition as for some weird reason it got stuck with maven dependencies broken paths issue. Specifically, SLF4J and javax libraries were shown in maven dependency broken paths in red. It happened few months ago. One day one of my maven projects that uses Lombok and had classes annotated with @Slf4J suddenly couldn't recognize the log (logger object) statements. When I checked project module dependencies there were broken maven dependency paths.

I tried all recommended and possible ways to recover from this issue like: Invalidate Caches and restart the IDE(A), blowing away specific libraries maven cache ~/.m2/libraries/.../*.* and letting it build again, blowing away entire maven local cache ~/.m2/*.* and letting it build again, downloading dependency jars and explicitly setting dependency paths in the project/module settings, reinstalling the Ultimate Edition, even reinstalling different version of the Ultimate Edition etc. Nothing worked. I spent lot of time few times since then and couldn't get it back to working. Finally gave up and moved to Community Edition. Same projects that were having broken dependency issues in Ultimate Edition, when opened in Community edition, had no issues with those dependencies, same maven local cache paths are used by both. The Ultimate Edition complains, the Community Edition doesn't. Alas!

The Fix

Here is what I did to fix it.

Started IntelliJ Ultimate Edition. At the startup there is a Customize link, click that and click Import Settings... link as shown below:

Selected the Community Edition settings directory (~/Library/Application Support/JetBrains/IdeaIC2023.2) of Community Edition in which my projects were fine with dependencies. It prompted to take a backup of current settings. I did that and imported ommunity Edition settings. It opened projects and the issue of broken dependency paths was gone.

It seemed like, there was some broken dependency path setting saved in the Ultimate Edition settings that was stuck and not getting fixed by any means.

Some Internals

IntelliJ saves all it's settings under the specific version's area. On Mac, by default, this area is under ~/Library/Application Support/JetBrains dir. The community edition directories start with IdeaIC<version> (e.g IdeaIC2023.2) and ultimate edition directories start with IntelliJIdea<version> (e.g. IntelliJIdea2023.2). The options sub-directory is where plugins settings get saved. 

Plugin Settings

Plugin settings are pat of IntelliJ settings and get stored in xml files under options sub-directory of specific IntelliJ version's settings directory. For instance, awesome editor plugin is a simple and pretty neat plugin which lets add image backgrounds. I did setup plugin awesome editor to display different kinds of images for different types of files. To get all the settings from one IntelliJ version to another, just copy the plugin settings file (in this case it is: awesome-editor-3.xml). 

Here is an example to copy plugin settings set in Ultimate edition to Community Edition.
NOTE: Once copied restart IntelliJ Community Edition.

$ cd ~/Library/"Application Support"/JetBrains $ ls -al |grep IdeaIC drwxr-xr-x 17 pottepalemg 163264107 544 Jul 26 2022 IdeaIC2022.1 drwxr-xr-x 20 pottepalemg 163264107 640 Nov 30 2022 IdeaIC2022.2 drwxr-xr-x 20 pottepalemg 163264107 640 Mar 30 15:23 IdeaIC2022.3 drwxr-xr-x 20 pottepalemg 163264107 640 Jul 7 11:31 IdeaIC2023.1 drwxr-xr-x 20 pottepalemg 163264107 640 Aug 28 10:13 IdeaIC2023.2 $ ls -al |grep IntelliJIdea drwxr-xr-x 22 pottepalemg 163264107 704 Aug 24 15:43 IntelliJIdea2020.1 drwxr-xr-x 19 pottepalemg 163264107 608 Nov 19 2020 IntelliJIdea2020.2 drwxr-xr-x 21 pottepalemg 163264107 672 Mar 2 2022 IntelliJIdea2020.3 drwxr-xr-x 20 pottepalemg 163264107 640 Jul 9 2021 IntelliJIdea2021.1 drwxr-xr-x 24 pottepalemg 163264107 768 Feb 9 2022 IntelliJIdea2021.2 drwxr-xr-x 25 pottepalemg 163264107 800 Apr 12 2022 IntelliJIdea2021.3 drwxr-xr-x 26 pottepalemg 163264107 832 Sep 23 2022 IntelliJIdea2022.1 drwxr-xr-x 25 pottepalemg 163264107 800 Nov 30 2022 IntelliJIdea2022.2 drwxr-xr-x 25 pottepalemg 163264107 800 Jun 28 13:26 IntelliJIdea2022.3 drwxr-xr-x 24 pottepalemg 163264107 768 Aug 16 15:40 IntelliJIdea2023.1 drwxr-xr-x 22 pottepalemg 163264107 704 Aug 28 10:36 IntelliJIdea2023.2 drwxr-xr-x 11 pottepalemg 163264107 352 Aug 25 15:35 IntelliJIdea2023.2-backup $ find . -name awesome* ./IntelliJIdea2023.2/options/awesome-editor-3.xml $ ls -ltr ./IdeaIC2023.2/options | grep awesome* $ cp ./IntelliJIdea2023.2/options/awesome-editor-3.xml ./IdeaIC2023.2/options

TIPS

Getting back lost database connection settings into Ultimate Edition

IntelliJ Ultimate Edition comes bundled with Database plugin that supports all features that are available in DataGrip (an SQL IDE which is a product of JetBrains). I had database connections set to connect to various PostgreSQL databases (local, int, cert, prod etc.) which I lost by importing Community Edition settings. But I had a settings backup prompted and done for the Ultimate Edition settings that my broken Ultimate Edition was setup with when I imported Community Edition Settings. The backup directory is also listed in the above list of Ultimate Editions setting directories. To get those settings back onto my new settings imported from Community Edition, I had to repeat the Customize > Import Settings... step two more times. First time pointing it to the backup directory and selecting and copying all database connections settings to the clipboard. Second time pointing it to the Community Edition Directory and pasting those connection settings from the clipboard. This was the only way I could get those database connections copied. I got all connection settings except passwords fro every connection. I had to set password for every single connection individually. This was not possible by simply copying xml files like I did for awesome editor plugin.

Summary

No software application or tool is bug-free. Applications do crash, tools do get corrupted. There is no one solution that works or fixes a similar issue for everybody. Some stupid, nasty, unknown, not very well documented internals of tools do take up lot of time to discover a fix that works for your situation and may help some others who get into similar situation.

Hope this blog post on my discovery saves someone's time sometime when that someone bumps into it.