Know test cases run order in Grails-3 app to help solve issues with "test pollution". . .

In a Grails 3 application that I upgraded recently from Grail 2.2.1 to Grails 3.2.1, it was very puzzling when an integration specification (test case upgraded from Grails 2.2.1) with few feature methods (test case methods) passed locally but failed on the Bamboo CI server. From the kind of failure, it was evident that this particular specification failed due to "data pollution"- some unwanted data hanging around in the database by the time it ran.

The failed specification was an integration test specification and I was under the impression when grails test-app command is run, the test specifications are run in the order of test categories: unit tests, followed by integration tests, followed by functional tests.  If that was the case, the integration test that failed should have never failed as all integration specifications were properly annotated with @Integration and @Rollback and there was no setup() method creating data that couldn't be rolled back by @Rollback annotation causing the pollution. Grails 3 doesn't distinguish integration tests from functional test (at least when they are run as part of integrationTest gradle task), though they are distinguished by code. Typically integration tests extend Spock's Specification and annotated with @Integration and @Rollback, where as functional tests simply extend GebSpec.

Both local and CI server ran test-app task against an empty database. The only difference was: local was on Mac OS X and Bamboo CI server was on Linux. There was a functional test (GebSpec) in the set of integration tests under integration-test/groovy/myapp dir which had a bunch of feature methods. That one obviously did not have any data cleanup methods like: cleanup() or cleanupSpec(). I didn't even want to do any cleanup at the end in that functional spec because it was perfectly fine with local run. That led me to think of test data pollution causing this issue. But the most puzzling question was: "Why this functional test was coming in between and polluting integration test cases?" The only way to find it out was to know the order in which test-cases run and compare local run with Bamboo CI run.

I read through some documentation of Spock and Junit but didn't find an easy way of knowing the order of tests. Spock supports @Stepwise annotation to specify the order, but it was only within a specification. I read through some Gradle documentation about the test task and found a way to tap into the lifecycle of test cases and print the description of each test case that gets run. This helped me finding test-cases run order, and compare local with CI run to nail down the issue. Locally on Mac OS X, test-cases ran in alphabetical order, where as on Bamboo CI server (Linux), they seemed running in random order and one GebSpec functional test that was part of integration tests that got run along with all integration tests during integrationTest task as part of grails test-app task was the culprit. It was a coincidence that the functional specification name alphabetically was the last in the set of integration tests. On Mac OS X, it ran as the very last test case but on Linux it was running in between causing the following test to fail by leaving data in the database and thus leading to "test data pollution".

Following is the code snippet I added in build.gralde that prints each test before it's run:

/** * Configure test and integrationTest tasks. * Added beforeTest closure to get notified before a test is run. The closure simply logs the test descriptor which * indicates the test method that is being executed. Added to help find the test execution order differences between two * test runs or even differences between two systems like local, ci etc. */ test { beforeTest { descriptor -> logger.lifecycle("Running test: " + descriptor) } } integrationTest { beforeTest { descriptor -> logger.lifecycle("Running test: " + descriptor) } }

This will print as shown below, for instance when unit tests are run with the command: grails test-app -unit

:compileJava UP-TO-DATE :compileGroovy :buildProperties :processResources :classes :compileTestJava UP-TO-DATE :compileTestGroovy :processTestResources UP-TO-DATE :testClasses :test Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=2048m; support was removed in 8.0 Running test: Test testIndex(myapp.FirstTestSpec) Running test: Test testList(myapp.FirstTestSpec) Running test: Test testCreate(myapp.FirstTestSpec) Running test: Test testList(myapp.SecondTestSpec) Running test: Test testCreate(myapp.SecondTestSpec) Running test: Test testIndex(myapp.ThirdTestSpec) . . .

Ideally the order in which test cases run should not be a concern at all. But when functional tests and integration tests run together as integration tests, there are high chances of functional tests and integration tests getting mixed up in the sequence of running integration tests, and thus polluting integration tests. So, knowing the order in which test cases run will help solving this problem.

After finding out the issue, I refactored test cases under integration-test/groovy/myapp directory and separated out functional from integration tests into two different folders/packages (myapp.integration and myapp.functional) and even separated out their executions by running unit, integration and functional in 3 steps instead of running all in one step (grails test-app) as follows:

grails -Dgrails.env=development test-app -unit grails -Dgrails.env=development test-app myapp.integration.* -integration grails -Dgrails.env=development test-app myapp.functional.* -integration

This will guarantee that my functional tests which are expected to leave data in the database after run (as they were written) are run as the last group of tests in a bit controlled manner.

References

Gradle Test task documentation