Grails JMS plugin 1.2 hands off messages too early during the plugin relay of application startup . . .

Rio 2016 Olympics concluded recently with the traditional last event of track and field: 4x100 meter relay. In relay, the teams that qualify are the ones that gracefully complete the race. This involves a great co-ordination among team runners. One important rule of this particular race is handing off the baton within a changeover box. Dropping the baton or failing to transfer within the box disqualifies the team. The baton has to be carried till finish line in the race. It is a synchronous race.

Messaging in software applications is like a relay except that it is asynchronous. It is a great way to integrate modern applications asynchronously achieving great performance and scalability with guaranteed delivery of messages. But it coms with certain complexities in handling messages gracefully. Java based applications using JMS API are no exceptions. Spring framework's JMS support greatly simplifies the use of JMS. Grails JMS plugin which underpins Spring JMS makes it even better.

Recently I had to investigate and put a fix for a known issue of Grails JMS plugin that we used in our Grails applications. The known issue is- Messages get queued up when the application is down (expected behavior, of course). But when the app comes up, listeners grab messages too early from queues before the application is fully started. If message processing involves accessing domain model or database, they error out forcing you to deal with this situation.

We have two Grails applications integrated asynchronously through JMS messaging with ActiveMQ as the underlying JMS implementation. Messages flow back and forth between two applications. The most recent app of the two is a RESTful API app developed using Grails 3.1.6 rest-api profile and JMS plugin ver 2.0.0.M1 whereas the other application is a bit older Grails web application using Grails 2.2.1 and Grails JMS plugin 1.2. The Grails 3.1.6 RESTful API app is a gateway for client web/mobile apps that exposes resources in RESTful way and hands off requests to the Grails 2.2.1 web app for further processing. The communication channel between these two apps is JMS through queues. So, it was important to make sure we don't lose any messages either side due to any application down times. During testing everything on the Grails 3.1.6 side looked good. After it came up, it successfully processed messages that were waiting in the queues during it's down time. The other Grails 2.2.1 app simply errored out with t's message processing during a similar situation when it came back up after being down when the messages arrived and waiting in the queues. The following is a high-lvil architectural diagram depicting this.

Applications integrated through JMS

Known Issue with Grails JMS plugin 1.2

As the application starts up, listener containers get started bit early in the plugin race, earlier than dataSource and Hibernate are fully up and ready. Hence, if message processing involves domain-model access or db access, it errors out.

Options tried

Upgrading the plugin to ver 1.3

The plugin version 1,3 seemed to have a fix for this issue when I looked into plugin's GitHub source code  repository. The fix seemed convincing and pretty simple from the source code change. There were three more plugin dependencies added to the loadAfter list of plugins: 'dataSource', 'hibernate', and 'hibernate4'.

Steps to upgrade plugin:

1. Change the dependency in BuildConfig.groovy from compile ":jms:1.2" to compile ":jms:1.3"

plugins { ... compile ":jms:1.3" ... }

2. Refresh dependencies using the following command:

grails -Dgrails.env=<your-env> refresh-dependencies

3. Grails build system prompts as below for a confirmation:

> You currently already have a version of the plugin installed [jms-1.2]. Do you want to update to [jms-1.3]? [y,n]

Press y to upgrade.

However, with this upgrade the issue became bit worse and all listeners completely stopped listening to messages. Not only the messages that were waiting in the queue when the app was down, but also any new messages that come after the application comes up were not picked up by application listeners. After looking at the source code, JmsGrailsPlugin.groovy and comparing it with that of 1.2 plugin, noticed that the method startListenerContainer which starts up all listener container beans was totally missing. 

Upgrading the plugin to 1.3-SNAPSHOT

Then I tried 1.3-SNAPSHOT version by following the above steps to update the plugin and it's  dependencies. This ran into spring ClassNotFoundException (java.lang.NoClassDefFoundError: org/springframework/core/type/classreading/AnnotationMetadataReadingVisitor) and the application wouldn't even come up. When looked into the plugin source code it had the three additional plugin dependencies added to the list loadAfter, and the method startListenerContainer was was also in place. When I compared this source with version to 1.2, the only change I noticed was the additional 3 plugins in the list for loadAfter.

That was puzzling to me and I read Grails documentation carefully word by word to understand how this loadAfter works. The sentence with an example: Here the plugin will be loaded after the controllers plugin if it exists, otherwise it will just be loaded caught my attention. Then I went back to check and see if we had hibernate4 plugin. Apparently, we didn't have that and as hibenate4 plugin was for Grails 2.5.0 or higher. That gave me a clue.

The fix

Having looked at 1.3, 1.3-SNAPSHOT and 1.2 and since we had 1.2 plugin checked into our source repository along with the application's source code, we decided to edit plugin code by adding just 2 dependencies: 'dataSource' and 'hibernate' to the list loadAfter. That worked beautifully as expected and we decided to go with that fix.

Here is the final fix, changed code in JmsGrailsPlugin.groovy from:

def loadAfter = ['services', 'controllers']

to:

// Load jms plugin after the following plugins. // BEWARE, if any of the listed plugins don't exist, the list is ignored // and this jms plugin gets loaded and executed // Ref: http://docs.grails.org/latest/guide/plugins.html#understandingPluginStructure def loadAfter = ['services', 'controllers', 'dataSource','hibernate']

With this fix, the application once came up, gracefully handled all pending messages that were awaiting when it  was down. Completed the message relay by handing off messages gracefully without dropping or erring out of the plugin execution race.

TIP

If due to any reason if it doesn't work, simply check application log file or by other means find out which plugin is the last one in the loading process and set loadAfter with a list containing that plugin. That should delay pushing this plugin to the end in the plugin loading process ;)

There seems no easy way to know all the plugin dependencies of a Grails project.

Running grails command grails dependency-report will list dependency graph.

If using IntelliJ, either Project Settings > Modules > dependencies,  or Project View > External Libraries or even trying to open a file (Mac: CMD + shift + O) for *GrailsPlugin.groovy should give the list of plugins for the project.

Useful Links & References

• Stack overflow question
• Old plugin repo
• Current plugin repo with the fix
• Grails JMS Plugin docs
• Understanding Grails Plugin loadAfter

Create a Secured Restful API App with Grails 3 and PostgreSQL- Step by Step: Part 3 of 5

Posts on this topic:

Part 1: Create and run basic Grails Restful app  

Part 2: Add Core Security and REST API Security
Part 4: Secure end-points fully and cleanly
Part 5: Assure REST  & Publish your API

Part 3: Add PostgreSQL and start Coding

In Part 2 of this series, I secured my RSET API application with two Grails 3 plugins: Grails Spring Security Core plugin and Grails Spring Security REST plugin. With this, if I start my application by running the command: grails run-app and check http://localhost:8080, I get JSON response containing standard grails application information. When I point my browser at http://localhost:8080/dbconsole I see the embedded H2 db console that grails provides. Grails comes with in-memory H2 database. That is all good for getting started.

Now, lets add PostgreSQL to the application and do some hands-on coding. 

PostgreSQL

PostgreSQL has been gaining momentum lately among developers community and is evolving as a preferred database over Oracle or MySQL. It's also been evolving steadily to offer features of both Relational and NoSQL databases and the best part is: it's open-source.

Download PostgreSQL and install it. Go with all default installation steps. On Mac OS, it gets installed under /Library/PostgreSQL/9.5. In addition, it's good to have a GUI tool to access PostgreSQL db. PgAdmin is one such administration and development GUI tool for PostgreSQL. It's easy to use.

Now, with PostgreSQL installed we are good to do some hands-on.

Environment: Grails 3.1.6, Java 1.8, IntelliJ 15 on Mac OS X 10.9.5

Step 1 Add PostgreSQL support - Add the following run-time dependency in build.gradle file:

dependencies{ ... runtime "org.postgresql:postgresql:9.4.1208.jre7" //Java JDBC 4.1 (JRE 7+) driver for PostgreSQL ... }

Step 2 Edit application.yml, change driverClassname, username, password and url properties of dataSource as shown below (highlighted are the changes):

dataSource: pooled: true jmxExport: true driverClassName: org.postgresql.Driver username: giriapi password: giriapi #dialect = org.hibernate.dialect.PostgreSQLDialect environments: development: dataSource: dbCreate: create-drop url: jdbc:postgresql://localhost:5432/giriapi test: dataSource: dbCreate: update url: jdbc:postgresql://localhost:5432/giriapi production: dataSource: dbCreate: update url: jdbc:postgresql://localhost:5432/giriapi

By making the above changes we changed from Grails default H2 database to PostgreSQL.

Now prepare PostgreSQL database for our application. The following 2 steps show minimal needed: 1) a user role and 2) a database.

Step 3 Create a user role (e.g. giriapi), I used pg Admin III postgres client tool to do this. Below are the screenshot:

Click Definition tab and
enter password for the user

Step 4 Create database (e.g. giriapi) and select the user (giriapi) as the owner created in the above step as shown in the screenshot below:

Step 5 Run the app (command: grails run-app) and check database. Now, I have 4 tables created and two users created under public Schemas in PostgreSQL 9.5 (localhost:5432) database as shown in the screenshot below:

With that we are be all set to get going with PostgreSQL.

Leverage Additional PostgrSQL capabilities

As this is a RESTFul application mostly dealing resources' state represented and transferred as JSON data, it makes sense to leverage JSON capabilities offered by PostgreSQL database. Also, out-of-the-box Grails uses surrogate keys for all tables and strategy for id generation is sequence. PostgreSQL also has good support for UUIDs and I would like to leverage that capability as well. For PostgreSQL UUID native datatype we don't need any extra plugins, but for JSON and many other useful PostgreSQL native types, the Grails Postgresql Extensions Plugin provides hibernate user types. Let's add this plugin to the application now.

Step 6 Add Grails Postgresql Extensions plugin. Make the following changes in build.gradle

Add jcenter repository and plugin dependency

repositories { ... jcenter() //Bintray JCenter repo for grails postgres-extensions plugin ... } dependencies { ... //grails postgresql extensions plugin for JSON type compile 'org.grails.plugins:postgresql-extensions:4.6.6' ... }

*You can delete runtime "com.h2database:h2" dependency as we no longer use H2 in-memory database.

Also, make the following change in application.yml to change the Hibernate dialect (these steps are pretty well documented in the PostgreSQL extensions plugin documentation)

hibernate: dialect: net.kaleidos.hibernate.PostgresqlExtensionsDialect #Grails Postgresql Extensions plugin

Step 7 Now we are all set with Postgresql and it's extension plugin. Now, let's add domain model to the application.

This is the Grails approach that I liked most right from the beginning. Typically, in Web applications using Spring MVC, Hibernate ORM, many projects start either top-down from the front-end creating views and controllers, or bottom-up from the back-end designing database tables & mapping tables to domain objects. Grails makes you start with domain model, from the middle.

Domain model is the core model behind your application and generally is mapped to database. It is an important aspect of any application that needs bit more design considerations that any other aspects. The properties, relationships, data validations etc. must be considered while designing domain model. Once you get the model rightly done and have in place, it becomes the middle ground for your Grails app. Grails will take it further and expand it's reach to both sides- front-end side: the view part of MVC and back-end side: the model part with persistence support.

For this application, I chose my primary hobby Painting as the domain. Like good developers, Artists also like simplicity. So, I kept this model very simple, simple enough to cover all aspects of Grails features that I wanted to explore. The interests of an Artist's small world can be represented in a simple domain model as follows:

Domain Objects

• Artist, ArtWork, Specification
• AppUser, Role, AppUserRole, AuthenticationToken - domain objects already in place for security

Relationships

• An Artist can own many ArtWorks (one-to-many)
• Each Artwork has a Specification (one-to-one)

Step 8 Generate domain classes

Grails comes with pretty good command-line support for both build tasks and code-generation. The command line works in both interactive and non-interactive mode. To start grails in interactive mode, simply run grails command from the project home dir. The interactive mode supports TAB completion for all command. For instance if you are not sure which command to use for creating a domain class, simply typing cr + TAB which lists all create commands available.

I prefer  command line in interactive mode for generating all my Grails artifacts. You don't have to use command line, you can simply create groovy classes in respective folders like domain, controllers, services etc. But I prefer creating my Grails artifacts through command line create* commands as it also generates associated unit/integration test cases. Of course, it generates just skeleton classes, but it's better than creating manually to make sure that test cases are not left behind.

For creating domain class there are two commands: create-domain-class, create-domain-resource. The difference between these two is very subtle. The create-domain-resource command generates domain object annotated with @Resource making the domain object a RESTful resource.

When generating domain classes using these commands, you typically need to specify the name of your Domain class including the package name. But if you have grails.codegen.default.package property set with default package name (I have it set to com.giri) in either application.yml or application.groovy with the base package name, you can ignore the package name. Grails uses the default package name configured.

That's enough insights into this Grails domain object creation. Let's create domain objects as shown below with grails running in interactive mode.

$ grails Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=2048m; support was removed in 8.0 BUILD SUCCESSFUL | Enter a command name to run. Use TAB for completion: grails> grails> create-domain-resource Artist | Created grails-app/domain/com/giri/Artist.groovy | Created src/test/groovy/com/giri/ArtistSpec.groovy grails> create-domain-resource ArtWork | Created grails-app/domain/com/giri/ArtWork.groovy | Created src/test/groovy/com/giri/ArtWorkSpec.groovy grails> create-domain-resource Specification | Created grails-app/domain/com/giri/Specification.groovy | Created src/test/groovy/com/giri/SpecificationSpec.groovy grails>

Highlighted are the commands used to create domain classes. Also, notice that for every domin class it also generates an associated unit testcase.

For instance, the generated domain class Artist looks like:

package com.giri import grails.rest.* @Resource(readOnly = false, formats = ['json', 'xml']) class Artist { }

We can start adding properties, constraints and relationships to these objects. Grails implicitly provides id, version properties. If two Date properties by name dareCreated, lastUpdated are added, Grails takes care of populating them appropriately when the object is created and updated.

The @Resource is the easiest way to create a RESTful resource, in other words, it is the easiest way to expose a domain object as a REST resource. Once you annotate your domain object as a Resource, you don't need to do anything. Grails provides needed controller and URLMappings in RESTful way.

Gotcha
The generated domain class annotated with @Resource is missing uri attribute, without that Grails will not expose your domain object as RESTful resource and when you run grails url-mappings-report command, you will not have RESTful end-points listed for the domain object. So, don't forget to define end-point path through uri property of @Resource annotation for the domain object like:

@Resource(uri = '/api/artists', readOnly = false, formats = ['json', 'xml'])

I have also added few basic properties to Artist domain object including and id of type UUID. Now, Artist domain object looks like:

package com.giri import grails.rest.Resource /** * @author Gpottepalem * Created on Aug 13, 2016 */ @Resource(uri = '/api/artists', readOnly = false, formats = ['json', 'xml']) class Artist { UUID id String firstName String lastName Date dateCreated Date lastUpdated static constraints = { id type: 'pg-uuid', generator: 'uuid2' firstName blank: false lastName blank: false } }

Highlighted is the explicitly defined id property and it's mapping for UUID, if you don't need id to be UUID, you can simply omit it and Grails provides an implicit long id mapped to appropriate db type.

If you run grails url-mappings-report, the output looks like:

Dynamic Mappings | * | ERROR: 500 | View: /error | | * | ERROR: 404 | View: /notFound | | * | /${controller}/${action}?/${id}?(.${format)? | Action: (default action)| Controller: application | * | / | Action: index | Controller: artist | GET | /api/artists/create | Action: create | | GET | /api/artists/${id}/edit | Action: edit | | POST | /api/artists | Action: save | | GET | /api/artists | Action: index | | DELETE | /api/artists/${id} | Action: delete | | PATCH | /api/artists/${id} | Action: patch | | PUT | /api/artists/${id} | Action: update | | GET | /api/artists/${id} | Action: show | Controller: restOauth | * | /oauth/access_token | Action: accessTok | | * | /oauth/${action}/${provider} | Action: (default action)| BUILD SUCCESSFUL

Now, let's start our application and try to access end-point /api/artists for getting the list of Artists.
As we have secured all our end-points /api/**, when you try to get list of Artists by issuing a curl command like this:

$ curl -i -X GET 'http://localhost:8080/api/artists'

We get a Forbidden response.

HTTP/1.1 403 Forbidden Server: Apache-Coyote/1.1 Content-Type: application/json;charset=UTF-8 Transfer-Encoding: chunked Date: Sun, 14 Aug 2016 16:13:49 GMT {"timestamp":1471191229379,"status":403,"error":"Forbidden","message":"Access Denied","path":"/api/artists"}

Step 9 The reason for Forbidden in the above step is because we have secured all /api/** end-points. We need to fix this. To start simple, we allow everyone to access /api/artists end-point. For this, all we need to do is:

• Annotate our Artists domain object with @Secured(['permitAll]) and
• add anonymousAuthenticationFilter

After adding the anonymousAuthenticationFilter, application.groovy looks like:

grails.plugin.springsecurity.filterChain.chainMap = [ //Stateless chain // [ pattern: '/api/**', filters: 'JOINED_FILTERS,-anonymousAuthenticationFilter,-exceptionTranslationFilter,-authenticationProcessingFilter,-securityContextPersistenceFilter,-rememberMeAuthenticationFilter'], [ pattern: '/api/**', filters: 'JOINED_FILTERS,-exceptionTranslationFilter,-authenticationProcessingFilter,-securityContextPersistenceFilter,-rememberMeAuthenticationFilter'],

All I did was, I removed -anonymousAuthenticationFilter from the stateless filter chain. This adds it back to the chain.

Now, if I try curl command to list artists:

$ curl -i -X GET 'http://localhost:8080/api/artists'

I get the following response (notice that the list is empty):

HTTP/1.1 200 OK Server: Apache-Coyote/1.1 X-Application-Context: application:development Content-Type: application/json;charset=UTF-8 Transfer-Encoding: chunked Date: Sun, 14 Aug 2016 16:37:33 GMT []

Since I allowed all to access this end-point, I don't need to login. Let's try a POST request to the end-point /api/artists to create an Artist.

curl -i -X POST -H "Content-Type:application/json" -d '{ "firstName": "Giri", "lastName": "Pottepalem" }' 'http://localhost:8080/api/artists'

I get the following response:

HTTP/1.1 201 Created Server: Apache-Coyote/1.1 X-Application-Context: application:development Location: http://localhost:8080/api/artists/ae5ff974-12e9-418a-b397-3f7cd104ea60 Content-Type: application/json;charset=UTF-8 Transfer-Encoding: chunked Date: Sun, 14 Aug 2016 16:53:20 GMT {"id":{"class":"java.util.UUID","leastSignificantBits":-5505862214121887136,"mostSignificantBits":-5881708311393779318},"dateCreated":"2016-08-14T16:53:20Z","firstName":"Giri","lastName":"Pottepalem","lastUpdated":"2016-08-14T16:53:20Z"}

Now, let's try GET command to get the list:

curl -i -X GET 'http://localhost:8080/api/artists'

I get the following response:

HTTP/1.1 200 OK Server: Apache-Coyote/1.1 X-Application-Context: application:development Content-Type: application/json;charset=UTF-8 Transfer-Encoding: chunked Date: Sun, 14 Aug 2016 16:55:07 GMT [{"id":{"class":"java.util.UUID","leastSignificantBits":-8933569886755140856,"mostSignificantBits":2853479693881265650},"dateCreated":"2016-08-14T16:44:36Z","firstName":"Giri","lastName":"Pottepalem","lastUpdated":"2016-08-14T16:44:36Z"}]

Notice, that id which is of type UUID is bit more elaborate than expected. Also, version is not listed. The Date fields dateCreated and lastUpdated are in the format that's not human friendly.

In the next post, I will customize the response and secure some of the end points.

References

Documentations

Grails Postgresql Extensions Plugin Documentation
Grails Command Line