Create a Secured Restful API App with Grails 3 and PostgreSQL - Step by Step: Part 5 of 5

Previous posts on this topic:

Part 1: Create and run basic Grails Restful app  

Part 2: Add Core Security and REST API Security
Part 3: Add PostgreSQL and start coding
Part 4: Secure end-points fully and cleanly

Part 5: Assure REST & Publish your API

At the end of my last post, we had a RESTful application with an end-point fully implemented and secured. However, we had not written any unit/integration test-specs. In this post we will write integration test-spec and mix it with REST Assured and Spring REST Docs to not only test the end-point but also to generate and publish API documents.

Importance of API Documentation

API is the middleware now. By following standard principles, the behavior of API can be consistent and predictable. Like any piece of code, APIs must be tested by all means from writing test-cases to assuring it's quality by QA. Ideally, when the application is assembled, API docs must also be generated and bundled into the artifact and be delivered together for deployment. That makes API docs a "living source of documents" and they become a common source of reference.

Spring REST Docs

There are few popular frameworks/tools to generate API documentation. Each one of such frameworks/tools has it's own adopted approach and comes with it's own benefits and drawbacks when compared with others. However, Spring IO has a project called Spring REST Docs that uniquely takes a very different approach. It's approach is centered around testing and is combined with hand-written Asciidoctor templates to produce high quality and maintainable API documentation. This approach definitely stands out as it promotes testing to it's greatest levels.

With the test-centric approach, it makes API document not only accurate-and-complete but also up-to-date and a living-resource reference. API documents generated this way are always as accurate as the code-base is. Also, as Spring framework is central to Grails framework, it becomes a natural-fit for Grails applications.

Having said all that, let's add Spring REST Docs to Grails 3 project and assure it with REST Assured.

Environment: Grails 3.2.11, Java 1.8, Apache Tomcat 8.0.20, IntelliJ IDEA Ultimate 2017.2 on Mac OS X 10.11.5 (El Capitan)

Step 0: Upgrade application from Grails 3.1.6 to 3.2.11

When I started this multi-part posts, Grails was at 3.1.6 and now it has advanced to 3.3.x. Just to catch up, I've upgraded this app from 3.1.6 to 3.2.11 (the latest on 3.2.x branch). It was an easy upgrade as it is a simple RESTful application. All I had to do was to bring gradle.properties and build.gradle files up-to-date with 3.2.11.

Step 1: Add Spring REST Docs to the Project (build configuration)

At the end of my last post, we had a secured resource (Artist) and we tested it's RESTful API for CRUD operations. That is good enough resource for taking it to the next level of generating & publishing it's API. Spring REST Docs' Getting Started has link to sample applications for reference. REST Assured Grails is the best-bet and is the basis for us. As a first step let's add Spring REST Docs support to the project as shown and described below:

Add Asciidoctor plugin.
build.gradle

plugins { ... id 'org.asciidoctor.convert' version '1.5.3' }

Run gradle tasks command and notice that asciidoctor task gets added by the plugin.

$./gradlew tasks ... Documentation tasks ------------------- asciidoctor - Converts AsciiDoc files and copies the output files and related resources to the build directory. groovydoc - Generates Groovydoc API documentation for the main source code. ... javadoc - Generates Javadoc API documentation for the main source code.

Spring REST Docs Build Configuration section has steps for Gradle build configuration. I will do this slightly different to extend build script for REST docs support by separating out additional build configuration for REST docs into it's own build file leveraging Gradle's script plugin concept. This way it is more cleaner and brings in some modularity to the build script.

Create a new build script file restdocs.gradle under project's gradle dir and reference it in the main build.gradle file at the very bottom as shown below:
build.gradle

apply from: 'gradle/restdocs.gradle'

Let's populate restdocs.gradle as shown below. I will add comments into the build script to explain certain code blocks.
gradle/restdocs.gradle

buildscript { repositories { maven { url 'https://repo.spring.io/libs-snapshot' } } } repositories { maven { url 'https://repo.spring.io/libs-snapshot' } } //add extra user-defined properties to the project through ext block ext { snippetsDir = file('build/docs/generated-snippets') //output dir of rest api doc snippets generated restDocsVersion = '2.0.0.BUILD-SNAPSHOT' restAssuredVersion = '2.9.0' } dependencies { testCompile "org.springframework.restdocs:spring-restdocs-core:$restDocsVersion" testCompile "org.springframework.restdocs:spring-restdocs-restassured:$restDocsVersion" testCompile "org.springframework.restdocs:spring-restdocs-asciidoctor:$restDocsVersion" }

Now, just run grails clean command. We will have spring-restdocs-core and spring-restdocs-restassured downloaded from maven central repo.

Let's keep expanding this script.

//task to clean generated rest api docs snippets dir task cleanSnippetsDir(type: Delete){ delete fileTree(dir: snippetsDir) }

Run ./gradlew tasks and notice that there is a new task added under Other tasks like:

Other tasks ----------- cleanIdeaWorkspace cleanSnippetsDir console

Configure test task as shown below:

test { dependsOn cleanSnippetsDir outputs.dir snippetsDir }

Now run ./gradlew test -m or ./gradew test --dry-run which will run gradle's test task in a dry run mode. It disables all tasks and shows the order in which tasks get executed. In this case, we can now see our new task cleanSnippetsDir in the list after all classes are created and before test-case classes get compiled.

Remember we got asciidoctor task by adding Gralde plugin as the very first step. We will customize it and specify that it depends on integrationTest task. With this dependency, every time when we run this task, it will have integration tests run. We want this kind of dependency as the approach that REST Docs brings in is to have REST API docs generated from the integration test cases. So, we need integration tests to run before we have docs generated.

Having said that, let's customize that task as follows:

//Configure asciidoctor task provided by Gradle asciidoctor plugin- https://github.com/asciidoctor/asciidoctor-gradle-plugin asciidoctor { doFirst{ //just print outputDir for reference during execution phase println "Running asccidoctor task. Check generated REST docs under: ${outputDir}" } dependsOn integrationTest logDocuments = true sourceDir = file('src/docs') inputs.dir snippetsDir separateOutputDirs = false attributes 'snippets': snippetsDir //configure snippets attribute for .adoc files }

Step 2: Run tests and make them pass: grails test-app

Grails test-app runs both unit tests and integration tests.
I have not written any test specifications so far but as part of creating domain objects using grails create-domain-class command, I have a few Spock Specification unit-tests created each with a default feature method "test something"(). All these default generated specifications are expected to fail to start with. I want to keep these tests around for future but want to make them pass. An easy way is to annotate all those methods with groovy's @NotYetImplemented annotation. It reverses the net result by making it pass when it actually fails. It makes sense for an un-implemented test. But when actually implemented, it fails forcing us to remove the annotation.

Spock's @PendingFeature is similar but is added only in Spock 1.1. Grails 3.2.x comes with Spock 1.0. For now, we are all good with that wonderful annotation provided by groovy. With this we have all unit-tests passing.

It's time now to write an integration test specification for our RESTful controller: ArtistController. Instead of writing a typical integration test-case, let's mix it with REST assured and REST API Docs and get both testing and API docs generation done in this phase.

Step 3: Assure REST by writing integration specification for RESTful Controller with a mix of REST Docs and a touch of REST assured.

Step 3a: Configure REST Assured testing framework (set up your test specification to generate documentation snippets)

The Spring REST Docs documentation has outlined these steps. Here is the gist of it:

The configuration of REST Assured is nothing but a request spec (RequestSpecifiction) using ResqusetSpecBuilder by adding documentation configuration as a JUnit filter to it.

Configure REST assured documentation output directory by declaring a restDocumentation field which is initialized with an instance of JUnitRestDocumentation and annotate it with JUnit's @Rule annotation. This rule gets executed before and after each feature method. A custom output directory can be specified by passing a constructor argument. We specify this custom dir, as in the build file, the snippetsDir property we set with is slightly different ('build/docs/generated-snippets') than the default ('build/generated-snippets').

Next, setup RequestSpecification by adding a filter and configure it with the above restDocumentation initialized as JUnit Rule.

Here is how our test spec looks after this configuration:
src/integration-test/groovy/com/giri/ApiDocumentationArtistSpec

package com.giri import geb.spock.GebSpec import grails.plugins.rest.client.RestBuilder import grails.test.mixin.integration.Integration import grails.transaction.Rollback import io.restassured.builder.RequestSpecBuilder import io.restassured.specification.RequestSpecification import org.junit.Rule import org.springframework.restdocs.JUnitRestDocumentation import static org.springframework.http.HttpStatus.* import static org.springframework.restdocs.restassured3.RestAssuredRestDocumentation.documentationConfiguration @Integration @Rollback class ApiDocumentationArtistSpec extends GebSpec { @Rule protected JUnitRestDocumentation restDocumentation = new JUnitRestDocumentation('build/docs/generated-snippets') private RequestSpecification documentationSpec def setup() { //set documentation specification this.documentationSpec = new RequestSpecBuilder().addFilter( documentationConfiguration(this.restDocumentation)) .build() } ...

Step 3b: Spockify, test RESTful end-point and get documentation snippets generated

With the above configuration, let's write a feature method to test GET request of /api/artists end-point. The following is a feature method added to the above specification along with a defined static constant whose value is set with relative end-point url and an injected application port property. The port is required to override the default port(8080) of REST assured testing framework. Note that grails start the application on a random available port each time when integration tests are run.

static final String ARTISTS_ENDPOINT = '/api/artists' @Value('${local.server.port}') protected int port ... void "test and document GET request (index action) of end-point: /api/artists"() { given: "" RequestSpecification requestSpecification = RestAssured.given(this.documentationSpec) .accept(MediaType.APPLICATION_JSON_VALUE) .filter( RestAssuredRestDocumentation.document( 'artists-list-example' ) ) when: def response = requestSpecification .when() .port(port) .get(ARTISTS_ENDPOINT) then: response.then() .assertThat() .statusCode(HttpStatus.OK.value()) }

The feature method name describes the intent of this feature method. In this step, we are only testing the GET request of an end-point. We will add the support for the highlighted and document intent of this feature.

With this, if you run grails dev test-app or grails -Dgrails.env=development test-app, the test will pass. Also, we will have the following six documentation snippets generated under build/docs/generated-snippets/artists-list-example directory:
 curl-request.adoc
 http-request.adoc
 http-response.adoc
 httpie-request.adoc
 request-body.adoc
 response-body.adoc

These are the snippet files to be included in the final API documentation. Just to see the contents check http-response.adoc and it will contain the actual response received as follows:

---- HTTP/1.1 200 OK X-Application-Context: application:development:0 Content-Type: application/json;charset=UTF-8 Transfer-Encoding: chunked Date: Tue, 29 Aug 2017 22:12:13 GMT Content-Length: 148 [{"id":"90ff9ac4-b1c0-4495-94d5-1550f463561a","dateCreated":"08/29/2017","firstName":"Giridhar","lastName":"Pottepalem","lastUpdated":"08/29/2017"}] ----

Step 3c: Create asciidoctor (.adoc) source templates
Create src/docs dir and create api-guide.adoc and artists.adoc files to start with. The api-guide.adoc is the main asciidoctor template which will include artists.adoc. The artists.adoc is the asciidoctor template for artists end-point.

Shown below is a portion of api-guide.adoc

= giri-api RESTful API Guide Giridhar Pottepalem :doctype: book :icons: font :source-highlighter: highlightjs :toc: left :toclevels: 4 :sectlinks: [[overview]] = Overview [[overview-http-verbs]] == HTTP Methods giri-api API follows standard HTTP and REST conventions as closely as possible in its exposure of resources as end-points and use of HTTP methods (verbs). ... [[resources]] = Resources include::artists.adoc[]

And portions of artists.adoc is shown below for creating an Artist (POST request/save action):

[[resources-artists]] == Artists An Artist is a resource which represents an Artist. [[resources-artists-create]] === Creating an Artist A `POST` request is used to create a new Artist. TIP: An Artist can be created only by an Admin user (with role `ROLE_ADMIN`) IMPORTANT: Once a new Artist is created... ==== Request structure include::{snippets}/artists-create-example/request-fields.adoc[] ==== Example request include::{snippets}/artists-create-example/curl-request.adoc[] ==== Response structure include::{snippets}/artists-create-example/response-fields.adoc[] ==== Example response include::{snippets}/artists-create-example/http-response.adoc[]

Highlighted are the references to generated snippets that get included in the generated end HTML5 doc.

Step 3d: Generate API doc
Now, lets run asciidoctor gradle task we got added through Step 1 as shown below:
./gradlew asciidoctor //runs in test env
./gradlew -Dgrails.env=development asciidoctor //runs in dev env

This task runs all integration test specifications because we configured it to depend on integrationTest task. Once it's run successfully with no failing tests, it converts our asciidoctor API templates to HTML5 doc by populating it with included generated snippets as we referenced in artists.adoc.

Now let's enhance our specification feature method to document request and response payload structure. Let's take the case of /api/artists end-point and GET request. There is no request payload for this request. So, we will simply add response payload specification as shown below:

void "Test and document show Artist request (GET request, show action) to end-point: /api/artists"() { given: "Pick an artist to show" Artist artist = Artist.first() and: "user logs in by a POST request to end-point: /api/login" String accessToken = authenticateUser('me', 'password') and: "documentation specification for showing an Artist" RequestSpecification requestSpecification = RestAssured.given(this.documentationSpec) .accept(MediaType.APPLICATION_JSON_VALUE) .filter( RestAssuredRestDocumentation.document( 'artists-retrieve-specific-example', PayloadDocumentation.responseFields( PayloadDocumentation.fieldWithPath('id').type(JsonFieldType.STRING).description('Artist id'), PayloadDocumentation.fieldWithPath('firstName').type(JsonFieldType.STRING).description('Artist first name'), PayloadDocumentation.fieldWithPath('lastName').type(JsonFieldType.STRING).description('Artist last name'), PayloadDocumentation.fieldWithPath('dateCreated').type(JsonFieldType.STRING).description("Date Created (format:MM/dd/yyyy)"), PayloadDocumentation.fieldWithPath('lastUpdated').type(JsonFieldType.STRING).description("Last Updated Date (format:MM/dd/yyyy)") ) ) ) when: "GET request is sent" def response = requestSpecification .header("X-Auth-Token", "${accessToken}") .when() .port(this.port) .get("${ARTISTS_ENDPOINT}/${artist.id}") def responseJson = new JsonSlurper().parseText(response.body().asString()) then: "The response is correct" response.then() .assertThat() .statusCode(HttpStatus.OK.value()) and: "response contains the id of Artist asked for" responseJson.id }

Similarly, we can write a test spec to test and document POST method (creating an Artist) as shown below. Remember, I have secured this method to role: ROLE_ADIN. So, it requires admin to be authenticated first to get a security token and then pass the security token in the subsequent secured requests like POST. The following is the complete test specification with a helper method added to authenticate the user:

/** * Helper method, authenticates the given user and returns the security token. * * @param username the user id * @param password the password * @return security token once successfully authenticated */ protected String authenticateUser(String username, String password) { String authResponse = RestAssured.given() .accept(MediaType.APPLICATION_JSON_VALUE) .contentType(MediaType.APPLICATION_JSON_VALUE) .body(""" {"username" : "$username", "password" : "$password"} """) .when() .port(this.port) .post(LOGIN_ENDPOINT) .body() .asString() return new JsonSlurper().parseText(authResponse).'access_token' } void "Test and document create Artist request (POST request, save action) to end-point: /api/artists"() { given: "current number of Artists" int nArtists = Artist.count() and: "admin logs in by a POST request to end-point: /api/login" String accessToken = authenticateUser('admin', 'admin') and: "documentation specification for creating an Artist" RequestSpecification requestSpecification = RestAssured.given(this.documentationSpec) .accept(MediaType.APPLICATION_JSON_VALUE) .contentType(MediaType.APPLICATION_JSON_VALUE) .filter( RestAssuredRestDocumentation.document( 'artists-create-example', PayloadDocumentation.requestFields( PayloadDocumentation.fieldWithPath('firstName').description('Artist first name'), PayloadDocumentation.fieldWithPath('lastName').description('Artist last name') ), PayloadDocumentation.responseFields( PayloadDocumentation.fieldWithPath('id').type(JsonFieldType.STRING).description('Artist id'), PayloadDocumentation.fieldWithPath('firstName').type(JsonFieldType.STRING).description('Artist first name'), PayloadDocumentation.fieldWithPath('lastName').type(JsonFieldType.STRING).description('Artist last name'), PayloadDocumentation.fieldWithPath('dateCreated').type(JsonFieldType.STRING).description("Date Created (format:MM/dd/yyyy)"), PayloadDocumentation.fieldWithPath('lastUpdated').type(JsonFieldType.STRING).description("Last Updated Date (format:MM/dd/yyyy)") ) ) ) when: "POST request is sent with valid data" def response = requestSpecification .header("X-Auth-Token", "${accessToken}") .body("""{ "firstName" : "Bhuvan", "lastName" : "Pottepalem" }""") .when() .port(this.port) .post(ARTISTS_ENDPOINT) def responseJson = new JsonSlurper().parseText(response.body().asString()) then: "The response is correct" response.then() .assertThat() .statusCode(HttpStatus.CREATED.value()) and: "response contains the id of Artist created" responseJson.id and: "Number of Artists in the system goes up by one" Artist.count() == nArtists + 1 }

Now, simply run
./gradlew asciidoctor

We will have API docs generated under build/asciidoc dir. Open api-guide.html in a browser to see how nicely the generated API doc looks.

TIP: The beauty of Spring REST Docs framework is that, if compares the actual request/response fields with the PayloadDocumentation filed descriptions and will fail the test if any field(s) are missed or mis-matched. This ensures that the API documentation is up-to-date with the implementation.

Step 4: Publish API

Now, we have fully integrated REST Assured and Spring REST Docs into integrationTest phase with an added asciidoctor Gradle test task. The result of this is an up-to-date API document generated for our Restful service.The API document is the source for clients using this service. So, it needs to be made available. One way to achieve this is to bundle the generated HTML5 API docs with the application's deployable war or executablejar and have it's own end-point to serve it.

Spring Boot (the framework Grails3 underpins) can be leveraged to achieve this. By default Boot serves static content placed under /static or /public in the class path or root of the application context. Here is the link for reference: Spring boot Static content.

Step 4a: Bundle API documentation into deployable artifact
We will enhance our build script (restdocs.gradle) and customize war task that comes with Gradle Java plugin little bit to achieve this. Below is the code snippet which is self explanatory:

/* Bundles generated API docs into war file. * Spring boot serves static content under /public or /static or /resources or /META-INF/resources. * Hooks into war task and adds asciidoctor task dependency, also copies generaed rest docs appropriately * for bundling into war file. */ def publicDocsDir = 'WEB-INF/classes/public/docs' war { dependsOn asciidoctor from ("${asciidoctor.outputDir}") { into publicDocsDir } }

We basically made war task depend on asciidoctor task and added a step to copy generated HTML5 API docs to WEB-INF/classes/public/docs dir in the generated war file.

Now, run grails war to generate deployable war artifact:
grails war

You can explode and see that generated API docs are bundled into the war generated (giri-api-0.1.war):
e.g. jar tvf build/libs/giri-api-0.1.war | grep html will list the following:
 59738 Mon Sep 04 07:16:34 EDT 2017 WEB-INF/classes/public/docs/api-guide.html
 47974 Mon Sep 04 07:16:34 EDT 2017 WEB-INF/classes/public/docs/artists.html

Step 4b: Make API documentation available from it's own end-point
Deploy the generated war file onto a locally running tomcat.
Deploy the war onto locally running Tomcat and point your browser at: http://localhost:8080/giri-api-0.1/static/docs/api-guide.html

This will result into Access Denied error. We need to open up security to serve API docs.

Lets change application.groovy and add /static/docs/** to both grails.plugin.springsecurity.controllerAnnotations.staticRules and filterChainChainMaps as shown below:

grails.plugin.springsecurity.controllerAnnotations.staticRules = [ ... [pattern: '/static/docs/**', access:['permitAll']] ] def filterChainChainMaps = [ ... pattern: '/static/docs/**', filters: statelessFilters], ... ]

Create a war file. Undeploy previously deployed war and deploy the latest war file.

Now, http://localhost:8080/giri-api-0.1/static/docs/api-guide.html (API docs) should be served and displayed by the app.

The test specification can be enhanced easily along these lines to test and document rest of the service methods: show, update and delete available for /api/artists end-point through HTTP methods GET specific resource by id, UPDATE and DELETE respectively.

The complete source code is hosted on GitHub at https://github.com/gpottepalem/giri-api for reference.

References

• Spring REST Docs Reference documentation
• Gradle commands Line usage
• Spring REST Docs Sample applications
• Github - rest assured
• Asciidoctor