Tuesday, September 05, 2017

Create a Secured Restful API App with Grails 3, PostgreSQL and JMS - Step by Step: Part 5 of 5

Part 5: Assure REST & Publish your API

At the end of my last post, we had a RESTful application with an end-point fully implemented and secured. However, we had not written any unit/integration test-specs. In this post we will write integration test-spec and mix it with REST Assured and Spring REST Docs to not only test the end-point but also to generate and publish API documents.

Importance of API Documentation

API is the middleware now. By following standard principles, the behavior of API can be consistent and predictable. Like any piece of code, APIs must be tested by all means from writing test-cases to assuring it's quality by QA. Ideally, when the application is assembled, API docs must also be generated and bundled into the artifact and be delivered together for deployment. That makes API docs a "living source of documents" and they become a common source of reference.

Spring REST Docs

There are few popular frameworks/tools to generate API documentation. Each one of such frameworks/tools has it's own adopted approach and comes with it's own benefits and drawbacks when compared with others. However, Spring IO has a project called Spring REST Docs that uniquely takes a very different approach. It's approach is centered around testing and is combined with hand-written Asciidoctor templates to produce high quality and maintainable API documentation. This approach definitely stands out as it promotes testing to it's greatest levels.

With the test-centric approach, it makes API document not only accurate-and-complete but also up-to-date and a living-resource reference. API documents generated this way are always as accurate as the code-base is. Also, as Spring framework is central to Grails framework, it becomes a natural-fit for Grails applications.

Having said all that, let's add Spring REST Docs to Grails 3 project and assure it with REST Assured.

Environment: Grails 3.2.11, Java 1.8, Apache Tomcat 8.0.20, IntelliJ IDEA Ultimate 2017.2 on Mac OS X 10.11.5 (El Capitan)

Step 0: Upgrade application from Grails 3.1.6 to 3.2.11

When I started this multi-part posts, Grails was at 3.1.6 and now it has advanced to 3.3.x. Just to catch up, I've upgraded this app from 3.1.6 to 3.2.11 (the latest on 3.2.x branch). It was an easy upgrade as it is a simple RESTful application. All I had to do was to bring gradle.properties and build.gradle files up-to-date with 3.2.11.

Step 1: Add Spring REST Docs to the Project (build configuration)

At the end of my last post, we had a secured resource (Artist) and we tested it's RESTful API for CRUD operations. That is good enough resource for taking it to the next level of generating & publishing it's API. Spring REST Docs' Getting Started has link to sample applications for reference. REST Assured Grails is the best-bet and is the basis for us. As a first step let's add Spring REST Docs support to the project as shown and described below:

Add Asciidoctor plugin.
build.gradle
plugins { ... id 'org.asciidoctor.convert' version '1.5.3' }

Run gradle tasks command and notice that asciidoctor task gets added by the plugin.
$./gradlew tasks ... Documentation tasks ------------------- asciidoctor - Converts AsciiDoc files and copies the output files and related resources to the build directory. groovydoc - Generates Groovydoc API documentation for the main source code. ... javadoc - Generates Javadoc API documentation for the main source code.

Spring REST Docs Build Configuration section has steps for Gradle build configuration. I will do this slightly different to extend build script for REST docs support by separating out additional build configuration for REST docs into it's own build file leveraging Gradle's script plugin concept. This way it is more cleaner and brings in some modularity to the build script.

Create a new build script file restdocs.gradle under project's gradle dir and reference it in the main build.gradle file at the very bottom as shown below:
build.gradle
apply from: 'gradle/restdocs.gradle'

Let's populate restdocs.gradle as shown below. I will add comments into the build script to explain certain code blocks.
gradle/restdocs.gradle
buildscript { repositories { maven { url 'https://repo.spring.io/libs-snapshot' } } } repositories { maven { url 'https://repo.spring.io/libs-snapshot' } } //add extra user-defined properties to the project through ext block ext { snippetsDir = file('build/docs/generated-snippets') //output dir of rest api doc snippets generated restDocsVersion = '2.0.0.BUILD-SNAPSHOT' restAssuredVersion = '2.9.0' } dependencies { testCompile "org.springframework.restdocs:spring-restdocs-core:$restDocsVersion" testCompile "org.springframework.restdocs:spring-restdocs-restassured:$restDocsVersion" testCompile "org.springframework.restdocs:spring-restdocs-asciidoctor:$restDocsVersion" }

Now, just run grails clean command. We will have spring-restdocs-core and spring-restdocs-restassured downloaded from maven central repo.

Let's keep expanding this script.
//task to clean generated rest api docs snippets dir task cleanSnippetsDir(type: Delete){ delete fileTree(dir: snippetsDir) }

Run ./gradlew tasks and notice that there is a new task added under Other tasks like:
Other tasks ----------- cleanIdeaWorkspace cleanSnippetsDir console

Configure test task as shown below:
test { dependsOn cleanSnippetsDir outputs.dir snippetsDir }

Now run ./gradlew test -m or ./gradew test --dry-run which will run gradle's test task in a dry run mode. It disables all tasks and shows the order in which tasks get executed. In this case, we can now see our new task cleanSnippetsDir in the list after all classes are created and before test-case classes get compiled.

Remember we got asciidoctor task by adding Gralde plugin as the very first step. We will customize it and specify that it depends on integrationTest task. With this dependency, every time when we run this task, it will have integration tests run. We want this kind of dependency as the approach that REST Docs brings in is to have REST API docs generated from the integration test cases. So, we need integration tests to run before we have docs generated.

Having said that, let's customize that task as follows:
//Configure asciidoctor task provided by Gradle asciidoctor plugin- https://github.com/asciidoctor/asciidoctor-gradle-plugin asciidoctor { doFirst{ //just print outputDir for reference during execution phase println "Running asccidoctor task. Check generated REST docs under: ${outputDir}" } dependsOn integrationTest logDocuments = true sourceDir = file('src/docs') inputs.dir snippetsDir separateOutputDirs = false attributes 'snippets': snippetsDir //configure snippets attribute for .adoc files }

Step 2: Run tests and make them pass: grails test-app

Grails test-app runs both unit tests and integration tests.
I have not written any test specifications so far but as part of creating domain objects using grails create-domain-class command, I have a few Spock Specification unit-tests created each with a default feature method "test something"(). All these default generated specifications are expected to fail to start with. I want to keep these tests around for future but want to make them pass. An easy way is to annotate all those methods with groovy's @NotYetImplemented annotation. It reverses the net result by making it pass when it actually fails. It makes sense for an un-implemented test. But when actually implemented, it fails forcing us to remove the annotation.

Spock's @PendingFeature is similar but is added only in Spock 1.1. Grails 3.2.x comes with Spock 1.0. For now, we are all good with that wonderful annotation provided by groovy. With this we have all unit-tests passing.

It's time now to write an integration test specification for our RESTful controller: ArtistController. Instead of writing a typical integration test-case, let's mix it with REST assured and REST API Docs and get both testing and API docs generation done in this phase.

Step 3: Assure REST by writing integration specification for RESTful Controller with a mix of REST Docs and a touch of REST assured.

Step 3a: Configure REST Assured testing framework (set up your test specification to generate documentation snippets)

The Spring REST Docs documentation has outlined these steps. Here is the gist of it:

The configuration of REST Assured is nothing but a request spec (RequestSpecifiction) using ResqusetSpecBuilder by adding documentation configuration as a JUnit filter to it.

Configure REST assured documentation output directory by declaring a restDocumentation field which is initialized with an instance of JUnitRestDocumentation and annotate it with JUnit's @Rule annotation. This rule gets executed before and after each feature method. A custom output directory can be specified by passing a constructor argument. We specify this custom dir, as in the build file, the snippetsDir property we set with is slightly different ('build/docs/generated-snippets') than the default ('build/generated-snippets').

Next, setup RequestSpecification by adding a filter and configure it with the above restDocumentation initialized as JUnit Rule.

Here is how our test spec looks after this configuration:
src/integration-test/groovy/com/giri/ApiDocumentationArtistSpec
package com.giri import geb.spock.GebSpec import grails.plugins.rest.client.RestBuilder import grails.test.mixin.integration.Integration import grails.transaction.Rollback import io.restassured.builder.RequestSpecBuilder import io.restassured.specification.RequestSpecification import org.junit.Rule import org.springframework.restdocs.JUnitRestDocumentation import static org.springframework.http.HttpStatus.* import static org.springframework.restdocs.restassured3.RestAssuredRestDocumentation.documentationConfiguration @Integration @Rollback class ApiDocumentationArtistSpec extends GebSpec { @Rule protected JUnitRestDocumentation restDocumentation = new JUnitRestDocumentation('build/docs/generated-snippets') private RequestSpecification documentationSpec def setup() { //set documentation specification this.documentationSpec = new RequestSpecBuilder().addFilter( documentationConfiguration(this.restDocumentation)) .build() } ...

Step 3b: Spockify, test RESTful end-point and get documentation snippets generated

With the above configuration, let's write a feature method to test GET request of /api/artists end-point. The following is a feature method added to the above specification along with a defined static constant whose value is set with relative end-point url and an injected application port property. The port is required to override the default port(8080) of REST assured testing framework. Note that grails start the application on a random available port each time when integration tests are run.

static final String ARTISTS_ENDPOINT = '/api/artists' @Value('${local.server.port}') protected int port ... void "test and document GET request (index action) of end-point: /api/artists"() { given: "" RequestSpecification requestSpecification = RestAssured.given(this.documentationSpec) .accept(MediaType.APPLICATION_JSON_VALUE) .filter( RestAssuredRestDocumentation.document( 'artists-list-example' ) ) when: def response = requestSpecification .when() .port(port) .get(ARTISTS_ENDPOINT) then: response.then() .assertThat() .statusCode(HttpStatus.OK.value()) }

The feature method name describes the intent of this feature method. In this step, we are only testing the GET request of an end-point. We will add the support for the highlighted and document intent of this feature.

With this, if you run grails dev test-app or grails -Dgrails.env=development test-app, the test will pass. Also, we will have the following six documentation snippets generated under build/docs/generated-snippets/artists-list-example directory:
 curl-request.adoc
 http-request.adoc
 http-response.adoc
 httpie-request.adoc
 request-body.adoc
 response-body.adoc

These are the snippet files to be included in the final API documentation. Just to see the contents check http-response.adoc and it will contain the actual response received as follows:
---- HTTP/1.1 200 OK X-Application-Context: application:development:0 Content-Type: application/json;charset=UTF-8 Transfer-Encoding: chunked Date: Tue, 29 Aug 2017 22:12:13 GMT Content-Length: 148 [{"id":"90ff9ac4-b1c0-4495-94d5-1550f463561a","dateCreated":"08/29/2017","firstName":"Giridhar","lastName":"Pottepalem","lastUpdated":"08/29/2017"}] ----

Step 3c: Create asciidoctor (.adoc) source templates
Create src/docs dir and create api-guide.adoc and artists.adoc files to start with. The api-guide.adoc is the main asciidoctor template which will include artists.adoc. The artists.adoc is the asciidoctor template for artists end-point.

Shown below is a portion of api-guide.adoc
= giri-api RESTful API Guide Giridhar Pottepalem :doctype: book :icons: font :source-highlighter: highlightjs :toc: left :toclevels: 4 :sectlinks: [[overview]] = Overview [[overview-http-verbs]] == HTTP Methods giri-api API follows standard HTTP and REST conventions as closely as possible in its exposure of resources as end-points and use of HTTP methods (verbs). ... [[resources]] = Resources include::artists.adoc[]

And portions of artists.adoc is shown below for creating an Artist (POST request/save action):
[[resources-artists]] == Artists An Artist is a resource which represents an Artist. [[resources-artists-create]] === Creating an Artist A `POST` request is used to create a new Artist. TIP: An Artist can be created only by an Admin user (with role `ROLE_ADMIN`) IMPORTANT: Once a new Artist is created... ==== Request structure include::{snippets}/artists-create-example/request-fields.adoc[] ==== Example request include::{snippets}/artists-create-example/curl-request.adoc[] ==== Response structure include::{snippets}/artists-create-example/response-fields.adoc[] ==== Example response include::{snippets}/artists-create-example/http-response.adoc[]
Highlighted are the references to generated snippets that get included in the generated end HTML5 doc.

Step 3d: Generate API doc
Now, lets run asciidoctor gradle task we got added through Step 1 as shown below:
./gradlew asciidoctor //runs in test env
./gradlew -Dgrails.env=development asciidoctor //runs in dev env

This task runs all integration test specifications because we configured it to depend on integrationTest task. Once it's run successfully with no failing tests, it converts our asciidoctor API templates to HTML5 doc by populating it with included generated snippets as we referenced in artists.adoc.

Now let's enhance our specification feature method to document request and response payload structure. Let's take the case of /api/artists end-point and GET request. There is no request payload for this request. So, we will simply add response payload specification as shown below:
void "Test and document show Artist request (GET request, show action) to end-point: /api/artists"() { given: "Pick an artist to show" Artist artist = Artist.first() and: "user logs in by a POST request to end-point: /api/login" String accessToken = authenticateUser('me', 'password') and: "documentation specification for showing an Artist" RequestSpecification requestSpecification = RestAssured.given(this.documentationSpec) .accept(MediaType.APPLICATION_JSON_VALUE) .filter( RestAssuredRestDocumentation.document( 'artists-retrieve-specific-example', PayloadDocumentation.responseFields( PayloadDocumentation.fieldWithPath('id').type(JsonFieldType.STRING).description('Artist id'), PayloadDocumentation.fieldWithPath('firstName').type(JsonFieldType.STRING).description('Artist first name'), PayloadDocumentation.fieldWithPath('lastName').type(JsonFieldType.STRING).description('Artist last name'), PayloadDocumentation.fieldWithPath('dateCreated').type(JsonFieldType.STRING).description("Date Created (format:MM/dd/yyyy)"), PayloadDocumentation.fieldWithPath('lastUpdated').type(JsonFieldType.STRING).description("Last Updated Date (format:MM/dd/yyyy)") ) ) ) when: "GET request is sent" def response = requestSpecification .header("X-Auth-Token", "${accessToken}") .when() .port(this.port) .get("${ARTISTS_ENDPOINT}/${artist.id}") def responseJson = new JsonSlurper().parseText(response.body().asString()) then: "The response is correct" response.then() .assertThat() .statusCode(HttpStatus.OK.value()) and: "response contains the id of Artist asked for" responseJson.id }

Similarly, we can write a test spec to test and document POST method (creating an Artist) as shown below. Remember, I have secured this method to role: ROLE_ADIN. So, it requires admin to be authenticated first to get a security token and then pass the security token in the subsequent secured requests like POST. The following is the complete test specification with a helper method added to authenticate the user:

/** * Helper method, authenticates the given user and returns the security token. * * @param username the user id * @param password the password * @return security token once successfully authenticated */ protected String authenticateUser(String username, String password) { String authResponse = RestAssured.given() .accept(MediaType.APPLICATION_JSON_VALUE) .contentType(MediaType.APPLICATION_JSON_VALUE) .body(""" {"username" : "$username", "password" : "$password"} """) .when() .port(this.port) .post(LOGIN_ENDPOINT) .body() .asString() return new JsonSlurper().parseText(authResponse).'access_token' } void "Test and document create Artist request (POST request, save action) to end-point: /api/artists"() { given: "current number of Artists" int nArtists = Artist.count() and: "admin logs in by a POST request to end-point: /api/login" String accessToken = authenticateUser('admin', 'admin') and: "documentation specification for creating an Artist" RequestSpecification requestSpecification = RestAssured.given(this.documentationSpec) .accept(MediaType.APPLICATION_JSON_VALUE) .contentType(MediaType.APPLICATION_JSON_VALUE) .filter( RestAssuredRestDocumentation.document( 'artists-create-example', PayloadDocumentation.requestFields( PayloadDocumentation.fieldWithPath('firstName').description('Artist first name'), PayloadDocumentation.fieldWithPath('lastName').description('Artist last name') ), PayloadDocumentation.responseFields( PayloadDocumentation.fieldWithPath('id').type(JsonFieldType.STRING).description('Artist id'), PayloadDocumentation.fieldWithPath('firstName').type(JsonFieldType.STRING).description('Artist first name'), PayloadDocumentation.fieldWithPath('lastName').type(JsonFieldType.STRING).description('Artist last name'), PayloadDocumentation.fieldWithPath('dateCreated').type(JsonFieldType.STRING).description("Date Created (format:MM/dd/yyyy)"), PayloadDocumentation.fieldWithPath('lastUpdated').type(JsonFieldType.STRING).description("Last Updated Date (format:MM/dd/yyyy)") ) ) ) when: "POST request is sent with valid data" def response = requestSpecification .header("X-Auth-Token", "${accessToken}") .body("""{ "firstName" : "Bhuvan", "lastName" : "Pottepalem" }""") .when() .port(this.port) .post(ARTISTS_ENDPOINT) def responseJson = new JsonSlurper().parseText(response.body().asString()) then: "The response is correct" response.then() .assertThat() .statusCode(HttpStatus.CREATED.value()) and: "response contains the id of Artist created" responseJson.id and: "Number of Artists in the system goes up by one" Artist.count() == nArtists + 1 }

Now, simply run
./gradlew asciidoctor

We will have API docs generated under build/asciidoc dir. Open api-guide.html in a browser to see how nicely the generated API doc looks.

TIP: The beauty of Spring REST Docs framework is that, if compares the actual request/response fields with the PayloadDocumentation filed descriptions and will fail the test if any field(s) are missed or mis-matched. This ensures that the API documentation is up-to-date with the implementation.

Step 4: Publish API
Now, we have fully integrated REST Assured and Spring REST Docs into integrationTest phase with an added asciidoctor Gradle test task. The result of this is an up-to-date API document generated for our Restful service.The API document is the source for clients using this service. So, it needs to be made available. One way to achieve this is to bundle the generated HTML5 API docs with the application's deployable war or executablejar and have it's own end-point to serve it.

Spring Boot (the framework Grails3 underpins) can be leveraged to achieve this. By default Boot serves static content placed under /static or /public in the class path or root of the application context. Here is the link for reference: Spring boot Static content.

Step 4a: Bundle API documentation into deployable artifact
We will enhance our build script (restdocs.gradle) and customize war task that comes with Gradle Java plugin little bit to achieve this. Below is the code snippet which is self explanatory:
/* Bundles generated API docs into war file. * Spring boot serves static content under /public or /static or /resources or /META-INF/resources. * Hooks into war task and adds asciidoctor task dependency, also copies generaed rest docs appropriately * for bundling into war file. */ def publicDocsDir = 'WEB-INF/classes/public/docs' war { dependsOn asciidoctor from ("${asciidoctor.outputDir}") { into publicDocsDir } }

We basically made war task depend on asciidoctor task and added a step to copy generated HTML5 API docs to WEB-INF/classes/public/docs dir in the generated war file.

Now, run grails war to generate deployable war artifact:
grails war

You can explode and see that generated API docs are bundled into the war generated (giri-api-0.1.war):
e.g. jar tvf build/libs/giri-api-0.1.war | grep html will list the following:
 59738 Mon Sep 04 07:16:34 EDT 2017 WEB-INF/classes/public/docs/api-guide.html
 47974 Mon Sep 04 07:16:34 EDT 2017 WEB-INF/classes/public/docs/artists.html

Step 4b: Make API documentation available from it's own end-point
Deploy the generated war file onto a locally running tomcat.
Deploy the war onto locally running Tomcat and point your browser at: http://localhost:8080/giri-api-0.1/static/docs/api-guide.html

This will result into Access Denied error. We need to open up security to serve API docs.

Lets change application.groovy and add /static/docs/** to both grails.plugin.springsecurity.controllerAnnotations.staticRules and filterChainChainMaps as shown below:
grails.plugin.springsecurity.controllerAnnotations.staticRules = [ ... [pattern: '/static/docs/**', access:['permitAll']] ] def filterChainChainMaps = [ ... pattern: '/static/docs/**', filters: statelessFilters], ... ]

Create a war file. Undeploy previously deployed war and deploy the latest war file.

Now, http://localhost:8080/giri-api-0.1/static/docs/api-guide.html (API docs) should be served and displayed by the app.

The test specification can be enhanced easily along these lines to test and document rest of the service methods: show, update and delete available for /api/artists end-point through HTTP methods GET specific resource by id, UPDATE and DELETE respectively.

The complete source code is hosted on GitHub at https://github.com/gpottepalem/giri-api for reference.

References

Sunday, May 28, 2017

Create a Secured Restful API App with Grails 3, PostgreSQL and JMS - Step by Step: Part 4 of 5

Posts on this topic

Part 4: Secure end-points fully and cleanly

I left my last post with basic domain objects (Artist, ArtWork, Specification) created, and one of the core domain objects (Artistexposed as a RESTful resource by leveraging Grails provided @Resource annotation. Without writing any further code, I got all CRUD operations for Artist resource working in RESTful way. That's pretty neat out-of-the-box default implementation provided by Grails framework. I also ended my last post with a note about not-so-readable UUID and date formats. This post is a continuation of previous and is all about securing Artist resource end-point fully.

Environment: Grails 3.1.6, Java 1.8, IntelliJ 15 on Mac OS X 10.9.5

Step 1 First, let's make id, dateCreated and lastUpdated formats more readable
There are multiple ways to customize data formatting.

i) The easiest way is to simply register JSON marshallers as shown below in Bootstrap.groovy. Grails runs grails-app/init/*Bootstrap classes' init closure(s) at the startup of the application.

grails-app/init/Bootstrap.groovy
class BootStrap { def init = { //register JSON marshaller for Date grails.converters.JSON.registerObjectMarshaller(Date){ return it?.format('MM/dd/yyyy') } //register JSON marshaller for UUID grails.converters.JSON.registerObjectMarshaller(UUID){ return it?.toString() } ... } ... }

ii) Another way to register marshallers is by defining a Spring bean that registers all marshallers as shown below:

src/main/groovy/com/giri/marshallers/CustomObjectMarshaller.groovy
package com.giri.marshallers /** * Custom object marshaller trait for all custom object marshallers to implement. */ trait CustomObjectMarshaller { abstract void register() }

src/main/groovy/com/giri/marshallers/UUIDMarshaller.groovy
package com.giri.marshallers import grails.converters.JSON /** * UUID marshaller, registers a {@link JSON} marshaller to output the string representation of {@link UUID} */ class UUIDMarshaller implements CustomObjectMarshaller { @Override void register(){ JSON.registerObjectMarshaller(UUID){ UUID uuid-> return uuid.toString() } } }

src/main/groovy/com/giri/marshallers/DateMarshaller.groovy
package com.giri.marshallers import grails.converters.JSON /** * Date marshaller, registers a {@link JSON} marshaller to output the string representation of {@link Date} */ class DateMarshaller implements CustomObjectMarshaller { @Override void register() { JSON.registerObjectMarshaller(Date) {Date date -> return date.format('MM/dd/yyyy') } } }

src/main/groovy/com/giri/marshallers/CustomMarshallerRegistrar.groovy
package com.giri.marshallers import javax.annotation.PostConstruct /** * Custom Marshaller Registrar, registers custom object marshallers with spring. * Configured as a spring managed bean in resources.groovy * * @see resources.groovy */ class CustomMarshallerRegistrar { /** List of custom marshallers to be registered, initialized with bean configuration in resources.groovy */ List marshallers @PostConstruct void registerCustomMarshallers() { marshallers.each{ it.register() } } }

grails-app/conf/spring/resources.groovy
import com.giri.marshallers.CustomMarshallerRegistrar import com.giri.marshallers.DateMarshaller import com.giri.marshallers.UUIDMarshaller beans = { //JSON Marshallers customMarshallerRegistrar(CustomMarshallerRegistrar) { marshallers = [ new UUIDMarshaller(), new DateMarshaller() ] } }

With this, UUID and date formats in the response look like:
$ curl -i -X GET 'http://localhost:8080/api/artists' HTTP/1.1 200 OK Server: Apache-Coyote/1.1 ... [{"id":"8d6698a1-03db-4676-973b-bb374aa1381c","dateCreated":"05/27/2017","firstName":"Giri","lastName":"Pottepalem","lastUpdated":"05/27/2017"}]

iii) There is even a better way of customizing the response using Grails recent addition- JSON views, which is not covered in this post.

Step 2 Secure Resource end-point

Let's start securing Artist resource end-point.

When domain class is annotated with @Resource, Grails provides  RestfulController implementation for CRUD actions, maps them to appropriate HTTP method verbs and makes the resource accessible at the end-point specified through uri property of @Resource annotation in RESTful way.

In addition, Spring security core plugin's @Secured annotation can be applied on the domain object to secure the resource. In my previous post's Step 9, I allowed everyone access to /api/artists end-point by annotating Artist domain object with @Secured(['permitAll']). With this all CRUD operations are allowed without a login. We need to secure this resource now.

Let's say we want to allow only Admin user to access Artist resource. This can easily be done by changing the annotation to @Secured(['ROLE_ADMIN']) or simply to @Secured('ROLE_ADMIN')

With that, the end-point /api/artists is now secured and is accessible to only users Admin role. Let's test it.

Get Artists
$ curl -i -X GET 'http://localhost:8080/api/artists' HTTP/1.1 403 Forbidden Server: Apache-Coyote/1.1 Content-Type: application/json;charset=UTF-8 ...

Login as Admin
$ curl -i -H "Accept: application/json" -H "Content-Type: application/json" -X POST -d '{"username":"admin","password":"admin"}' http://localhost:8080/api/login HTTP/1.1 200 OK Server: Apache-Coyote/1.1 Cache-Control: no-store Pragma: no-cache Content-Type: application/json;charset=UTF-8 Content-Length: 93 Date: Wed, 24 May 2017 22:14:07 GMT {"username":"admin","roles":["ROLE_ADMIN"],"access_token":"ucsbqbd3f26fjpb5b6794ph7cbu3fqq2"}

Get Artists as logged in Admin
$ curl -i -H "X-Auth-Token: ucsbqbd3f26fjpb5b6794ph7cbu3fqq2" http://localhost:8080/api/artists HTTP/1.1 200 OK Server: Apache-Coyote/1.1 X-Application-Context: application:development Content-Type: application/json;charset=UTF-8 Transfer-Encoding: chunked Date: Wed, 24 May 2017 22:15:13 GMT []

Post an Artist
$ curl -i -X POST -H "Content-Type:application/json" -d '{ "firstName": "Giri", "lastName": "Pottepalem" }' 'http://localhost:8080/api/artists' HTTP/1.1 403 Forbidden Server: Apache-Coyote/1.1 Content-Type: application/json;charset=UTF-8 Transfer-Encoding: chunked Date: Wed, 24 May 2017 22:25:52 GMT {"timestamp":1495664752925,"status":403,"error":"Forbidden","message":"Access Denied","path":"/api/artists"}

Post an Artist as logged in Admin
$ curl -i -X POST -H "X-Auth-Token: ucsbqbd3f26fjpb5b6794ph7cbu3fqq2" -H "Content-Type: application/json" -d '{ "firstName": "Giri", "lastName": "Pottepalem" }' 'http://localhost:8080/api/artists' HTTP/1.1 201 Created Server: Apache-Coyote/1.1 X-Application-Context: application:development Location: http://localhost:8080/api/artists/a0480de2-d5df-43eb-a919-196e34c40ab5 Content-Type: application/json;charset=UTF-8 Transfer-Encoding: chunked Date: Wed, 24 May 2017 22:53:07 GMT {"id":"a0480de2-d5df-43eb-a919-196e34c40ab5","dateCreated":"05/24/2017","firstName":"Giri","lastName":"Pottepalem","lastUpdated":"05/24/2017"}

Step 3 Secure Resource end-point properly and fully

Though @Secured('ROLE_ADMIN') makes the resource secured easily to the role specified, this may not meet the actual security requirements. Let's say, the right level of security we want to apply to the end-point: /api/artists is as follows:
  • Allow everyone to see the list of Artists
  • Only allow admin to create/delete an Artist
  • A logged in Artist can only see/update his/her details
We have now specific security logic that we need to apply to different actions on the resource. This level of customization is not possible with @Secured annotation applied at the resource-level. It requires some customization at the action level and this is where we can implement our own REST controller for the resource to achieve this. Grails comes with grails.rest.RestfulController base implementation that can be extended. This is not required but gives you some common base logic that can be leveraged.

Let's generate a REST controller now for the resource/domain object. Grails 3 offers create-restful-controller command for creating a RESTful controller.

$ grails create-restful-controller com.giri.Artist | Created grails-app/controllers/com/giri/ArtistController.groovy

The generated class looks like:
package com.giri import grails.rest.* import grails.converters.* class ArtistController extends RestfulController { static responseFormats = ['json', 'xml'] ArtistController() { super(Artist) } }

The generated controller is minimal with default implementation for all actions derived from the base RestfulController class provided by Grails. With this controller in place for our customization we no longer need @Resource and @Secured annotations on the domain class. Let's remove those and add URL mappings in UrlMappings.groovy for the resource.

grails-app/contrllers/giri/api/UrlMappings.groovy
package giri.api class UrlMappings { static mappings = { ... "/api/artists"(resources: 'artist') }

With this we can run grails url-mappings-report command to check url mappings for the resource end-point. It should look same as it was with @Resource applied on the Artist domain class. Now we can provide necessary action-methods implementations and secure each action-method with @Secured annotation and achieve custom security that we wanted for the end-point's each HTTP verb mapped to a specific action-method.

The customized ArtistController class looks like(with custom security highlighted):
package com.giri import grails.plugin.springsecurity.annotation.Secured import grails.rest.RestfulController import grails.transaction.Transactional /** * Customized Artists RestfulController. * * @author Gpottepalem * Created on May 26, 2017 */ class ArtistController extends RestfulController { static responseFormats = ['json', 'xml'] ArtistController() { super(Artist) } @Secured('permitAll') @Override def index(Integer max) { super.index(max) } @Secured('ROLE_USER') @Override def show() { super.show() } @Secured('ROLE_ADMIN') @Override def save() { super.save() } @Secured('ROLE_USER') @Override def update() { super.update() } @Secured('ROLE_ADMIN') @Override def delete() { super.delete() } }
Not much customization is done to achieve this other than overwriting just needed action-methods and annotating them properly per security requirements. All overwritten methods simple delegate the implementation to the super class.
Gotcha: There is no need for @Transactional annotation for methods like save(), update() and delete() as they all simply call corresponding super methods and all super methods are annotated appropriately for transactionality. In fact, annotating these methods again in this kind of implementation results into exception ;)

Step 4 Test fully customized Resource end-point

With the required customization done, let's take a spin and test it. Note that I have bootstrapped an admin user and a me user as specified in Step 4 of my earlier post.

# GET Artists (index) $ curl -i -X GET 'http://localhost:8080/api/artists' HTTP/1.1 200 OK ... []

# Login as Admin $ curl -i -H "Accept: application/json" -H "Content-Type: application/json" -X POST -d '{"username":"admin","password":"admin"}' http://localhost:8080/api/login HTTP/1.1 200 OK Server: Apache-Coyote/1.1 ... {"username":"admin","roles":["ROLE_ADMIN"],"access_token":"h1tdbs1cc8e7qt1bt7ohpsar57nt8car"}

# Login as me user $ curl -i -H "Accept: application/json" -H "Content-Type: application/json" -X POST -d '{"username":"me","password":"password"}' http://localhost:8080/api/login HTTP/1.1 200 OK Server: Apache-Coyote/1.1 ... {"username":"me","roles":["ROLE_USER"],"access_token":"ci9ct5hocreljl5pbqga60npsi8ol03f"}

# POST Artist as user (save) $ curl -i -X POST -H "X-Auth-Token: ci9ct5hocreljl5pbqga60npsi8ol03f" -H "Content-Type: application/json" -d '{ "firstName": "Giri", "lastName": "Potte" }' 'http://localhost:8080/api/artists' HTTP/1.1 403 Forbidden Server: Apache-Coyote/1.1 ... {"timestamp":1495891839637,"status":403,"error":"Forbidden","message":"Access is denied","path":"/api/artists"}

# POST Artist as admin (save) $ curl -i -X POST -H "X-Auth-Token: h1tdbs1cc8e7qt1bt7ohpsar57nt8car" -H "Content-Type: application/json" -d '{ "firstName": "Giri", "lastName": "Potte" }' 'http://localhost:8080/api/artists' HTTP/1.1 201 Created Server: Apache-Coyote/1.1 ... {"id":"8d6698a1-03db-4676-973b-bb374aa1381c","dateCreated":"05/27/2017","firstName":"Giri","lastName":"Potte","lastUpdated":"05/27/2017"}

# GET Artists (index) $ curl -i -X GET 'http://localhost:8080/api/artists' HTTP/1.1 200 OK Server: Apache-Coyote/1.1 ... [{"id":"8d6698a1-03db-4676-973b-bb374aa1381c","dateCreated":"05/27/2017","firstName":"Giri","lastName":"Potte","lastUpdated":"05/27/2017"}]

# GET an Artist (show) $ curl -i -X GET 'http://localhost:8080/api/artists/8d6698a1-03db-4676-973b-bb374aa1381c' HTTP/1.1 403 Forbidden Server: Apache-Coyote/1.1 ... {"timestamp":1495893259443,"status":403,"error":"Forbidden","message":"Access Denied","path":"/api/artists/8d6698a1-03db-4676-973b-bb374aa1381c"}

# GET an Artist (show) as admin - secured for ROLE_USER $ curl -i -X GET -H "X-Auth-Token: h1tdbs1cc8e7qt1bt7ohpsar57nt8car" 'http://localhost:8080/api/artists/8d6698a1-03db-4676-973b-bb374aa1381c' HTTP/1.1 403 Forbidden Server: Apache-Coyote/1.1 ... {"timestamp":1495893471587,"status":403,"error":"Forbidden","message":"Access is denied","path":"/api/artists/8d6698a1-03db-4676-973b-bb374aa1381c"}

# GET an Artist (show) as me user - secured for ROLE_USER $ curl -i -X GET -H "X-Auth-Token: ci9ct5hocreljl5pbqga60npsi8ol03f" 'http://localhost:8080/api/artists/8d6698a1-03db-4676-973b-bb374aa1381c' HTTP/1.1 200 OK Server: Apache-Coyote/1.1 ... {"id":"8d6698a1-03db-4676-973b-bb374aa1381c","dateCreated":"05/27/2017","firstName":"Giri","lastName":"Potte","lastUpdated":"05/27/2017"}

# PUT Artist as admin (update) $ curl -i -X PUT -H "X-Auth-Token: h1tdbs1cc8e7qt1bt7ohpsar57nt8car" -H "Content-Type: application/json" -d '{ "lastName": "Pottepalem" }' 'http://localhost:8080/api/artists/8d6698a1-03db-4676-973b-bb374aa1381c' HTTP/1.1 403 Forbidden ... {"timestamp":1495892176757,"status":403,"error":"Forbidden","message":"Access is denied","path":"/api/artists/8d6698a1-03db-4676-973b-bb374aa1381c"}

# PUT Artist as me user (update) $ curl -i -X PUT -H "X-Auth-Token: ci9ct5hocreljl5pbqga60npsi8ol03f" -H "Content-Type: application/json" -d '{ "lastName": "Pottepalem" }' 'http://localhost:8080/api/artists/8d6698a1-03db-4676-973b-bb374aa1381c' HTTP/1.1 200 OK Server: Apache-Coyote/1.1 ... {"id":"8d6698a1-03db-4676-973b-bb374aa1381c","dateCreated":"05/27/2017","firstName":"Giri","lastName":"Pottepalem","lastUpdated":"05/27/2017"}

# GET Artists (index) $ curl -i -X GET 'http://localhost:8080/api/artists' HTTP/1.1 200 OK Server: Apache-Coyote/1.1 ... [{"id":"8d6698a1-03db-4676-973b-bb374aa1381c","dateCreated":"05/27/2017","firstName":"Giri","lastName":"Pottepalem","lastUpdated":"05/27/2017"}]

# DELETE Artist as user (delete) $ curl -i -X DELETE -H "X-Auth-Token: ci9ct5hocreljl5pbqga60npsi8ol03f" 'http://localhost:8080/api/artists/8d6698a1-03db-4676-973b-bb374aa1381c' HTTP/1.1 403 Forbidden Server: Apache-Coyote/1.1 ... {"timestamp":1495892582172,"status":403,"error":"Forbidden","message":"Access is denied","path":"/api/artists/8d6698a1-03db-4676-973b-bb374aa1381c"}

# DELETE Artist as admin (delete) $ curl -i -X DELETE -H "X-Auth-Token: h1tdbs1cc8e7qt1bt7ohpsar57nt8car" 'http://localhost:8080/api/artists/8d6698a1-03db-4676-973b-bb374aa1381c' HTTP/1.1 204 No Content Server: Apache-Coyote/1.1 ...

# GET Artists (index) $ curl -i -X GET 'http://localhost:8080/api/artists' HTTP/1.1 200 OK Server: Apache-Coyote/1.1 ... []

# Logout admin $ curl -i -X POST -H "X-Auth-Token: h1tdbs1cc8e7qt1bt7ohpsar57nt8car" http://localhost:8080/api/logout HTTP/1.1 200 OK Server: Apache-Coyote/1.1 ...

# Logout user $ curl -i -X POST -H "X-Auth-Token: ci9ct5hocreljl5pbqga60npsi8ol03f" http://localhost:8080/api/logout HTTP/1.1 200 OK Server: Apache-Coyote/1.1 ...

Everything looks good except that any logged-in user can see/update any other user as we only secured update method to ROLE_USER. We can easily add some custom logic to show() and update() action-methods to lock-down these actions further so that a logged-in user can only see/update his/her own user. Grails Spring Security core plugin provides SpringSecurityService class that can be leveraged to achieve this. Since the basic domain model currently I have has not evolved enough for making this check, I am only showing pseudo-coding-steps here:
class ArtistController extends RestfulController { SpringSecurityService springSecurityService ... def update() { AppUser currentUser = springSecurityService.currentUser as AppUser AppUser updateArtist = //Find user account of the Artist's id (params.id) being updated if(currentUser != updateArtist) { respond([message: 'Access Denied'], status: HttpStatus.FORBIDDEN) return } else { ... } }

Gotchas

Grails Spring Security Core plugin's login form
The default Grails Spring Security Core plugin provided login action url: /login/auth when accessed runs into an exception upon not finding an associated view resulting into Internal Server Error response. This is available due to "/$controller/$action?/$id?(.$format)?" mapping in UrlMappings.groovy and /login/auth is mapped to LoginController's auth() action-method provided by Grails Spring Security Core plugin. There is no point in having this wide-open anymore as it provides a form-based login for web application which is not used with Grails Spring Security REST plugin. So, let's lock it down.

When /login/auth is accessed, it runs into the following exception:
javax.servlet.ServletException: Could not resolve view with name '/login/auth' in servlet with name 'grailsDispatcherServlet'

And the response looks like:
$ curl -i -X GET 'http://localhost:8080/login/auth' HTTP/1.1 500 Internal Server Error Server: Apache-Coyote/1.1 ... Connection: close {"message":"Internal server error","error":500}

Lock it down by adding the following pattern to staticRules in application.groovy
grails.plugin.springsecurity.controllerAnnotations.staticRules = [ [pattern: '/login/auth', access: ['denyAll']] //lock down spring security login form url ... ] //Spring Security REST API plugin config String statelessFilters = 'JOINED_FILTERS, -exceptionTranslationFilter, -authenticationProcessingFilter, -securityContextPersistenceFilter, -rememberMeAuthenticationFilter' //common def filterChainChainMaps = [ //Stateless chain [pattern: '/api/**', filters: statelessFilters], [pattern: '/**', filters: statelessFilters] //Traditional stateful chain - We are stateless, no stateful chain is required ] grails.plugin.springsecurity.filterChain.chainMap = filterChainChainMaps

With this, when we access /login/auth, we get the following response:
$ curl -i -X GET 'http://localhost:8080/login/auth' HTTP/1.1 403 Forbidden Server: Apache-Coyote/1.1 ... {"timestamp":1495926839720,"status":403,"error":"Forbidden","message":"Access Denied","path":"/login/auth"}

Using Custom subclass of RestfulController with @Resource annotation
If you prefer to annotate your custom RestfulController with @Resource instead of mapping the resource in UrlMappings.groovy, there is a small section in Grails docs that describes how to get this done. However, it has some limitations at the time of my exploration as I had to place the controller under src/main/groovy instead of under grails-app/controllers.

References

Saturday, May 13, 2017

Running Tests - differences between Grails 2 and Grails 3 . . .

Grails - A highly-productive and rapid application development framework for developing modern applications on JVM which not only promotes several good design & development practices but also promotes testing. It takes testing as an integral part of development by making many hard things simple. It also makes writing & running tests as fun as development ;)

Grails 3 got so much better than Grails 2. However, there are several differences between Grails 2 and Grails 3 applications and testing is no exception. Due to these differences, confusions arise running tests while switching between Grails 2 and Grails 3 applications. Grails documentation covers this in detail, but it's handy to have all these at one place, and hence I write this blog post ;)

Grails 2 way

Usage grails [environment] test-app [SpecificTest ...] [test-phase:[spock]...]

[environment] is optional, defaults to test, when specified it can be just dev (shortcut for development) or -Dgrails.env=<environment> where <environment> can be development, test or custom-env .

[SpecificTest ...] is optional and when specified, it can take one or more SpecificTests to run separated by a space, each SpecificTest can also take wild card * at either class-level or package-level.

[test-phase:[spock] ...] is optional, defaults to all test phases, unit:, integration: and functional:test phases.  When specified it can be one, any two or all three of test phases separated by space.  The extra spock is only needed for running spock specifications.

Examples # run all tests: unit, integration and functional in test environment grails test-app # run all tests: unit, integration and functional in development environment grails dev test-app grails -Dgrails.env=development test-app # run all tests in 'specific-env' grails -Dgrails.env=<specific-env> test-app # run all unit tests in test environment grails test-app unit: # run all unit and functional tests in test environment grails test-app unit: functional: # run all unit tests in a specific package in test environment grails test-app com.giri.* unit: # run all unit & integration tests in a specific package in test environment grails test-app com.giri.* unit: integration: # run selected unit tests in test environment grails test-app Test1 Test2 unit: # run all unit tests in a package, particular test method of a test-case & specific test-case in test environment grails test-app com.giri.* Test1.testMethod1 MyTest2 unit: # run all spock unit test specifications in test environment grails test-app unit:spock

Grails 3 - Grails way

Grails 3 changed it's build system from GANT (Groovy ANT) to Gradle. It provides Grails commands for Gradle tasks. Gradle doesn't distinguish integration and functional tests and hence in Grails 3 both integration and functional test-phases are combined into integration test phase. In other words, both integration and functional tests are run as part of integration test-phase. Unlike Grails 2 which takes test-phase as test-phase:, Grails 3 takes test-phase as -test-phase.

Note the difference: at the end of test-phase vs. - in the beginning of the test-phase. Also, there is no need for giving any special indication for Spock specifications.

Usage grails [environment] test-app [SpecificTest ...] [-test-phase...]

[environment] is optional, defaults to test, when specified it can be dev (shortcut for development) or -Dgrails.env=environment where environment can be development, test or custom-env .

[SpecificTest ...] is optional and when specified, it can take one or more SpecificTests to run separated by space, each SpecificTest can also take wild card * at either class-level or package-level

[-test-phase ...] is optional, defaults to all: -unit and -integration. When specified it can be one, or both test phases separated by space.

Examples (with grails command) # run all tests: unit, integration and functional in test environment grails test-app # run all tests: unit, integration and functional in development environment grails dev test-app grails -Dgrails.env=development test-app # run all tests in 'specific-env' grails -Dgrails.env=<specific-env> test-app # run all unit tests grails test-app -unit # run all unit, integration & functional tests. Equivalent to not specifying test-phases at all grails test-app -unit -integration # run all unit tests in a specific package grails test-app com.giri.* -unit # run all unit, integration and functional tests in a specific package grails test-app com.giri.* -unit -integration # run selected unit tests grails test-app Test1 Test2 -unit # run all unit tests in a package, particular test method of a test-case & specific test-case grails test-app com.giri.* Test1.testMethod1 MyTest2 -unit

Grails 3 - Gradle way

Grails 3 has Gradle as it's build system and hence one can also use gradle tasks directly to run tests. There are only two test-phases: unit(test) and integration (integrationTest). Integration test-phase covers both integration and functional. Following are examples of running tests with gradle command:

Examples (with gradle command, it's recommended to use gradle wrapper: gradlew) # run all unit tests, units tests don't need to be run in a specific environment ./gradlew test # run all unit tests in a specific package ./gradlew test --tests com.giri.* # run all unit tests in a specific package matching the specific test class name pattern ./gradlew test --tests com.giri.My*Spec # run specific unit test spec ./gradlew test --tests com.giri.Test1Spec # run specific unit test-spec and a specific feature method (when method name is in JUnit style) ./gradlew test --tests com.giri.Test1Spec.testFeatureMethod # run specific unit test-spec and a specific feature method (method name is in Spock style) ./gradlew test --tests com.giri.Test1Spec."test feature method" # run all integration and functional tests in development environment ./gradlew -Dgrails.env=development integrationTest #run integration and functional tests in 'specific-env' ./gradlew -Dgrails.env=<specific-env> integrationTest # clean & run all integration and functional tests in development environment. Continue on failing unit tests ./gradlew -Dgrails.env=development clean test integrationTest --continue # assemble but skip running tests (assemble depends on integrationTest) ./gradlew -Dgrails.env=development assemble -x integrationTest NOTE # run all unit tests in a package, a particular test method of a test-case and a specific test-case I haven't found a way to get this... #./gradlew test --tests com.giri.* Test1.testMethod1 MyTest2

Knowing all these differences and possible ways of running tests will definitely be a time saver in a developer's day-to-day development.

My Other posts on Grails Testing

References

Sunday, January 01, 2017

Get release details into your Grails-3 application . . .

It is always a good practice to make application release details like: commit id, version number and build tag etc. available within the application for display as a reference to know the current deployed code state of the application. Typically, these details can be taken into the application during build, specifically during the assembling phase of creating a deployable artifact (war or jar archive file). In an integrated development environment, Continuous Integration (CI) platforms like Jenkins, Bamboo can pass in release details to the build job which is setup to run application's build task(s).

In Grails-3 applications, this can simply be achieved by hooking into the underlying Gradle build system's assemble task. This involves writing customizing processResources task in build.gradle build file something like as shown below:

build.gradle
... //filters, populates BUILD_TAG and GIT_REVISION in the resource file: application.yml with their //actual values passed through -P flag when 'assemble' task is run processResources { println "Processing resources..." def buildTag = project.properties['BUILD_TAG'] ?: 'undefined' def gitRevision = project.properties['GIT_REVISION'] ?: 'undefined' filesMatching("**/application.yml") { println "Replacing BUILD_TAG and GIT_REVISION in application.yml with ${buildTag} and ${gitRevision}..." expand( [ 'BUILD_TAG' : buildTag, 'GIT_REVISION': gitRevision ] ) } } ...

That's no big deal, but if you have more than one Grails-3 application in your organization, with this strategy, you end up copying such customized build code snippet across all projects.

Grails framework applies and promotes DRY (Don't Repeat Yourself) principle. Gradle offers a plugin architecture for sharing reusable build logic across many projects'  build scripts. Gradle's plugin architecture can be leveraged to put a DRY solution to this problem. So, an ideal DRY solution to this problem in a Grails-3 environment is to make this piece of build logic packaged into as a simple custom Gradle Plugin.

I recently created a simple Gradle plugin as a DRY solution to this problem and hosted it on our organization's Bintray repository. The source code is hosted and available on Github.

Plugin Source Code - https://github.com/gpottepalem/gradle-release-info
Plugin repository - https://bintray.com/goodstartgenetics/gradle-plugins/release-info

References