Showing posts with label API. Show all posts
Showing posts with label API. Show all posts

Thursday, October 08, 2020

Make your Spring Boot application's API documentation a complete specification with enhanced Swagger-UI annotations . . .

In a RESTful application, documenting end-point specification/schema is very important. There are various frameworks with different approaches available in Java space addressing this problem. It is obvious the best way is: to generate API specification from the source code so that it stays on up-to-date and accurate with your source code.

Spring RESTDocs offers a very good solution. It generates API doc from hand-written Asciidoctor templates merged with auto-generated snippets that are generated from unit/integration tests by promoting end-point testing to great levels. (Refer to my earlier post on this in a Grails application.)

Swagger UI is another solution which generates visual documentation from the source code. This also generates a testable Swagger UI page for all end-points along with Open API specification for each end-point. To get this right and complete, it requires adding additional details for documenting API specification/schema either in an yml file or by annotating source code, basically end-point action methods and objects involved in request/response handling.

Swagger UI is very useful and convenient to not only to know the specification details, but also to test REST APIs both from the same page. Spring boot comes with good support for this. I am not going to go into details of how to add Swagger UI support with Open API specification for a Spring Boot application. There are numerous posts on this.

This post is more on leveraging Swagger (OpenAPI 3 implementation) annotations in order to get better API specification/schema generated. Also, it goes into details on customizing the example end-point request/response JSON that shows sample request with meaningful data, rather than default data. Without adding any specific annotations for API specification, you will get a decent Swagger UI page. However, it is good to add little more details and make the specification much cleaner and clear.

Environment: Java 13, Spring Boot 2.2.4.RELEASE, PostgreSQL 12.1, Maven 3.6.2 on macOS Catalina 10.15.6

Without any additional Swagger annotations

For instance, in a Spring Boot application, a POST operation end-point method to create a Person, and the request objects with no additional swagger annotations like shown below:

@RestController @Slf4j public class PersonController { ... @PostMapping(value = "/person", produces = { MediaType.APPLICATION_JSON_VALUE, MediaTypes.HAL_JSON_VALUE }) public ResponseEntity create(@Valid @RequestBody Person person) { ... return new ResponseEntity<>(newPerson, HttpStatus.OK); } } @Data public class Person { private String firstName; private String lastName; private Gender gender private int age; private String email; private Address address; } @Data public class Address { private String address1; private String address2; private String city; private String state; private String zip; } public enum Gender { FEMALE, MALE }

would result into Swagger UI as shown below:


and request schema details look like:


Note that the example request JSON is not good with respect to data for fields. When you click on Try it out button to test the API, you will have to edit the values of all fields with good data. To have a  good example request with good sample data generated in Swagger UI page requires additional Swagger annotations.

With additional Swagger annotations

Enhancing code by adding annotations as shown below:

@RestController @Slf4j public class PersonController { ... @Operation(summary = "Creates a new Person.", tags = { "Person" }) @ApiResponses(value = { @ApiResponse(responseCode = "200", description = "Returns newly created Person."), @ApiResponse(responseCode = "403", description = "Authorization key is missing or invalid."), @ApiResponse(responseCode = "400", description = "Invalid request.") }) @PostMapping(value = "/person", produces = { MediaType.APPLICATION_JSON_VALUE, MediaTypes.HAL_JSON_VALUE }) public ResponseEntity create(@Valid @RequestBody Person person) { ... return new ResponseEntity<>(newPerson, HttpStatus.OK); } } @Data @Schema( description = "A JSON request object to create Person" ) public class Person { @NotNull @Size(min = 4, max = 128) @Schema(example = "John") private String firstName; @NotNull @Size(min = 4, max = 128) @Schema(example = "Smith") private String lastName; @NotNull @Schema(type = "enum", example = "MALE") private Gender gender; @NotNull @Min(1) @Max(100) @Schema(type = "integer", example = "25") private int age; @NotEmpty @Email @Schema(example = "john.smith@smith.com") private String email; @NotNull @Valid private Address address; } @Data @Schema( example = """ { "address1" : "1240 E Diehl Rd.", "address2" : "#560", "city" : "Naperville", "state" : "IL", "zip" : "60563" } """ ) public class Address { @NotNull @Size(min = 4, max = 128) @Schema(example = "1 N Main St.") private String address1; @Schema(example = "Apt. 100") private String address2; @NotNull @Size(min = 4, max = 128) @Schema(example = "Sharon") private String city; @NotEmpty @Size(min = 2, max = 2) @Schema(example = "MA") private String state; @NotEmpty @Size(min = 5, max = 5) @Schema(example = "02067") private String zip; } public enum Gender { FEMALE, MALE }


would result into Swagger UI as shown below:


and request schema details look like:


Note that the specification and example is much cleaner with good data for all elements.

@NotNull, @NotEmpty, etc. - javax Validation Annotations

Also, javax field constraint annotations used for validation are very well considered. For instance all required fields (annotated with @NotNull or @NotEmpty) are marked as required elements with suffix * added to the element name.

Also, any invalid request results with more meaningful error response. In the example shown below, required field gender is missing and age has invalid value 0 sent in the request: 




@Operation - swagger ui annotation

Annotate resource operations (controller methods) to add more details. The summary element of this annotation can be leveraged to add a meaningful description about the operation. The default will not add any description. The tags element can be leveraged to logically group operations so that they all show up under that tag on the page. If not specified, the default tag value is hyphenated class-name, in the above code example (without annotations), the default tag value is: person-controller.

@Operation(summary = "Returns a list of MyDomain", tags = { "MyDomain" })

@ApiResponsse - swagger ui annotation

Further enhance Response descriptions by annotating controller method with @ApiResponses and describing every possible response code as shown below:

@ApiResponses(value = { @ApiResponse(responseCode = "200", description = "Returns list of MyDomains."), @ApiResponse(responseCode = "403", description = "Authorization key is missing or invalid."), @ApiResponse(responseCode = "400", description = "Invalid request.") })

Also, the class can be annotated with @ApiResponse annotation for describing all common response codes like 400, 401, 404, 500 etc. to keep annotations DRY. The controller methods can just describe 200 and any additional specific response codes. Also, can override class level annotated common response code descriptions. The following is an example annotation at the class level common for all controller methods:
 
@ApiResponses(value = { @ApiResponse( responseCode = "400", description = "Bad Request.", content = { @Content(mediaType = "application/json", schema = @Schema(implementation = Errors.class)) } ), @ApiResponse( responseCode = "401", description = "Unauthorized. Authorization key is missing or invalid.", content = { @Content(schema = @Schema(implementation = Void.class)) } ), @ApiResponse( responseCode = "404", description = "Not Found.", content = { @Content(schema = @Schema(implementation = Errors.class)) } ), @ApiResponse( responseCode = "500", description = "Internal Server Error.", content = { @Content(schema = @Schema(implementation = Errors.class)) } ) }) public class PersonController { ... @ApiResponses(value = { @ApiResponse(responseCode = "200", description = "Returns newly created Person.") }) @PostMapping(value = "/person", produces = { MediaType.APPLICATION_JSON_VALUE, MediaTypes.HAL_JSON_VALUE }) public ResponseEntity create(@Valid @RequestBody Person person) { ... return new ResponseEntity<>(newPerson, HttpStatus.OK); } }

For error responses with codes like 400, 404, 500 etc., that return spring Errors object for any kind of failures like validation, exceptions etc. the implementation class can be specified as shown above. If there no-content for any response like 401, then Void.class is suitable which results with no details/schema for the response.

@Schema - swagger ui annotation

Annotate request and response objects with this annotation to describe it. Also, annotate object properties to add data type and example data in order to enhance sample request with more meaningful data.

The type element of this annotation can be used to specify type data type and example element to specify example value. Otherwise, the value defaults to Java default. For enums it picks the first one in the list of enumerations.

For String data types, if there is a specific set of values expected the list can be specified as an array of Strings for allowedValues element. This shows up in the schema for that element as an enumeration of values allowed.

TIPS

  • @Schema annotation can be used at class level to specify a JSON representation of the object with meaningful data for all the object fields as an example as shown for the Address object in the code snippet above. When specified at this level it takes precedence over field level example data. I used Java 13 preview feature of multiline string in there.
  • http://localhost:8080/swagger-ui.html shows Swagger UI page of your application. It basically redirects to http://localhost:8080/swagger-ui/index.html?configUrl=/v3/api-docs/swagger-config.
  • http://localhost:8080/swagger-ui/index.html gives the Swagger UI page for pet-store based on https://petstore.swagger.io/v2/swagger.json. This is enabled by default. I have not found a way to disable this :(
  • If there is a collection property like List in the object, for instances a List<Address> address; then you need to annotate it as shown below:
@ArraySchema(schema = @Schema(implementation = Address.class)) List<Address> addresses;
  • Operations can be logically grouped by tags. Each tag can have a name and description properties. If annotations are used, then @Operation annotation can only take tag names, but no description. This is a limitation. The @Tag annotation supports both name, and description properties. So, if there are couple of operations that need to be grouped into one by one tag name, but also want to have a description, then one operation/method can use @Tag annotation with name and description, the other operation/method can use @Operation with tags property. This works and both operations get grouped under same tag name, and tag description is also shown along with tag name. So, @Tag and @Operation can mix and match across various operations/methods for the same tag group.
  • By default all response messages are generated for response codes: 200, 400, 403, 404, 405, 406, 500, 503 in the responses section of the page, though the method is annotated with @ApiResponses annotation for only response codes 200, 400, and 403. In order to fix this you need to add the following property in application.yml or application.properties appropriately (springdoc properties).
springdoc: override-with-generic-response: false

Tuesday, September 05, 2017

Create a Secured Restful API App with Grails 3 and PostgreSQL - Step by Step: Part 5 of 5

Part 5: Assure REST & Publish your API

At the end of my last post, we had a RESTful application with an end-point fully implemented and secured. However, we had not written any unit/integration test-specs. In this post we will write integration test-spec and mix it with REST Assured and Spring REST Docs to not only test the end-point but also to generate and publish API documents.

Importance of API Documentation

API is the middleware now. By following standard principles, the behavior of API can be consistent and predictable. Like any piece of code, APIs must be tested by all means from writing test-cases to assuring it's quality by QA. Ideally, when the application is assembled, API docs must also be generated and bundled into the artifact and be delivered together for deployment. That makes API docs a "living source of documents" and they become a common source of reference.

Spring REST Docs

There are few popular frameworks/tools to generate API documentation. Each one of such frameworks/tools has it's own adopted approach and comes with it's own benefits and drawbacks when compared with others. However, Spring IO has a project called Spring REST Docs that uniquely takes a very different approach. It's approach is centered around testing and is combined with hand-written Asciidoctor templates to produce high quality and maintainable API documentation. This approach definitely stands out as it promotes testing to it's greatest levels.

With the test-centric approach, it makes API document not only accurate-and-complete but also up-to-date and a living-resource reference. API documents generated this way are always as accurate as the code-base is. Also, as Spring framework is central to Grails framework, it becomes a natural-fit for Grails applications.

Having said all that, let's add Spring REST Docs to Grails 3 project and assure it with REST Assured.

Environment: Grails 3.2.11, Java 1.8, Apache Tomcat 8.0.20, IntelliJ IDEA Ultimate 2017.2 on Mac OS X 10.11.5 (El Capitan)

Step 0: Upgrade application from Grails 3.1.6 to 3.2.11

When I started this multi-part posts, Grails was at 3.1.6 and now it has advanced to 3.3.x. Just to catch up, I've upgraded this app from 3.1.6 to 3.2.11 (the latest on 3.2.x branch). It was an easy upgrade as it is a simple RESTful application. All I had to do was to bring gradle.properties and build.gradle files up-to-date with 3.2.11.

Step 1: Add Spring REST Docs to the Project (build configuration)

At the end of my last post, we had a secured resource (Artist) and we tested it's RESTful API for CRUD operations. That is good enough resource for taking it to the next level of generating & publishing it's API. Spring REST Docs' Getting Started has link to sample applications for reference. REST Assured Grails is the best-bet and is the basis for us. As a first step let's add Spring REST Docs support to the project as shown and described below:

Add Asciidoctor plugin.
build.gradle
plugins { ... id 'org.asciidoctor.convert' version '1.5.3' }

Run gradle tasks command and notice that asciidoctor task gets added by the plugin.
$./gradlew tasks ... Documentation tasks ------------------- asciidoctor - Converts AsciiDoc files and copies the output files and related resources to the build directory. groovydoc - Generates Groovydoc API documentation for the main source code. ... javadoc - Generates Javadoc API documentation for the main source code.

Spring REST Docs Build Configuration section has steps for Gradle build configuration. I will do this slightly different to extend build script for REST docs support by separating out additional build configuration for REST docs into it's own build file leveraging Gradle's script plugin concept. This way it is more cleaner and brings in some modularity to the build script.

Create a new build script file restdocs.gradle under project's gradle dir and reference it in the main build.gradle file at the very bottom as shown below:
build.gradle
apply from: 'gradle/restdocs.gradle'

Let's populate restdocs.gradle as shown below. I will add comments into the build script to explain certain code blocks.
gradle/restdocs.gradle
buildscript { repositories { maven { url 'https://repo.spring.io/libs-snapshot' } } } repositories { maven { url 'https://repo.spring.io/libs-snapshot' } } //add extra user-defined properties to the project through ext block ext { snippetsDir = file('build/docs/generated-snippets') //output dir of rest api doc snippets generated restDocsVersion = '2.0.0.BUILD-SNAPSHOT' restAssuredVersion = '2.9.0' } dependencies { testCompile "org.springframework.restdocs:spring-restdocs-core:$restDocsVersion" testCompile "org.springframework.restdocs:spring-restdocs-restassured:$restDocsVersion" testCompile "org.springframework.restdocs:spring-restdocs-asciidoctor:$restDocsVersion" }

Now, just run grails clean command. We will have spring-restdocs-core and spring-restdocs-restassured downloaded from maven central repo.

Let's keep expanding this script.
//task to clean generated rest api docs snippets dir task cleanSnippetsDir(type: Delete){ delete fileTree(dir: snippetsDir) }

Run ./gradlew tasks and notice that there is a new task added under Other tasks like:
Other tasks ----------- cleanIdeaWorkspace cleanSnippetsDir console

Configure test task as shown below:
test { dependsOn cleanSnippetsDir outputs.dir snippetsDir }

Now run ./gradlew test -m or ./gradew test --dry-run which will run gradle's test task in a dry run mode. It disables all tasks and shows the order in which tasks get executed. In this case, we can now see our new task cleanSnippetsDir in the list after all classes are created and before test-case classes get compiled.

Remember we got asciidoctor task by adding Gralde plugin as the very first step. We will customize it and specify that it depends on integrationTest task. With this dependency, every time when we run this task, it will have integration tests run. We want this kind of dependency as the approach that REST Docs brings in is to have REST API docs generated from the integration test cases. So, we need integration tests to run before we have docs generated.

Having said that, let's customize that task as follows:
//Configure asciidoctor task provided by Gradle asciidoctor plugin- https://github.com/asciidoctor/asciidoctor-gradle-plugin asciidoctor { doFirst{ //just print outputDir for reference during execution phase println "Running asccidoctor task. Check generated REST docs under: ${outputDir}" } dependsOn integrationTest logDocuments = true sourceDir = file('src/docs') inputs.dir snippetsDir separateOutputDirs = false attributes 'snippets': snippetsDir //configure snippets attribute for .adoc files }

Step 2: Run tests and make them pass: grails test-app

Grails test-app runs both unit tests and integration tests.
I have not written any test specifications so far but as part of creating domain objects using grails create-domain-class command, I have a few Spock Specification unit-tests created each with a default feature method "test something"(). All these default generated specifications are expected to fail to start with. I want to keep these tests around for future but want to make them pass. An easy way is to annotate all those methods with groovy's @NotYetImplemented annotation. It reverses the net result by making it pass when it actually fails. It makes sense for an un-implemented test. But when actually implemented, it fails forcing us to remove the annotation.

Spock's @PendingFeature is similar but is added only in Spock 1.1. Grails 3.2.x comes with Spock 1.0. For now, we are all good with that wonderful annotation provided by groovy. With this we have all unit-tests passing.

It's time now to write an integration test specification for our RESTful controller: ArtistController. Instead of writing a typical integration test-case, let's mix it with REST assured and REST API Docs and get both testing and API docs generation done in this phase.

Step 3: Assure REST by writing integration specification for RESTful Controller with a mix of REST Docs and a touch of REST assured.

Step 3a: Configure REST Assured testing framework (set up your test specification to generate documentation snippets)

The Spring REST Docs documentation has outlined these steps. Here is the gist of it:

The configuration of REST Assured is nothing but a request spec (RequestSpecifiction) using ResqusetSpecBuilder by adding documentation configuration as a JUnit filter to it.

Configure REST assured documentation output directory by declaring a restDocumentation field which is initialized with an instance of JUnitRestDocumentation and annotate it with JUnit's @Rule annotation. This rule gets executed before and after each feature method. A custom output directory can be specified by passing a constructor argument. We specify this custom dir, as in the build file, the snippetsDir property we set with is slightly different ('build/docs/generated-snippets') than the default ('build/generated-snippets').

Next, setup RequestSpecification by adding a filter and configure it with the above restDocumentation initialized as JUnit Rule.

Here is how our test spec looks after this configuration:
src/integration-test/groovy/com/giri/ApiDocumentationArtistSpec
package com.giri import geb.spock.GebSpec import grails.plugins.rest.client.RestBuilder import grails.test.mixin.integration.Integration import grails.transaction.Rollback import io.restassured.builder.RequestSpecBuilder import io.restassured.specification.RequestSpecification import org.junit.Rule import org.springframework.restdocs.JUnitRestDocumentation import static org.springframework.http.HttpStatus.* import static org.springframework.restdocs.restassured3.RestAssuredRestDocumentation.documentationConfiguration @Integration @Rollback class ApiDocumentationArtistSpec extends GebSpec { @Rule protected JUnitRestDocumentation restDocumentation = new JUnitRestDocumentation('build/docs/generated-snippets') private RequestSpecification documentationSpec def setup() { //set documentation specification this.documentationSpec = new RequestSpecBuilder().addFilter( documentationConfiguration(this.restDocumentation)) .build() } ...

Step 3b: Spockify, test RESTful end-point and get documentation snippets generated

With the above configuration, let's write a feature method to test GET request of /api/artists end-point. The following is a feature method added to the above specification along with a defined static constant whose value is set with relative end-point url and an injected application port property. The port is required to override the default port(8080) of REST assured testing framework. Note that grails start the application on a random available port each time when integration tests are run.

static final String ARTISTS_ENDPOINT = '/api/artists' @Value('${local.server.port}') protected int port ... void "test and document GET request (index action) of end-point: /api/artists"() { given: "" RequestSpecification requestSpecification = RestAssured.given(this.documentationSpec) .accept(MediaType.APPLICATION_JSON_VALUE) .filter( RestAssuredRestDocumentation.document( 'artists-list-example' ) ) when: def response = requestSpecification .when() .port(port) .get(ARTISTS_ENDPOINT) then: response.then() .assertThat() .statusCode(HttpStatus.OK.value()) }

The feature method name describes the intent of this feature method. In this step, we are only testing the GET request of an end-point. We will add the support for the highlighted and document intent of this feature.

With this, if you run grails dev test-app or grails -Dgrails.env=development test-app, the test will pass. Also, we will have the following six documentation snippets generated under build/docs/generated-snippets/artists-list-example directory:
 curl-request.adoc
 http-request.adoc
 http-response.adoc
 httpie-request.adoc
 request-body.adoc
 response-body.adoc

These are the snippet files to be included in the final API documentation. Just to see the contents check http-response.adoc and it will contain the actual response received as follows:
---- HTTP/1.1 200 OK X-Application-Context: application:development:0 Content-Type: application/json;charset=UTF-8 Transfer-Encoding: chunked Date: Tue, 29 Aug 2017 22:12:13 GMT Content-Length: 148 [{"id":"90ff9ac4-b1c0-4495-94d5-1550f463561a","dateCreated":"08/29/2017","firstName":"Giridhar","lastName":"Pottepalem","lastUpdated":"08/29/2017"}] ----

Step 3c: Create asciidoctor (.adoc) source templates
Create src/docs dir and create api-guide.adoc and artists.adoc files to start with. The api-guide.adoc is the main asciidoctor template which will include artists.adoc. The artists.adoc is the asciidoctor template for artists end-point.

Shown below is a portion of api-guide.adoc
= giri-api RESTful API Guide Giridhar Pottepalem :doctype: book :icons: font :source-highlighter: highlightjs :toc: left :toclevels: 4 :sectlinks: [[overview]] = Overview [[overview-http-verbs]] == HTTP Methods giri-api API follows standard HTTP and REST conventions as closely as possible in its exposure of resources as end-points and use of HTTP methods (verbs). ... [[resources]] = Resources include::artists.adoc[]

And portions of artists.adoc is shown below for creating an Artist (POST request/save action):
[[resources-artists]] == Artists An Artist is a resource which represents an Artist. [[resources-artists-create]] === Creating an Artist A `POST` request is used to create a new Artist. TIP: An Artist can be created only by an Admin user (with role `ROLE_ADMIN`) IMPORTANT: Once a new Artist is created... ==== Request structure include::{snippets}/artists-create-example/request-fields.adoc[] ==== Example request include::{snippets}/artists-create-example/curl-request.adoc[] ==== Response structure include::{snippets}/artists-create-example/response-fields.adoc[] ==== Example response include::{snippets}/artists-create-example/http-response.adoc[]
Highlighted are the references to generated snippets that get included in the generated end HTML5 doc.

Step 3d: Generate API doc
Now, lets run asciidoctor gradle task we got added through Step 1 as shown below:
./gradlew asciidoctor //runs in test env
./gradlew -Dgrails.env=development asciidoctor //runs in dev env

This task runs all integration test specifications because we configured it to depend on integrationTest task. Once it's run successfully with no failing tests, it converts our asciidoctor API templates to HTML5 doc by populating it with included generated snippets as we referenced in artists.adoc.

Now let's enhance our specification feature method to document request and response payload structure. Let's take the case of /api/artists end-point and GET request. There is no request payload for this request. So, we will simply add response payload specification as shown below:
void "Test and document show Artist request (GET request, show action) to end-point: /api/artists"() { given: "Pick an artist to show" Artist artist = Artist.first() and: "user logs in by a POST request to end-point: /api/login" String accessToken = authenticateUser('me', 'password') and: "documentation specification for showing an Artist" RequestSpecification requestSpecification = RestAssured.given(this.documentationSpec) .accept(MediaType.APPLICATION_JSON_VALUE) .filter( RestAssuredRestDocumentation.document( 'artists-retrieve-specific-example', PayloadDocumentation.responseFields( PayloadDocumentation.fieldWithPath('id').type(JsonFieldType.STRING).description('Artist id'), PayloadDocumentation.fieldWithPath('firstName').type(JsonFieldType.STRING).description('Artist first name'), PayloadDocumentation.fieldWithPath('lastName').type(JsonFieldType.STRING).description('Artist last name'), PayloadDocumentation.fieldWithPath('dateCreated').type(JsonFieldType.STRING).description("Date Created (format:MM/dd/yyyy)"), PayloadDocumentation.fieldWithPath('lastUpdated').type(JsonFieldType.STRING).description("Last Updated Date (format:MM/dd/yyyy)") ) ) ) when: "GET request is sent" def response = requestSpecification .header("X-Auth-Token", "${accessToken}") .when() .port(this.port) .get("${ARTISTS_ENDPOINT}/${artist.id}") def responseJson = new JsonSlurper().parseText(response.body().asString()) then: "The response is correct" response.then() .assertThat() .statusCode(HttpStatus.OK.value()) and: "response contains the id of Artist asked for" responseJson.id }

Similarly, we can write a test spec to test and document POST method (creating an Artist) as shown below. Remember, I have secured this method to role: ROLE_ADIN. So, it requires admin to be authenticated first to get a security token and then pass the security token in the subsequent secured requests like POST. The following is the complete test specification with a helper method added to authenticate the user:

/** * Helper method, authenticates the given user and returns the security token. * * @param username the user id * @param password the password * @return security token once successfully authenticated */ protected String authenticateUser(String username, String password) { String authResponse = RestAssured.given() .accept(MediaType.APPLICATION_JSON_VALUE) .contentType(MediaType.APPLICATION_JSON_VALUE) .body(""" {"username" : "$username", "password" : "$password"} """) .when() .port(this.port) .post(LOGIN_ENDPOINT) .body() .asString() return new JsonSlurper().parseText(authResponse).'access_token' } void "Test and document create Artist request (POST request, save action) to end-point: /api/artists"() { given: "current number of Artists" int nArtists = Artist.count() and: "admin logs in by a POST request to end-point: /api/login" String accessToken = authenticateUser('admin', 'admin') and: "documentation specification for creating an Artist" RequestSpecification requestSpecification = RestAssured.given(this.documentationSpec) .accept(MediaType.APPLICATION_JSON_VALUE) .contentType(MediaType.APPLICATION_JSON_VALUE) .filter( RestAssuredRestDocumentation.document( 'artists-create-example', PayloadDocumentation.requestFields( PayloadDocumentation.fieldWithPath('firstName').description('Artist first name'), PayloadDocumentation.fieldWithPath('lastName').description('Artist last name') ), PayloadDocumentation.responseFields( PayloadDocumentation.fieldWithPath('id').type(JsonFieldType.STRING).description('Artist id'), PayloadDocumentation.fieldWithPath('firstName').type(JsonFieldType.STRING).description('Artist first name'), PayloadDocumentation.fieldWithPath('lastName').type(JsonFieldType.STRING).description('Artist last name'), PayloadDocumentation.fieldWithPath('dateCreated').type(JsonFieldType.STRING).description("Date Created (format:MM/dd/yyyy)"), PayloadDocumentation.fieldWithPath('lastUpdated').type(JsonFieldType.STRING).description("Last Updated Date (format:MM/dd/yyyy)") ) ) ) when: "POST request is sent with valid data" def response = requestSpecification .header("X-Auth-Token", "${accessToken}") .body("""{ "firstName" : "Bhuvan", "lastName" : "Pottepalem" }""") .when() .port(this.port) .post(ARTISTS_ENDPOINT) def responseJson = new JsonSlurper().parseText(response.body().asString()) then: "The response is correct" response.then() .assertThat() .statusCode(HttpStatus.CREATED.value()) and: "response contains the id of Artist created" responseJson.id and: "Number of Artists in the system goes up by one" Artist.count() == nArtists + 1 }

Now, simply run
./gradlew asciidoctor

We will have API docs generated under build/asciidoc dir. Open api-guide.html in a browser to see how nicely the generated API doc looks.

TIP: The beauty of Spring REST Docs framework is that, if compares the actual request/response fields with the PayloadDocumentation filed descriptions and will fail the test if any field(s) are missed or mis-matched. This ensures that the API documentation is up-to-date with the implementation.

Step 4: Publish API
Now, we have fully integrated REST Assured and Spring REST Docs into integrationTest phase with an added asciidoctor Gradle test task. The result of this is an up-to-date API document generated for our Restful service.The API document is the source for clients using this service. So, it needs to be made available. One way to achieve this is to bundle the generated HTML5 API docs with the application's deployable war or executablejar and have it's own end-point to serve it.

Spring Boot (the framework Grails3 underpins) can be leveraged to achieve this. By default Boot serves static content placed under /static or /public in the class path or root of the application context. Here is the link for reference: Spring boot Static content.

Step 4a: Bundle API documentation into deployable artifact
We will enhance our build script (restdocs.gradle) and customize war task that comes with Gradle Java plugin little bit to achieve this. Below is the code snippet which is self explanatory:
/* Bundles generated API docs into war file. * Spring boot serves static content under /public or /static or /resources or /META-INF/resources. * Hooks into war task and adds asciidoctor task dependency, also copies generaed rest docs appropriately * for bundling into war file. */ def publicDocsDir = 'WEB-INF/classes/public/docs' war { dependsOn asciidoctor from ("${asciidoctor.outputDir}") { into publicDocsDir } }

We basically made war task depend on asciidoctor task and added a step to copy generated HTML5 API docs to WEB-INF/classes/public/docs dir in the generated war file.

Now, run grails war to generate deployable war artifact:
grails war

You can explode and see that generated API docs are bundled into the war generated (giri-api-0.1.war):
e.g. jar tvf build/libs/giri-api-0.1.war | grep html will list the following:
 59738 Mon Sep 04 07:16:34 EDT 2017 WEB-INF/classes/public/docs/api-guide.html
 47974 Mon Sep 04 07:16:34 EDT 2017 WEB-INF/classes/public/docs/artists.html

Step 4b: Make API documentation available from it's own end-point
Deploy the generated war file onto a locally running tomcat.
Deploy the war onto locally running Tomcat and point your browser at: http://localhost:8080/giri-api-0.1/static/docs/api-guide.html

This will result into Access Denied error. We need to open up security to serve API docs.

Lets change application.groovy and add /static/docs/** to both grails.plugin.springsecurity.controllerAnnotations.staticRules and filterChainChainMaps as shown below:
grails.plugin.springsecurity.controllerAnnotations.staticRules = [ ... [pattern: '/static/docs/**', access:['permitAll']] ] def filterChainChainMaps = [ ... pattern: '/static/docs/**', filters: statelessFilters], ... ]

Create a war file. Undeploy previously deployed war and deploy the latest war file.

Now, http://localhost:8080/giri-api-0.1/static/docs/api-guide.html (API docs) should be served and displayed by the app.

The test specification can be enhanced easily along these lines to test and document rest of the service methods: show, update and delete available for /api/artists end-point through HTTP methods GET specific resource by id, UPDATE and DELETE respectively.

The complete source code is hosted on GitHub at https://github.com/gpottepalem/giri-api for reference.

References

Saturday, June 25, 2016

Create a Secured Restful API App with Grails 3 and PostgreSQL - Step by Step: Part 2 of 5

Part 2: Add Core Security and REST API Security 

In Part 1, I created a Grails 3 application using rest-api profile. In this part, I will list steps to add security. The Grails Spring Security Core plugin is one of the very popular Grails plugins and when Grails 3 was released, this was the most desired plugin by the Grails community to be migrated to Grails 3. Thanks to Burt Beckwith!

Security

Typically, web application's resources (pages/urls) are secured through a login form. Spring Security framework is a very flexible framework and meets most of web application's security needs. Though it simplifies many aspects of security, it comes with some level of complexities.

In Grails world, the popular Grails Spring Security Core plugin which underpins Spring Security makes security trivial. However, the core plugin is not just enough for securing RESTful Grails application. In a RESTful application, all end-points need to be secured and a stateless/session-less mechanism for safely accepting HTTP requests through HTTP methods to urls should be in place. The Grails Spring Security REST plugin which is written on top of Grails Spring Security Core plugin makes this possible in a simple manner. It supports token-based RESTful authentication. It also supports JSON Web Token (JWT).

Let's try some hands on...

Environment: Grails 3.1.6, Java 1.8, IntelliJ 15 on Mac OS X 10.9.5

Before I start adding Security, I will import the grails app created into my IntelliJ IDEA. IntelliJ 15 supports Grails 3. Also, Grails 3 switched it's build system from GANT to Gradle. So the Grails 3 app created can easily be imported as a Gradle project. IntelliJ recognizes it as a Grails project and adds all needed support.

To import into IntelliJ IDEA 15, simply follow these steps:

1. Open IntelliJ IDEA and go to File > New > Project from existing sources... 2. Select build.gradle file under the grails3 app created (in my example it's giri-api) 3. IntelliJ recognizes it as a Gradle project. Go with default options in the next step. If multiple JVMs are available in IntelliJ select the one you prefer for Gradle JVM. I have 1.6, 1.7 and 1.8 and I selected 1.8. Press OK. 4. IntelliJ recognizes it as Gradle project, imports and builds the project. It also recognizes the project dir as root module and prompts for modules. Just press OK. 5. It imports the project and prompts for Grails shell. Click the link 'Select Grails SDK Home'. I have multiple Grails versions installed and I select  ~/.gvm/grails/3.1.6 6. If you miss step-5 due to any reason, you can add Grails SDK Home from: File > Other Settings > Default Settings > Languages & Frameworks > Grails

Also, notice that IntelliJ adds a run configuration with the project name to run the main Application.groovy class. This lets you run the grails application from IntelliJ. Just click Run > Edit Configurations menu item or the drop down from the run configurations icon and take a look at the configuration. You can add more run configurations and setup various grails commands to run directly from IntelliJ.

Step 1 Add Security - Grails Spring security core plugin dependency to the application's build.gradle file

dependencies { ... //grails security-core plugin compile "org.grails.plugins:spring-security-core:3.1.0" ... }

Step 2 Generate security related classes quickly

This step is documented well in the quick start section of the plugin documentation.

Minimally, we need User and Role domain classes for security. As domain classes have equivalent persistent db tables created by Grails, some databases wouldn't allow to have tables with names that are reserved words. At least, PostgresSQL will not let a table named User. So we I will use AppUser instead.

From the project home directory execute the following:
$ grails s2-quickstart com.giri.security AppUser Role BUILD SUCCESSFUL | Creating User class 'AppUser' and Role class 'Role' in package 'com.giri.security' | Rendered template Person.groovy.template to destination grails-app/domain/com/giri/security/AppUser.groovy | Rendered template Authority.groovy.template to destination grails-app/domain/com/giri/security/Role.groovy | Rendered template PersonAuthority.groovy.template to destination grails-app/domain/com/giri/security/AppUserRole.groovy | ************************************************************ * Created security-related domain classes. Your * * grails-app/conf/application.groovy has been updated with * * the class names of the configured domain classes; * * please verify that the values are correct. * ************************************************************

The above command generates 3 classes: AppUser.groovy, Role.groovy and AppUserRoles.groovy under grails-app/domain/com/giri/security directory.

Also, take a look at application.groovy. It will have security plugin's default properties added. Some of these properties and values are obvious. We don't need to change anything at this time. Just make a note of grails.plugin.springsecurity.filterChain.chainMap, this needs some attention later when we secure rest end points using annotations.
// Added by the Spring Security Core plugin: grails.plugin.springsecurity.userLookup.userDomainClassName = 'com.giri.security.AppUser' grails.plugin.springsecurity.userLookup.authorityJoinClassName = 'com.giri.security.AppUserRole' grails.plugin.springsecurity.authority.className = 'com.giri.security.Role' grails.plugin.springsecurity.controllerAnnotations.staticRules = [ [pattern: '/', access: ['permitAll']], [pattern: '/error', access: ['permitAll']], [pattern: '/index', access: ['permitAll']], [pattern: '/index.gsp', access: ['permitAll']], [pattern: '/shutdown', access: ['permitAll']], [pattern: '/assets/**', access: ['permitAll']], [pattern: '/**/js/**', access: ['permitAll']], [pattern: '/**/css/**', access: ['permitAll']], [pattern: '/**/images/**', access: ['permitAll']], [pattern: '/**/favicon.ico', access: ['permitAll']] ] grails.plugin.springsecurity.filterChain.chainMap = [ [pattern: '/assets/**', filters: 'none'], [pattern: '/**/js/**', filters: 'none'], [pattern: '/**/css/**', filters: 'none'], [pattern: '/**/images/**', filters: 'none'], [pattern: '/**/favicon.ico', filters: 'none'], [pattern: '/**', filters: 'JOINED_FILTERS'] ]

Gotcha

When I ran s2-quickstart command for generating security domain classes, I passed package com.giri.security explicitly. Otherwise, grails uses the default package and the defaultPackage is defined in application.yml under elements grails: codegen: Out-of-the-box, grails uses application name as the default package with hiphens (-) converted to dots (.), if hiphens are used in the application name.  My application name is giri-api and hence when I created the app, the defaultPackage was set to giri.api in application.yml. You can change it to the default package you would like to have. Once changed, any grails artifacts generated using grails create-xxxxx commands use this as the default package and generate all classes under that package. I changed the default to com.giri for convenience.

Step 3 Run the app

$ grails run-app Configuring Spring Security Core ... ... finished configuring Spring Security Core Grails application running at http://localhost:8080 in environment: development

Point your browser at: http://localhost:8080
It will return {"message":"Internal server error","error":500}
Check the console, you will see:
ERROR org.apache.catalina.core.ContainerBase.[Tomcat].[localhost].[/].[grailsDispatcherServlet] - Servlet.service() for servlet [grailsDispatcherServlet] in context with path [] threw exception [Could not resolve view with name '/login/auth' in servlet with name 'grailsDispatcherServlet'] with root cause
javax.servlet.ServletException: Could not resolve view with name '/login/auth' in servlet with name 'grailsDispatcherServlet'

This is because, now the application is secured and as the security plugin added a chain of filters, the request is redirected to a login page. we do not have a login page yet. We do not need one anyway as this is a RESTful API application.

Step 4 Add roles and users

As outlined in the security plugin documentation in Bootstrap.groovy, add the following code:
import com.giri.security.AppUser import com.giri.security.AppUserRole import com.giri.security.Role class BootStrap { def init = { servletContext -> def init = { def adminRole = new Role('ROLE_ADMIN').save() def userRole = new Role('ROLE_USER').save() def testUser = new AppUser('me', 'password').save() def adminUser = new AppUser('admin', 'admin').save() AppUserRole.create testUser, userRole AppUserRole.create adminUser, adminRole AppUserRole.withSession { it.flush() it.clear() } assert AppUser.count() == 2 assert Role.count() == 2 assert AppUserRole.count() == 2 } } def destroy = { } }

Tip
You can even make the above code to create roles and users better with GORM's findOrSaveByXxxx() methods.

def adminRole = Role.findOrSaveByAuthority('ROLE_ADMIN') def userRole = Role.findOrSaveByAuthority('ROLE_USER') def testUser = AppUser.findOrSaveByUsernameAndPassword('me', 'password') def adminUser = AppUser.findOrSaveByUsernameAndPassword('admin', 'admin')

Step 5 Start the application

The application results with the following exception:
ERROR org.springframework.boot.SpringApplication - Application startup failed groovy.lang.GroovyRuntimeException: Could not find matching constructor for: com.giri.security.Role(java.lang.String)

Fix
Add the following missing constructors for AppUser.groovy and Role.groovy
AppUser(String username, String password) { this() this.username = username this.password = password } Role(String authority) { this() this.authority = authority }

Step 6 Add REST Security API plugin with GORM support

Add REST API Security plugin dependency to build.gradle file
ext{ springSecurityRestVersion = '2.0.0.M2' } dependencies { //grails REST API Security plugin compile "org.grails.plugins:spring-security-rest:${springSecurityRestVersion}" compile "org.grails.plugins:spring-security-rest-gorm:${springSecurityRestVersion}" }

Step 7  Restart the application

$ grails run-app Configuring Spring Security Core ... ... finished configuring Spring Security Core Configuring Spring Security REST 2.0.0.M2... ... finished configuring Spring Security REST ... with GORM support Grails application running at http://localhost:8080 in environment: development
Notice the highlighted.

Step 8 Add plugin chainMap property in application.groovy file as described in the documentation

Add the following as outlined in the plugin documentation:
grails.plugin.springsecurity.filterChain.chainMap = [ //Stateless chain [ pattern: '/api/**', filters: 'JOINED_FILTERS,-anonymousAuthenticationFilter,-exceptionTranslationFilter,-authenticationProcessingFilter,-securityContextPersistenceFilter,-rememberMeAuthenticationFilter'], //Traditional chain //[ pattern: '/**', filters: 'JOINED_FILTERS,-restTokenValidationFilter,-restExceptionTranslationFilter'] ]

Step 9 Generate domain class AuthenticationToken as described in the plugin documentation


$ grails create-domain-class com.giri.security.AuthenticationToken | Created grails-app/domain/com/giri/security/AuthenticationToken.groovy | Created src/test/groovy/com/giri/security/AuthenticationTokenSpec.groovy

Step 10 Update AuthenticationToken.groovy

Open file AuthenticationToken.groovy and update as shown below:
class AuthenticationToken { String tokenValue String username static mapping = { version false } static constraints = { } }

Step 11 Open application.groovy and add the following plugin related properties:

grails.plugin.springsecurity.rest.token.storage.useGorm = true grails.plugin.springsecurity.rest.token.storage.gorm.tokenDomainClassName = 'com.giri.security.AuthenticationToken'

Step 12 Restart the application and point your browser at: http://localhost:8080

Now, you will see the default JSON response:


Step 13 Check dbconsole

Grails comes with database console for the in-memory database H2. By default when the application is run, it runs in development environment and H2 in-memory database is used.

Point your browser at http://localhost:8080/dbconsole to see the in-memory H2 database console. Select Generic H2 Embedded and click connect.  Notice that you will have 4 tables: APP_USER, ROLE, APP_USER_ROLE and AUTHENTICATION_TOKEN
Just click connect and notice all the domain classes added now have corresponding tables created:

Step 14 Test it

The Grails Security REST API plugin provides three end-points: /api/login for login, /api/logout for logout and /api/validate for validating the token.

Let's test it little bit. Since it is a RESTful application, we do not have regular gsp pages/views. We need to send RESTful requests to test our application.

Let's use Curl - a simple command line tool that comes with Mac and is also available for Windows.

Login - Send a POST request with right user credentials

Run the following command to test end-point /api/login. Highlighted is the response received.
$ curl -i -H "Accept: application/json" -H "Content-Type: application/json" -X POST -d '{"username":"me","password":"password"}' http://localhost:8080/api/login HTTP/1.1 200 OK Server: Apache-Coyote/1.1 Cache-Control: no-store Pragma: no-cache Content-Type: application/json;charset=UTF-8 Content-Length: 112 Date: Sun, 19 Jun 2016 15:12:48 GMT {"username":"me","roles":["ROLE_USER"],"token_type":"Bearer","access_token":"664dkbafcuo4prsd02vocvlfvaok5nvl"}

Login - Send a POST request with wrong user credentials

Run the following command to test end-point /api/login. Highlighted is the response received.

$ curl -i -H "Accept: application/json" -H "Content-Type: application/json" -X POST -d '{"username":"me","password":"passwordWrong"}' http://localhost:8080/api/login HTTP/1.1 401 Unauthorized Server: Apache-Coyote/1.1 WWW-Authenticate: Bearer Content-Length: 0 Date: Sun, 19 Jun 2016 15:19:02 GMT

Notice the Unauthorized response

The plugin also has an endpoint to validate the token received upon successful login.

Validate token

Run the following command to test end-point /api/validate. Highlighted is the response received.

$ curl -i -H "Authorization: Bearer 664dkbafcuo4prsd02vocvlfvaok5nvl" http://localhost:8080/api/validate HTTP/1.1 200 OK Server: Apache-Coyote/1.1 Cache-Control: no-store Pragma: no-cache Content-Type: application/json;charset=UTF-8 Content-Length: 112 Date: Tue, 21 Jun 2016 01:03:05 GMT {"username":"me","roles":["ROLE_USER"],"token_type":"Bearer","access_token":"664dkbafcuo4prsd02vocvlfvaok5nvl"}

Highlighted is the response.

NOTE
"Authorization: Bearer <token>" is the default header to be passed.
Set the following in application.grovy to change the header.
grails.plugin.springsecurity.rest.token.validation.useBearerToken = false grails.plugin.springsecurity.rest.token.validation.headerName = 'X-Auth-Token'

If you change the header name as above, your request to login should be like:
$ curl -i -H "X-Auth-Token: 664dkbafcuo4prsd02vocvlfvaok5nvl" http://localhost:8080/api/validate HTTP/1.1 200 OK Server: Apache-Coyote/1.1 Cache-Control: no-store Pragma: no-cache Content-Type: application/json;charset=UTF-8 Content-Length: 90 Date: Tue, 21 Jun 2016 01:20:36 GMT {"username":"me","roles":["ROLE_USER"],"access_token":"664dkbafcuo4prsd02vocvlfvaok5nvl"}

Highlighted is the response

Validate wrong token Run the following. Highlighted is the response


$ curl -i -H "X-Auth-Token: wrong-token-anyway" http://localhost:8080/api/validate HTTP/1.1 401 Unauthorized Server: Apache-Coyote/1.1 Content-Length: 0 Date: Tue, 21 Jun 2016 01:23:38 GMT

Logout Let's test the logout end-point: /api/logout. Run the following and highlighted is the response.


$ curl -i -H "X-Auth-Token: 664dkbafcuo4prsd02vocvlfvaok5nvl" http://localhost:8080/api/logout HTTP/1.1 403 Forbidden Server: Apache-Coyote/1.1 Content-Type: application/json;charset=UTF-8 Transfer-Encoding: chunked Date: Tue, 21 Jun 2016 01:26:40 GMT {"timestamp":1466472400548,"status":403,"error":"Forbidden","message":"Access is denied","path":"/api/logout"}

Surprisingly we received 403 Forbidden.

Gotcha
For logout to work, add the following pattern to the list of static rules in application.groovy
grails.plugin.springsecurity.controllerAnnotations.staticRules = [ //omitted the rest for brevity [pattern: '/api/logout', access: ['isAuthenticated()'] ]

By adding the above pattern to staticRules, we are specifying to allow the end-point api/logout only if the user is authenticated. Now try to logout.

curl -i -H "X-Auth-Token: 664dkbafcuo4prsd02vocvlfvaok5nvl" http://localhost:8080/api/logout HTTP/1.1 405 Method Not Allowed Server: Apache-Coyote/1.1 X-Application-Context: application:development Content-Length: 0 Date: Tue, 21 Jun 2016 01:34:48 GMT

Highlighted is the response. This time it's 405 Method not allowed.

The reason: We used a GET request to logout. The security plugin has a property grails.plugin.springsecurity.logout.postOnly which is set to true by default. So, it only allows POST request to logout.

Try the following now

$ curl -i -H "X-Auth-Token:664dkbafcuo4prsd02vocvlfvaok5nvl" -X POST http://localhost:8080/api/logout HTTP/1.1 200 OK Server: Apache-Coyote/1.1 X-Application-Context: application:development Content-Length: 0 Date: Tue, 21 Jun 2016 01:36:29 GMT

Highlighted is the response. Now it successfully logged me out.

Now double check by running the validate end-point
$ curl -i -H "X-Auth-Token:664dkbafcuo4prsd02vocvlfvaok5nvl" http://localhost:8080/api/validate HTTP/1.1 401 Unauthorized Server: Apache-Coyote/1.1 Content-Length: 0 Date: Tue, 21 Jun 2016 01:49:02 GMT

Highlighted is the response. Notice that the token that was valid now resulted with 401 Unauthorized.

TIP
To make the logout work with GET request, set grails.plugin.springsecurity.logout.postOnly = false in application.groovy. The GET request to logout should work.

TIP
If you are not comfortable with the command line tool like curl for testing, thee are several graphical tools available. I use this chrome plugin called DTC REST Client from Restlet which is really nice, easy to use and also all API tests can be organized, saved and shared with your team.

Step 15 Lockdown Spring Boot Actuator end-points

Grails 3 underpins Spring Boot. Spring Boot has a module called Actuator which provides various end-points for monitoring and managing a running application. In Part-1 side note, I mentioned about it. These end-points are available at root the application context path.

The stateless chainMap pattern that we added to secure all our /api/** end-points won't secure Actuator end points. In reality, it makes sense to secure Actuator provided application monitoring and management  end-points. There could be many ways to secure these end-points. Here is what I tried which worked.
  • Customize the Actuator end-points path by adding the following spring property in application.yml
  • management: context-path: /api/management
  • With the above spring property we changed end-points path from application root context to '/api/manamgement'. With this change for example the earlier http://localhost:8080/info will be available at http://localhost:8080/api/management/info. Now, Add the following static rule in application.groovy to secure all Actuator end-points.
  • grails.plugin.springsecurity.controllerAnnotations.staticRules = [ ... [pattern: '/api/management/**', access:['isAuthenticated()']] ]
With the above settings, all Actuator end-points are available to logged in users. You can further restrict the access by limiting it to just administrators by adding the role ROLE_ADMIN to the statesRules like:
[pattern: '/api/management/**', access:['ROLE_ADMIN']]

Now we have successfully added security to our app in RESTful way with the two Grails plugins. In the next post, we will add PostgreSQL support to the application.

References

Documentations
Grails Spring Core Plugin Documentation
Grails Spring Security REST Plugin Documentation
Spring Security Documentation
Grails Goodness: Adding Health Check Indicators

Code/APIs
Grails Spring Security Core Plugin code
Grails Spring Security REST Plugin code
Spring Security Code
Spring Security API