Creating a Mobile-Optimized REST API Using Oracle Mobile Cloud Service – Part 2

Introduction

To build functional and performant mobile apps, the back-end data services need to be optimized for mobile consumption. RESTful web services using JSON as payload format are widely considered as the best architectural choice for integration between mobile apps and back-end systems. At the same time, many existing enterprise back-end systems provide a SOAP-based web service application programming interface (API). In this article series we will discuss how Oracle Mobile Cloud Service (MCS) can be used to transform these enterprise system interfaces into a mobile-optimized REST-JSON API. This architecture layer is sometimes referred to as Mobile Backend as a Service (MBaaS). A-Team has been working on a number of projects using MCS to build this architecture layer. We will explain step-by-step how to build an MBaaS, and we will  share tips, lessons learned and best practices we discovered along the way. No prior knowledge of MCS is assumed. In part 1 we discussed the design of the REST API, in this second part we will discuss the implementation of the “read” (GET) RESTful services in MCS by transforming ADF BC SOAP service methods.

Main Article

This article is divided in the following section:

• Understanding the JavaScript Scaffold
• Implementing Dynamic Mock Data
• Defining and Testing the HR SOAP connector
• Implementing the GET Resources

Understanding the JavaScript Scaffold

MCS uses Node.js and Express.js as the transformation and shaping engine of your custom API’s. It is beyond the scope of this article to explain Node.js in detail, to be able to follow the remainder of this article it is sufficient to know that Node.js is a high-performance platform that allows you to run JavaScript on the server using a so-called asynchronous event model. Express.js is a web application framework that works on top of Node.js which, among others, makes it easy to implement RESTful services. To jumpstart the implementation of our REST API using Node and Express we can download a so-called JavaScript scaffold. First click on the Manage link on our Human Resources API overview page

This brings us to the API implementation page which allows us to download a zip file with a skeleton of our Node.js project that we will use to implement the various REST resources that we defined in part 1 of this article series.

Save the zip file hr_1.0.zip to your file system and unzip it. It creates a directory named hr, with 4 files:

Here is a short explanation of each file:

• package.json: this file holds various metadata relevant to the project. It is used to give information to the Node Package Manager (npm) that allows it to identify the project as well as handle the project’s dependencies. Later on we will edit this file to add a dependency to the SOAP connector that we are going to use.
• hr.js: this is the “main” JavaScript file. If you open it, you will see skeleton methods for each REST resources that we have defined in part1. This is the file that MCS will use to handle the REST calls to your API.
• samples.txt: this is a file with various code snippets that come in handy when implementing a custom API, as we will see later on
• readMe.md: readme file that explains the purpose of the files included in the scaffold.

You can use your favorite JavaScript editor to inspect and edit the content of these files.

If you are new to JavaScript development and you don’t have a strong preference for a JavaScript editor yet, we recommend you to start using the free, open source IDE NetBeans. Traditionally known as a Java IDE, NetBeans now also provides excellent support for JavaScript development with code completion and support for Node.js and many other popular JavaScript tools and frameworks.

The hr.js looks like this in Netbeans:

Note the function called on the service object, which reflects the HTTP method we defined for the resource in MCS. The order in which the functions are generated into the skeleton is random, so it is a good practice to re-order them and have all methods on the same resource grouped together as we already did in the above screen shot.

The req and res parameters passed in each function are Express objects and provide access to the HTTP request and response.

To learn which properties and functions are available on the request and response objects it is useful to check out the Express API documentation.

Implementing Dynamic Mock Data

As discussed in part 1, MCS provides static mock data out of the box by defining sample payloads as part of the API design specification that is stored in the RAML document. Making the mock data dynamic, taking into account the value of query and path parameters, is easy and quick, and can be useful for the mobile developer to start with the development of the mobile app before the actual API implementation is in place. We therefore briefly explain how to implement dynamic mock data before we start implementing the “real” API by transforming the XML output from the ADF BC SOAP service to REST-JSON.

When implementing the /departments GET resource, we need to return different payloads based on the value of the expandDetails query parameter. It is easiest to create separate JSON files for each return payload. We create a file named DepartmentsSummary.json that holds the return payload, a list of department id and name, when expandDetails query parameter is false or not set, and store that file in the sampledata sub directory under our hr root folder. Likewise, we create a file DepartmentsDetails.json in the same directory that holds the list of departments with all department attributes and nested employees data which we return when expandDetails is set to true.

To return the correct payload based on this query parameter, we implement the corresponding method as follows:

    service.get('/mobile/custom/hr/departments', function (req, res) {
        if (req.query.expandDetails === "true") {
            var result = require("./sampledata/DepartmentsDetails");
            res.send(200, result);

        } else {
            var result = require("./sampledata/DepartmentsSummary");
            res.send(200, result);
        }
    });

As you can see we check the value of the query parameter using the expression req.query.expandDetails. Based on the value we read one of the two sample payloads and return it as response with HTTP status code 200.

Likewise, we can make /departments/:id GET resource somewhat dynamic by setting the department id in the return payload to the value of the department id path parameter:

    service.get('/mobile/custom/hr/departments/:id', function (req, res) {
        var department = require("./sampledata/Department");
        department.id = req.params.id;
        res.send(200, department);
    });

Here we check the value of the path parameter using the expression req.params.id.  Obviously, we could make the return payload more dynamic by creating multiple sample JSON files for each department id, named Department10.json, Department20.json, etc, and return the correct payload using the following code:

    service.get('/mobile/custom/hr/departments/:id', function (req, res) {
        var department = require("./sampledata/Department"+req.params.id);
        res.send(200, department);
    });

To test our dynamic mock implementation, we simply zip up the hr directory that was created by unzipping the scaffold zip file, and then upload this zip file to MCS again.

After uploading the zip file you can switch to the Postman tab in your browser and test the two GET resources for its dynamic behavior.

The Postman REST client allows you to save resources with its request headers and request payload in so-called collections. We recommend to create a collection for each MCS API you are developing so you can quickly retest resources after uploading a new implementation zip file. Postman also allows you to export and import collections so you can easily share your test collection with other developers.

You can also test the resource using the Test tab inside MCS but this is slower as you have to navigate back and forth between the test page and upload page and there is no option to save request parameters and request payloads.

Defining and Testing the HR SOAP connector

The input data for the HR Rest API is primarily coming from an ADF Business Components application where we exposed some of the view objects as SOAP web services using the application module service interface wizards. MCS eases the consumption of SOAP web services through so-called connectors. A SOAP connector hides the complexity of XML-based SOAP messages and allows you to invoke the SOAP services using a simple JSON payload without bothering about namespaces.

So, before we can define our SOAP connector, we need to have our ADF BC SDO SOAP service up and running. We can test the service using SOAP UI, or using the HTTP Analyzer inside JDeveloper. As shown in screen shot below, the findDepartments method returns a list of departments, including a child list of employees for each department.

There is also a getDepartments method that takes a departmentId as parameter, this method returns one department and its employees. We will use these two methods as the data provider for the two RESTful resources that we are going to implement in this article, the /departments GET resource and the /departments/{id} GET resource.

We put the WSDL URL on the clipboard by copying it from the HTTP Analyzer window in JDeveloper, and then we click on the Development tab in MCS and click the Connectors icon.

We click on the green New Connector button and from the drop down options, we choose SOAP connector.

In the New SOAP Connector API dialog that pops up, we set the API name, display name, and a short description as below, and we paste the WSDL URL from the clipboard.

We also changed the host IP address from 127.0.0.1 (or localhost) to the actual IP address of our machine that is running the SOAP web service so MCS can actually reach our web service, After clicking Create button we see the connector configuration wizard where we can change the port details, set up security policies (we will look into security later in this article series), and test the connector. We click the Test tab, and then we click on the green bar with the findDepartments method so we can start testing this method.

The first thing that might surprise you is the default value for the Content Type request header parameter. It is set to application/json. And this content-type matches with the sample JSON payload displayed in the HTTP Body field. You might wonder, is this wrong, and do you need to change this to xml? The answer is no, MCS automatically converts the JSON payload that you enter to to the XML format required for the actual SOAP service invocation. You  probably noticed that it took a while before the spinning wheel disappeared, and the sample request payload was displayed. This is because MCS parses the WSDL and associated XML schemas to figure out the XML request payload structure and then converts this to JSON. This is a pretty neat feature because you can continue to work with easy-to-use JSON object structures to call SOAP services rather than the more complex XML payloads where you also have to deal with various namespace declarations.

Now, to actually test the findDepartments method we can greatly simplify the sample payload that is pre-displayed in the HTTP Body field. ADF BC SOAP services support a very sophisticated mechanism to filter the results using (nested) filter objects consisting of one or more attributes, with their value and operator. We don’t need all that as we simply want all departments returned by the web service, so we change the HTTP Body as follows:

We set the Mobile Backend field to HR1 (or any other mobile backend, the value of this field doesn’t matter when testing a connector), and click on Test Endpoint.The response with status 200 should now be displayed:

Again, MCS already converted the XML response payload to JSON which is very convenient later on, when we transform this payload into our mobile-optimized format using JavaScript.

If for whatever reason you do not want MCS to perform this auto-conversion from XML to JSON for you, you can click the Add HTTP Header button and add a parameter named Accept and set the value to application/xml. This will return the raw XML payload from the SOAP service.Likewise, if you want to specify the request body as XML envelope rather than JSON, you should change the value of the Content-Type parameter to application/xml as well.

 Implementing the GET Resources

With the HR SOAP connector in place we can now add real implementation code to our JavaScript scaffold, where we use the SOAP connector to call our web service and then transform the payload to the format as we specified in our API design in part 1.

You might be inclined to start adding implementation code to the main hr.js script like we did for the dynamic mock data, however we recommend to do this in a separate file.

For readability and easier maintenance, it is better to treat the main JavaScript file (hr.js in our example) file as the “interface” or “contract” of your API design, and keep it as clean as possible. Add the resource implementation functions in a separate file (module) and call these functions from the main file.

To implement the above guideline, we first create a JavaScript file named hrimpl.js in a subdirectory named lib with two functions that we will implement later:

exports.getDepartments = function (req, res) {
        var result = {};
        res.send(200, result);
};

exports.getDepartmentById = function (req, res) {
        var result = {};
        res.send(200, result);
};

In the main hr.js file, we get access to the implementation file by adding the following line at the top:

var hr = require("./lib/hrimpl");

When using a sophisticated JavaScript editor like NetBeans, we get code insight to pick the right function from our implementation file for each resource implementation:

To be able to use the HR SOAP connector we created in the previous section in our implementation code, we need to define a dependency on it in package.json:

We can copy the dependency path we need to enter here from the connector General page:

We can now start adding implementation code to our hrimpl.js file. Let’s start with code that simply passes on the JSON as returned by the SOAP findDepartments method:

exports.getDepartments = function (req, res) {
  var handler = function (error, response, body) {
    var responseMessage = body;
    if (error) {
      responseMessage = error.message;
    } 
    res.status(response.statusCode).send(responseMessage);
    res.end();
  };

  var optionsList = {uri: '/mobile/connector/hrsoap1/findDepartments'};
  optionsList.headers = {'content-type': 'application/json;charset=UTF-8'};
  var outgoingMessage = {Header: null, Body: {"findDepartments": null}};
  optionsList.body = JSON.stringify(outgoingMessage);
  var r = req.oracleMobile.rest.post(optionsList, handler);
};

Let’s analyze each line of this code, starting in the middle:

• line 11: here we set  up the optionsList object that we use to pass in the connector URI of the SOAP method we want to invoke. The URI path can be copied from the connector test page.
• line 12: we set the Content-Type header parameter, just like we did in the connector test page
• line 13: we set the request body object just like we did in the connector test page
• line 14: we convert the request body to a string
• line 15: we invoke the SOAP service through the connector, passing the optionsList object and a so-called callback handler. This is needed because of the asynchronous programming model we briefly mentioned when introducing Node: the SOAP method is called asynchrously and the Node engine in MCS calls this callback handler function when the SOAP service returns a response.
• line 2: the declaration of the callback handler function with the expected signature.
• line 3: we default the response message we will return to the body of the SOAP service.
• line 4 and 5: if an error occurred and the SOAP call did not succeed, we set the response message to the error message
• line 7: we set the HTTP status code of our REST call (/departments in this case) to the same status as returned by the SOAP call, and we set the response body of this REST call to the response message we set before.
• line 8: Calling the end() function on the REST response tells Node/Express that we are done and that the response can be returned to the caller.

Note that in the samples.txt file included in the scaffold zip file you can find similar sample code to call a SOAP (or REST) connector and to declare a callback handler function.

We can now zip up the hr directory and upload this new implementation and go to Postman to test whether the /departments resource indeed returns the full SOAP response body in JSON format. Once this works as expected we know we have set up the SOAP connector call correctly and we can move on to transform the SOAP response payload based on the value of the expandDetails query parameter.

To transform JSON arrays, the JavaScript map function comes in very handy. This function creates a new array with the results of calling a provided function on every element of the array on which the method is called. So, we need to specify two transformation functions for the department, and use one or the other in our map() function call based on the value of the query parameter.

We recommend to define transformation functions in a separate file to increase maintainability and reusability. If you have prior experience with a product like Oracle Service Bus you can make the analogy with the reusable XQuery or XSLT definitions, they serve the same purpose as these transformation functions although the implementation language is different.

To follow the above guidelines, we create a new Javascript file named transformations.js, store it in the lib subfolder and add the transformations we need for the getDepartments function:

exports.departmentSummarySOAP2REST = function (dep) {
  var depRest = {id: dep.DepartmentId, name: dep.DepartmentName};
  return depRest;
};

exports.departmentSOAP2REST = function (dep) {
  var emps = dep.EmployeesView ? dep.EmployeesView.map(employeeSOAP2REST) : [];
  var depRest = {id: dep.DepartmentId, name: dep.DepartmentName, 
    managerId: dep.ManagerId, locationId: dep.LocationId, employees: emps};
  return depRest;
};

function employeeSOAP2REST(emp) {
  var empRest = {id: emp.EmployeeId, firstName: emp.FirstName, 
    lastName: emp.LastName, email: emp.Email, phoneNumber: emp.PhoneNumber, 
    jobId: emp.JobId, salary: emp.Salary, commission: emp.CommissionPct, 
    managerId: emp.ManagerId, departmentId: emp.DepartmentId};
  return empRest;
};

The first function on line 1 will be used when expandDetails query parameter is not set or set to false: we only extract the department id and name attributes from the SOAP response department object. The second function is used when this query parameter is set to true, we need to transform to a department object which includes all department attributes as well as a nested array of employees that work in the department. To transform the nested employees array we use the map function just like we are going to do for the top-level transformation of the department array. Since some departments might not have employees, we check in line 7 whether the EmployeesView attribute in the department SOAP object exists. If this attribute is not present,we set the employees attribute to an empty array.

We get access to these transformation functions in the hrimpl.js file by using the require statement:

var transform = require("./transformations");

Here is the completed implementation of our getDepartments function using the transformation functions we just defined:

exports.getDepartments = function (req, res) {
  var handler = function (error, response, body) {
    var responseMessage = body;
    if (error) {
      responseMessage = error.message;
    } else if (parseInt(response.statusCode) === 200) {
      var json = JSON.parse(body);
      var expandDetails = req.query.expandDetails;
      var resultArray = json.Body.findDepartmentsResponse.result;
      removeNullAttrs(resultArray);
      var transformFunction = expandDetails === 'true' ? 
        transform.departmentSOAP2REST : transform.departmentSummarySOAP2REST;
      var departments = resultArray.map(transformFunction);
      responseMessage = JSON.stringify(departments);
    }
    res.status(response.statusCode).send(responseMessage);
    res.end();
  };

  var optionsList = {uri: '/mobile/connector/hrsoap1/findDepartments'};
  optionsList.headers = {'content-type': 'application/json;charset=UTF-8'};
  var outgoingMessage = {Header: null, Body: {"findDepartments": null}};
  optionsList.body = JSON.stringify(outgoingMessage);
  var r = req.oracleMobile.rest.post(optionsList, handler);
};

Compared to the previous “pass-through” implementation, we now added an else if branch in the callback handler where we first check the HTTP status code of the SOAP call. If the SOAP call has been successful with status code 200 (line 6), we obtain the value of the expandDetails query parameter (line 8). We traverse the response body in line 9 and store the actual array of departments in variable resultArray. Based on the value of the query parameter, we either use the summary transformation function or the “canonical” transformation function (line 11 and 12), and then we pass these transformation function with the map() function call on the array of SOAP department objects (line 13). The method call on line 10 is explained in the section below “Handling Null Values in SOAP Responses”.

Building on the concepts we have learned so far, you should be able to understand the code below which implements the getDepartmentById function used for the /departments/{id} GET resource:

exports.getDepartmentById = function (req, res) {

  var handler = function (error, response, body) {
    var responseMessage = body;
    var statusCode = response.statusCode;
    if (error) {
      responseMessage = error.message;
    } else if (parseInt(response.statusCode) === 200) {
      var json = JSON.parse(body);
      if (json.Body.getDepartmentsResponse) {
        var dep = json.Body.getDepartmentsResponse.result;
        removeNullAttrs(dep);
        var depResponse = transform.departmentSOAP2REST(dep);
        responseMessage = JSON.stringify(depResponse);
      } else {
        responseMessage = "Invalid department ID " + req.params.id;
        statusCode = 400;
      }
    }
    res.status(statusCode).send(responseMessage);
    res.end();
  };

  var optionsList = {uri: '/mobile/connector/hrsoap1/getDepartments'};
  optionsList.headers = {'content-type': 'application/json;charset=UTF-8'};
  var outgoingMessage = {Body: {"getDepartments": {"departmentId": req.params.id}}};
  optionsList.body = JSON.stringify(outgoingMessage);
  var r = req.oracleMobile.rest.post(optionsList, handler);
};

A few observations on the above method implementation:

• We check for the HTTP status code, if the 200 code is not returned, we assume the department id was invalid, and we return a status code of 400 Bad Request.
• We reuse one of the transformation functions that we created for the /departments resource
• In the request body to call the SOAP getDepartments method, we pass in the departmentId path parameter. We obtain the value of this parameter using the expression req.params.id.

Handling Null Values in SOAP Responses

Your XML SOAP responses might includes a null value like this CommissionPct attribute:

then the auto-converted JSON body from the SOAP response will include the CommissionPct attribute like this

"Salary": 11000,
"CommissionPct": {"@nil": "true"},
"ManagerId": 100,

It is common practice in JSON payloads to leave out attributes that are null. As a matter of fact, In JavaScript when you set an attribute to null and convert the object to a string, the attribute will be left out automatically. To ensure we do not pass on this “nil” object, we apply the following removeNullAttrs method to the SOAP result array before performing the transformations:

function removeNullAttrs(obj) {
  for (var k in obj)
  {
    var value = obj[k];
    if (typeof value === "object" && value['@nil'] === 'true') {
      delete obj[k];
    }
    // recursive call if an object
    else if (typeof value === "object") {
      removeNullAttrs(value);
    }
  }
}

Conclusion

We have provided you with detailed step-by-step instructions on how to create two RESTFul resources that follow the design we have described in part 1 of this article series. If you are used to the more visual and declarative approach used in products like Oracle Service Bus, this code-centric approach using JavaScript and Node.js might feel somewhat strange in the beginning. However, it is our own experience that you very quickly adapt to the new programming model, and the more you will use MCS for transformations like this, the more you will like it. Once you understand the core concepts explained in this article series, you will notice how easy and fast MCS is for creating or modifying transformations.

In the next parts of this article series we will dive into more advanced concepts. We will implement the PUT and POST resources, discuss troubleshooting techniques, add error handling and security, and we will take a look at caching using the MCS storage facility, and we will look into techniques for sequencing multiple API calls in a row without ending up in the infamous “callback hell”. Stay tuned!

All content listed on this page is the property of Oracle Corp. Redistribution not allowed without written permission