Saturday, September 29, 2012

Continuous Deployment for Mobile Apps with Jenkins: Multi-Device Android Builds


In a previous blog post in this series about continuous integration for mobile apps with Jenkins, I showed examples of how to run Android builds using remote slaves, in the same way that you might run iOS builds (see this blog for more details on how to set that up).  That's all very well and may be what you need to do in some circumstances, such as when you want to run tests using an actual, tethered device.  However, when developing for Android that's probably not what you would want to do most of the time: you just want your changes to be built and tested automatically against the Android SDK's emulator, only doing actual physical device testing at key milestones.  For these regular automatic build jobs, Jenkins as a service running on the CloudBees PaaS is the perfect tool since we can allocate build executors on-demand, without any software set-up or hardware required.

Another feature of Android development is the number of possible device configurations that a mobile app might need to support.  This contrasts with the iOS world, where there are only a certain number of combinations you need to worry about: sure, you have different iOS versions and several devices (iPad, iPhone, iPod Touch) which could have different memory specs, for example, but with Android there are so many more combinations of underlying system image and device capability to worry about.  Again, Jenkins is a perfect match for this challenge: with its multi-configuration "matrixed" build jobs, Jenkins can kick off any number of similar build/test cycles, all modeled on a single "template" but each running against a distinct set of device characteristics and emulator specs.  You can see an example of a multi-configuration Android build online here and here's the matrix of builds it defines:


As you can see, I've chosen three axes for the matrix build (screen density, locale and architecture) but for a real build system, I could and would add other axes and many more combinations.  The beauty of being able to run all these in the cloud is that I don't have to worry about resources for these builds: Jenkins will automatically allocate executors from the pool of available resources on-demand.  Here's how you would go about setting this up: first, create a new Jenkins job and choose the "Build multi-configuration project" option; then define the axes like this in the Configuration Matrix section:



These axes (density, locale and target-abi) map to the properties (${density}, ${locale}, ${target-abi}) used in the Android Emulator configuration, like so:


You'll find more information on the Jenkins Android Emulator Plug-in page, which also has a nice example of how you can remove elements from the Matrix of builds (for example, where a particular screen density is nonsensical for certain device configurations) using a "Combination Filter" in Jenkins.  Another interesting use for Combination Filters is to have Jenkins automatically select a subset of build configurations from the matrix: during development, you might want only a "sample" subset of device configurations to be tested, as a sanity check, with the full matrix used for formal build milestones.  An example of that would be a combination filter like this: index%2==0; this would select half the build configurations to run (Jenkins will choose the set to ensure that all the axes get as much coverage as possible, within the constraints you set).

When running Android builds (single or multi-configuration) using Jenkins in the cloud, the Android SDK will automatically be mounted: you specify the appropriate Android target in your build properties as in the example below.  The Android project is updated from SCM (and the project workspace refreshed), then we run the android project update command to set the target for the build and update the local.properties file to point to the Android SDK used by the Jenkins executor(s).  Here's what the build configuration looks like, with the relevant output from the Jenkins console:



Running multi-configuration build/test jobs makes continuous integration for Android development a practical reality, without the need to configure test environments and schedule runs manually.  Every time you make changes to the code, you know that those changes will be tested across the huge range of devices, locales, settings and environments that you will need to support.  

Cheers, Mark


Mark Prichard, Senior Director of Product Management
CloudBees


Mark Prichard is Java PaaS Evangelist for CloudBees. He came to CloudBees after 13 years at BEA Systems and Oracle, where he was Product Manager for the WebLogic Platform. A graduate of St John's College, Cambridge and the Cambridge University Computer Laboratory, Mark works for CloudBees in Los Altos, CA.  Follow Mark on Twitter and via his blog Clouds, Bees and Blogs.

Continuous Deployment for Mobile Apps with Jenkins: Android


In a previous blog, I showed how you can run iOS mobile application builds using MacOS slaves managed by Jenkins running as a service on the CloudBees PaaS.  Now let's have a look at how to do the same for Android.  I'm sure that most Java developers will find many things about the Android development environment pretty familiar and it will probably come as no surprise to find that Jenkins has excellent support for Android builds.

Let's start with the remote slave: just as I did for the iOS build, I'm going to use my MacBook laptop to run the Android build jobs.  Here's the script I use to start the remote executor:

java -jar jenkins-cli.jar -s https://partnerdemo.ci.cloudbees.com -i ~/.ssh/id_rsa_partnerdemo customer-managed-slave -fsroot ~/Jenkins -labels android -labels 16 -executors 1 -name mark-android

This is almost identical to the script I use to start the slave for the iOS builds: I have used the labels "android" and "16" (I'll specify these in the Jenkins build configuration).  Note that you need to use a version of jenkins-cli.jar 2.13 or greater.  You would also use your CloudBees Jenkins account URL rather than https://partnerdemo.ci.cloudbees.com and your own private key with the -i flag.  I'll go into a bit more detail on the -fsroot flag at the end of this blog: for now, just remember this is where Jenkins will store various runtime artifacts, project workspaces and Android SDK packages.  For convenience, I have two remote slaves defined, one to run iOS builds and the other for Android builds.  There's no reason why these should be on a single system and for production CI environments you would almost certainly have a number of dedicated, headless servers to run these builds, but for demos and personal development projects it is easy to set things up like this.  Here's a view of the two nodes (using the Jenkins console "Manage Nodes" view):


One of the big differences between running iOS and Androids builds with Jenkins is that the Jenkins Android Emulator plugin will automatically download not only the Android SDK but also any system image required for the particular device configuration you want to run your tests against. The first time you run a build using the slave, you will see that it will download the appropriate Android SDK for that OS (it will take quite a bit longer as result, but subsequent builds are fast); whenever you specify a new device configuration for your Jenkins builds, you will similarly see the system image files for the configuration being downloaded and the plugin will automatically configure and initialize the Android emulator.  Here's an example of the sort of output you should see in the latter case (for clarity, I have omitted the fairly verbose download progress messages):

$ /Users/markprichard/Jenkins/tools/android-sdk/tools/android list target
[android] The configured Android platform needs to be installed: android-16
$ /Users/markprichard/Jenkins/tools/android-sdk/tools/android list target
$ /Users/markprichard/Jenkins/tools/android-sdk/tools/android list target
[android] Installing the 'android-16,sysimg-16' SDK component(s)...
$ /Users/markprichard/Jenkins/tools/android-sdk/tools/android update sdk -u -a -t android-16,sysimg-16
Refresh Sources:
...
  Installing Mips System Image, Android API 16, revision 1
  Unzipping Mips System Image, Android API 16, revision 1 (1%)
  Unzipping Mips System Image, Android API 16, revision 1 (99%)
    Installed Mips System Image, Android API 16, revision 1
  Done. 4 packages installed.
[android] Using Android SDK: /Users/markprichard/Jenkins/tools/android-sdk
[android] Creating Android AVD: /Users/markprichard/.android/avd/hudson_en-US_240_WVGA_android-16.avd
[android] /Users/markprichard/Jenkins/tools/android-sdk/tools/android create avd -f -a -c 64M -s WVGA800 -n hudson_en-US_240_WVGA_android-16 -t android-16 --abi armeabi-v7a
$ /Users/markprichard/Jenkins/tools/android-sdk/platform-tools/adb start-server
$ /Users/markprichard/Jenkins/tools/android-sdk/tools/emulator -snapshot-list -no-window -avd hudson_en-US_240_WVGA_android-16
[android] Starting Android emulator and creating initial snapshot
$ /Users/markprichard/Jenkins/tools/android-sdk/tools/emulator -no-boot-anim -ports 55496,55497 -prop persist.sys.language=en -prop persist.sys.country=US -avd hudson_en-US_240_WVGA_android-16 -no-snapshot-load -no-snapshot-save -wipe-data
* daemon not running. starting it now on port 55499 *
* daemon started successfully *
$ /Users/markprichard/Jenkins/tools/android-sdk/platform-tools/adb connect localhost:55497
[android] Waiting for emulator to finish booting...
$ /Users/markprichard/Jenkins/tools/android-sdk/platform-tools/adb -s localhost:55497 shell getprop dev.bootcomplete
$ /Users/markprichard/Jenkins/tools/android-sdk/platform-tools/adb -s localhost:55497 logcat -v time
[android] Giving the system some time to settle before creating initial snapshot...
$ /Users/markprichard/Jenkins/tools/android-sdk/platform-tools/adb connect localhost:55497
$ /Users/markprichard/Jenkins/tools/android-sdk/platform-tools/adb -s localhost:55497 logcat -c
$ /Users/markprichard/Jenkins/tools/android-sdk/platform-tools/adb -s localhost:55497 shell log -p v -t Jenkins "Creating snapshot..."
[android] Creating snapshot...
$ /Users/markprichard/Jenkins/tools/android-sdk/platform-tools/adb connect localhost:55497
[android] Emulator is ready for use (took 148 seconds)

Note the line in bold above: the plugin has created an AVD (Android Virtual Device) named "hudson_en-US_240_WVGA_android-16" (we'll see in a minute where that name and those settings come from).  The device configuration will be stored by default in the ~/.android directory for the user associated with the Jenkins remote slave.  Assuming that you just ran jenkins-cli.jar from a terminal window, you can see the configuration either with the the Android AVD Manager (below, top) or the command android list avd (below, bottom) - in this example, I have one virtual device that was defined using the Android AVD Manager and the one that was created by the Jenkins plugin.  You can use AVDs created by Jenkins just like any other - for example, the following command would start the emulator on my desktop: emulator -avd hudson_en-US_240_WVGA_android-16

Android Virtual Device Manager

android list avd
Now let's look at the Jenkins configuration: you can see an example here.  In this example, I'm using a clone of Peter Ă–sterlund's excellent DroidFish port of StockFish: I've simplified the build slightly by including a jar with the CuckooChess classes with the source code in my GitHub repository.  The Jenkins job is configured to run whenever a push is made to that repository, using the same WebHook URL as in the iOS example (https://partnerdemo.ci.cloudbees.com/github-webhook/).  I have checked the box "Restrict where this project can be run" and specified the Label Expression "android" to associate the build with the mark-android remote slave we defined earlier on.

The key part of the configuration is the Jenkins Android Emulator plugin (note that the Port Allocator Plugin is a pre-requisite, although there is no additional configuration needed).  This plugin has very rich functionality and I'm only using a fairly simple configuration here: check out the documentation for details of what it can do, including very powerful matrix build capability.  Here's the configuration that I'm using:


I could have used a pre-defined device configuration, but I prefer to specify the device parameters in the plugin configuration and not rely on any assumptions about the setup on the slave.  As you can see, I have chosen a Droid-like configuration (the hudson_en-US_240_WVGA_android-16 AVD definition we saw earlier).  The plugin will automatically start the emulator before running the build phase and will stop it after the run completes.  The build phase itself is simple, following the standard Android build process: first we use the Execute Shell plugin to run android update project and android update test-project to configure the project correctly for the Android SDK installation, and then we use the built-in Ant plugin to run ant clean debug install test for the DroidFishTest project:


If I run the build, either manually from the console or by pushing a change to the source repository, then I'll see Jenkins start the AVD emulator, run the Android ant build (ant clean debug install test) to install and run the app with the test .apk files on the device, and finally stop the emulator.  You can see the full output if you look at the console output from one of the runs.  Here's a screenshot:


Finally, let's take a look at the contents of the directory we specified with the -fsroot flag when starting the remote slave (in my example ~/Jenkins).  As you can see, it contains the runtime jars need for the Jenkins remoting, plus the workspaces for the various projects built on that slave and, under ./tools, the Android SDK and all the necessary components to run the build/test.


This is very useful (particularly when you want to run tests using an actual, tethered Android device) but the real value of using Jenkins to control your Android build/test CI cycle comes when you take advantage of the cloud to run many test scenarios in parallel using matrixed Jenkins configurations; more on that in my next blog.

Cheers, Mark



Mark Prichard, Senior Director of Product Management
CloudBees


Mark Prichard is Java PaaS Evangelist for CloudBees. He came to CloudBees after 13 years at BEA Systems and Oracle, where he was Product Manager for the WebLogic Platform. A graduate of St John's College, Cambridge and the Cambridge University Computer Laboratory, Mark works for CloudBees in Los Altos, CA.  Follow Mark on Twitter and via his blog Clouds, Bees and Blogs.

Wednesday, September 12, 2012

Continuous Deployment for Mobile Apps with Jenkins: Back-end Services and Integration Testing

In an earlier post, I described how to configure Jenkins (running in the cloud using the CloudBees PaaS service) to manage Xcode builds for iOS applications using MacOS slaves.  As part of our demo at the Silicon Valley Cloud Computing Group Meetup,  Kohsuke and I also showed how easy it is to do full, end-to-end continuous integration that includes integration testing with back-end services as well.  For this demo, I built a simple servlet-based chess server with REST web services to record games and moves, using JAX-RS, JAX-B and a MongoDB backend.  You can find the source code on GitHub at https://github.com/mqprichard/mongo-chess.

The MongoChess Server Project

Just to give you a quick sense for the application, here's a view of the Eclipse project:


The com.cloudbees.service package contains two classes (GameServlet and MoveServlet) which provide REST web service interfaces to create games, record moves and get the history for a particular game.  There is also a third class (MongoDAO) which abstracts away access to the MongoDB database, where the actual results are stored: it's a straightforward implementation, with a separate connection used for each REST service call.  I'm using the free MongoDB service provided by MongoHQ, which you can sign up for via the CloudBees Partner Ecosystem page on GrandCentral.  

There are classes (in com.cloudbees.model) to represent Game and Move objects and I use the standard JAX-B @XmlRootElement annotation to handle the mapping to the JSON format used to pass data via the REST interface and also for inserts and queries against the MongoDB database.  Incidentally, this was one of the reasons why I chose to use MongoDB for this example - this is where a NoSQL database that works naturally with JSON-formatted data is much simpler to work with than a more traditional relational database (by way of comparison, I used MySQL for my first cut at this application and you can see that version at https://github.com/mqprichard/chess).  

The Chess Server REST Interface

I'll describe in a minute how I use a Jenkins build pipeline to test the REST service interface, but first here's a quick description of what the interface looks like using curl to make POST and GET requests (notice that the service is configured to accept and return data using MIME type application/json, hence the need to set the Accept and Content-type headers):

Create a new game:
curl -s -H "Accept: application/json" -H "Content-Type: application/json" -X POST -d '{"white":"Player1","black":"Player2","description":"Championship"}' http://localhost:8080/chess/game/new
{"id":"5047b5e703640e8d8faf6b2a"}

Get information about a game:
curl -H "Accept: application/json" http://localhost:8080/chess/game/5047b5e703640e8d8faf6b2a
{ "white" : "Player1" , "black" : "Player2" , "description" : "Championship" , "result" : "" , "next" : "W" , "move" : 1}

Submit moves:
curl -H "Accept: application/json" -H "Content-Type: application/json" http://localhost:8080/chess/moves/new -X POST -d '{"game":"5047b5e703640e8d8faf6b2a","white":"e2-e4","move":"1"}'
{"game":"5047b5e703640e8d8faf6b2a","move":1,"white":"e2-e4"}
curl -H "Accept: application/json" -H "Content-Type: application/json" http://localhost:8080/chess/moves/new -X POST -d '{"game":"5047b5e703640e8d8faf6b2a","black":"e7-e5","move":"1"}'
{"game":"5047b5e703640e8d8faf6b2a","move":1,"black":"e7-e5"}
curl -H "Accept: application/json" -H "Content-Type: application/json" http://localhost:8080/chess/moves/new -X POST -d '{"game":"5047b5e703640e8d8faf6b2a","white":"g1-f3","move":"2"}'
{"game":"5047b5e703640e8d8faf6b2a","move":2,"white":"g1-f3"}
curl -H "Accept: application/json" -H "Content-Type: application/json" http://localhost:8080/chess/moves/new -X POST -d '{"game":"5047b5e703640e8d8faf6b2a","black":"b8-c6","move":"2"}'
{"game":"5047b5e703640e8d8faf6b2a","move":2,"black":"b8-c6"}
curl -H "Accept: application/json" -H "Content-Type: application/json" http://localhost:8080/chess/moves/new -X POST -d '{"game":"5047b5e703640e8d8faf6b2a","white":"f1-b5","move":"3"}'
{"game":"5047b5e703640e8d8faf6b2a","move":3,"white":"f1-b5"}

Get move history for a game:
curl -H "Accept: application/json" http://localhost:8080/chess/moves/5047b5e703640e8d8faf6b2a ; echo
[{ "white" : "e2-e4"}, { "black" : "e7-e5"}, { "white" : "g1-f3"}, { "black" : "b8-c6"}, { "white" : "f1-b5"}]

There are also a set of JUnit tests, which I use to test all the MongoDAO methods as well as the GameServlet/MoveServlet interfaces.  Of course, I need to run these every time I do a build but I also want to invoke the REST services over the wire to be certain that the JAX-RS interfaces are working as expected.  There are several ways to do this and the folks at SOASTA have a very nice capability to automate test like these: however, as an example of how to do this using the Jenkins community plug-ins, I put together some scripts that can be run as part of a Jenkins build pipeline, that run a set of curl commands like those above and check the actual output against that expected.  

The only thing that's at all complicated about these is the need to parse the JSON data using only basic shell utilities like awk: if you use tools like jsawk then of course the scripts are smaller and simpler, but in this case, I wanted to be sure that the scripts would execute on the lightweight, JEOS shell that runs the Jenkins executor processes in the CloudBees Paas platform.  I'm sure that there are shell script gurus out there who can do better, but you can see the scripts I use on GitHub at https://github.com/mqprichard/mongo-chess/tree/master/scripts.

Building the Integration Test Pipeline 

Here's how to configure the Jenkins build pipeline: you can see the full configuration here.  First, we need a Jenkins job to build the Maven project and run the JUnit tests: that's called mongo-chess (view configuration) and apart from the normal build/test configuration the only other thing we need is to set up the pipeline so that we archive the built artifacts (in this example I'm using the Jenkins Enterprise Fast Archive plugin that I described in the earlier blog, but that's just icing on the cake) and run the integration test build (mongo-chess-integration) on a successful build, as shown below:


The second part of the pipeline is the integration test itself and you can view the configuration here.  As you can see from the following screenshots, this runs after the mongo-chess build/test completes and it spins up an instance of the archived application (remember this all happens on-demand in the cloud), then calls the Execute Shell Jenkins plugin to run the integration tests.



Take a look at the console output for one of the build jobs and you'll see the result: provided the interface behavior has remained, there is no diff between the observed and expected outputs and the shell script returns zero.  The build is successful and we pass in turn to the next stage of the pipeline (mongo-chess-end2end-integration), which is where we would configure the SOASTA TouchTests for the mobile application.  More on that in an upcoming blog/webinar - for now that's just a placeholder.

Want to see this technology in action?

My friends at SOASTA and I will demo it live on Sept 27 at 10amPT on our new webinar, Automated Testing & Continuous Deployment for Mobile Apps in the Cloud. Sign up and join us!

Attend Jenkins User Conference SF - A Special Discount For You!

To find out more about this and all things Jenkins, please join us at the 2012 San Francisco Jenkins User Conference on Sunday, 30 September and enjoy a full day of Jenkins-focused learning and networking opportunities.  Register here and as a special "Thank You" for reading all the way to the bottom of my blog, enter promotional code JUC-2012SF to get a $100 discount off the full-price registration!



Mark Prichard, Senior Director of Product Management
CloudBees


Mark Prichard is Java PaaS Evangelist for CloudBees. He came to CloudBees after 13 years at BEA Systems and Oracle, where he was Product Manager for the WebLogic Platform. A graduate of St John's College, Cambridge and the Cambridge University Computer Laboratory, Mark works for CloudBees in Los Altos, CA.  Follow Mark on Twitter and via his blog Clouds, Bees and Blogs.



Follow CloudBees:

   

Continuous Deployment for Mobile Apps with Jenkins: Xcode Builds with MacOS Slaves

At a recent meetup organized by the Silicon Valley Cloud Computing Group, Kohsuke, Matt Solnit of SOASTA and I did a demonstration of how to manage end-to-end continuous deployment for mobile app development using Jenkins CI, with everything managed in the cloud using CloudBees DEV@cloud PaaS service.

In this blog, I’ll walk you through the setup for the demo.  You can find the slides online at SlideShare here and the Jenkins configuration we used is available at https://partnerdemo.ci.cloudbees.com/. For the demo, we showed how to configure Jenkins on CloudBees to manage the build, testing and deployment of the StockFish iOS chess app, as well as continuous deployment and integration testing with a set of back-end REST web services to record games and moves.  You can find the source code for both on GitHub: the front-end StockFish app (https://GitHub.com/mqprichard/stockfishchess-ios) and the back-end MongoChess server (https://GitHub.com/mqprichard/mongo-chess).

Configuring the GitHub Hook URL

First, let’s look at the setup for building and testing the iOS mobile application: you can see the configuration here.  As you can see from the Source Code Management section of the configuration, the build is triggered whenever a change is pushed to the GitHub repository: to do this, I simply have to enter my GitHub credentials at the bottom of the Jenkins System Configuration page and tell Jenkins to auto-manage hook URLs, like this:





You can see the hook URL by going to the Repository Administration tab on GitHub: click on Test Hook and then go to the Jenkins GitHub Hook Log and you should see confirmation of the trial push.




Running a MacOS Slave

Next we need to configure the Xcode build for the iOS application.  Obviously, this needs to run on a MacOS slave and, in production scenarios, one would normally configure headless MacOS servers to handle the builds – however, you can easily do the same thing using a MacBook laptop, by running jenkins-cli.war in a terminal window:



This is very convenient to test out the Jenkins configuration with your local Xcode development environment. You will need to make sure your SSH keys are configured correctly, but you should just be able to use the same key pair you use for your CloudBees/Jenkins account. There are full instructions on how to run customer provided slaves with a link to download an up-to-date version of jenkins-cli.war here: http://wiki.cloudbees.com/bin/view/DEV/Customer+Provided+Slaves

For example, on my MacBook, I run the following command:
java -jar jenkins-cli.jar -s https://partnerdemo.ci.cloudbees.com -i ~/.ssh/id_rsa_partnerdemo customer-managed-slave -fsroot ~/Jenkins -labels xcode -labels 4.4 -executors 1 -name markmacbook

Here ~/.ssh/id_rsa_partnerdemo is the private key I use with my Jenkins/CloudBees account, ~/Jenkins is a scratch directory that is used for temporary artifacts and the labels “xcode” and “4.4” are examples that show how different builds can be targeted to particular build environments. In this example, we check “Restrict where this project can be run” and specify “xcode” as the Label Expression to ensure that this project will only be built in the MacOS environment:




Setting up Xcode iOS Builds 

Once the channel has been established (as shown in the screenshot above), you will see the new node (here it’s “markmacbook”) appear in the Jenkins console, and we are ready to configure the actual Xcode build using the Jenkins Xcode Plugin.  In this example, to keep things simple, I am running jenkins-cli.war in my Xcode project directory, so most of the fields are either blank or take default values:





You will probably want to define a unique version number for each build and the easiest way to do that is with Jenkins macros, such as 2.2.${BUILD_NUMBER} as we have done in this example. Note that we have also checked the option to save the build as an .ipa archive, which is what we will want when distributing the app for beta testing using a service like TestFlight or Appaloosa.

Distributing the Mobile App for Beta Testing

Finally, we want to upload the app (as an .ipa archive) for distribution to beta testers: here we are using the Jenkins Appaloosa Plugin.  To speed up the upload process, we have taken advantage of the Fast Archiver Plugin, which is part of Jenkins Enterprise by CloudBees and which uses rsync-style delta encoding to optimize archiving.  You will need to make sure that you have the device IDs for the mobile devices that the beta tests will use included in the Provisioning Profile that you are including with your app; there are lots of helpful videos and how-tos on the iOS Provisioning Portal page (note: Apple iOS Developer Program login required).




In a follow-up blog, I’ll describe how to set up continuous deployment and integration testing for the back-end services.

Want to see this technology in action?

My friends at SOASTA and I will demo it live on Sept 27 at 10amPT on our new webinar, Automated Testing & Continuous Deployment for Mobile Apps in the Cloud. Sign up and join us!

Attend Jenkins User Conference SF - A Special Discount For You!

To find out more about this and all things Jenkins, please join us at the 2012 San Francisco Jenkins User Conference on Sunday, 30 September, and enjoy a full day of Jenkins-focused learning and networking opportunities.  Register here and as a special "Thank You" for reading all the way to the bottom of my blog, enter promotional code JUC-2012SF to get a $100 discount off the full-price registration!

Mark Prichard, Senior Director of Product Management
CloudBees


Mark Prichard is Java PaaS Evangelist for CloudBees. He came to CloudBees after 13 years at BEA Systems and Oracle, where he was Product Manager for the WebLogic Platform. A graduate of St John's College, Cambridge and the Cambridge University Computer Laboratory, Mark works for CloudBees in Los Altos, CA.  Follow Mark on Twitter and via his blog Clouds, Bees and Blogs.



Follow CloudBees:

   

Wednesday, July 25, 2012

REST Web Services in the Cloud: Part 2

The Client Side: JavaScript, HTML/CSS and nginx

In the first post of this 2-part series, I looked at how to implement a cloud-based REST web service using JSON, Jersey/JDBC and the CloudBees Java PaaS platform.  In this second blog, I'd like to look at a simple AJAX-style browser client using a combination of JavaScript and HTML/CSS, served from a local nginx web/proxy server. As a reminder, here's what the basic web service looks like when accessed directly from the browser:






Let's start with the HTML page that I am using: it's very simple, all the presentation elements (such as they are) are handled by the CSS stylesheet and all the work of retrieving the JSON-formatted data and turning it into simple HTML is handled by a piece of JavaScript that is called as soon as the browser loads the page:

<!DOCTYPE html>
<html>
  <head>
    <title>Countries</title>
    <link rel="stylesheet" type="text/css" href=/resources/stats.css />
    <script src="/resources/get-countries.js"></script>
    <script language="JavaScript" type="application/javascript">
      window.onload = getCountries();
    </script>
  </head>
  <body>
    <span id="stats"></span>
  </body>
</html>

The stylesheet is basic: I've re-used a simple scheme that just makes HTML tables a bit cleaner and more modern than the usual browser defaults:

#stats { font-family:Trebuchet MS, Arial, Helvetica, sans-serif; width:100%; border-collapse:collapse; }
#stats h3 { font-size:1.4em; border:none; margin-bottom:3px; }
#stats td, #stats th { font-size:1.2em; border:1px solid #696969; padding:3px 7px 2px 7px; }
#stats th { text-align:left; color:#fff; background-color:#808080; }
#stats tr.alt td { color:#000; background-color:#DCDCDC; }

In a later post, I'll look at more interesting things you can do with JavaScript frameworks like JQuery and Dojo, but for now I hope this helps to illustrate what's going on - I also like this clear separation between the roles of HTML, CSS and JavaScript.  The JavaScript code in particular is closely based on the examples in David Flanagan's outstanding book JavaScript - The Definitive Guide which is one of the most highly-used books on my shelf.  BTW, all the code in this example is available on github in case you want to try it out for yourself: please do - you can get a CloudBees account for free!

Here's the getCountries() function that gets called from the window.onload event in the HTML page.  The basic idea is that the function makes an XMLHttpRequest (XHR) call to /mark/countries (two variants: one for IE and one for Chrome/Firefox/Safari/Opera etc) and this retrieves the raw JSON data from the web service and calls getCountriesJSON(), which does the work of constructing the final HTML page which is inserted into the page <body> using document.getElementById("stats") and innerHTML - recall that the top-level HTML doc simply has <body><span id="stats"></span></body>.

function getCountries()
{

  var url = "/mark/countries";



  if (window.XMLHttpRequest) {
    xmlhttp = new XMLHttpRequest();
    xmlhttp.open("GET", url);
    xmlhttp.onreadystatechange = function() {
      if (xmlhttp.readyState === 4 && xmlhttp.status === 200) {
        var countriesDoc = getCountriesJSON(xmlhttp.responseText);
        var elt = document.getElementById("stats");
        elt.innerHTML = countriesDoc;
      }
    };
    xmlhttp.send(null);
  }
  else if (window.ActiveXObject)  {
    xmlhttp=new ActiveXObject("Microsoft.XMLHTTP");
    if (xmlhttp) {
      xmlhttp.open("GET", url);
      xmlhttp.onreadystatechange = function() {
        if (xmlhttp.readyState === 4 && xmlhttp.status === 200) {
          var countriesDoc = getCountriesJSON(xmlhttp.responseText);
          var elt = document.getElementById("stats");
          elt.innerHTML = countriesDoc;
        } 
      };
      xmlhttp.send(null);
    }
  }
}

Here's the code for getCountriesJSON()the XMLHttpRequest.ResponseText object that is passed as a parameter contains the raw JSON.  This gets loaded into a JS variable (countriesJSON) via an eval() call and then it's a simple matter of iterating through the array to create the table rows:

function getCountriesJSON(response)
{
  var countriesJSON = eval('(' + response + ')');
  var countries = countriesJSON.countries;

  var countriesPage = "";
  countriesPage += "<table id=\"stats\">";
  countriesPage += "<th>Id</th><th>Country</th><th>Capital</th>";

  for (var j=0; j < countries.length; j++) {
    if (j%2 == 0) countriesPage += "<tr>";
    else countriesPage += "<tr class=\"alt\">";
    countriesPage += "<td>" + countries[j].id + "</td>"
                  + "<td>" + countries[j].country + "</td>"
                  + "<td>" + countries[j].capital + "</td>"
                  + "</tr>";
  }
  countriesPage += "</table>";
  return countriesPage;
}


Using nginx as a web/proxy server


For now, I'm simply using a local nginx server to do double duty:
  1. As a Web Server hosting my HTML pages and CSS/JS resources
  2. As a Reverse Proxy Server for my cloud web services
I'll look at various options for how you would handle this in production scenarios in a later post, as it's an interesting subject that deserves its own discussion, but for now just note that the reverse proxy is important, as otherwise I wouldn't be able to make that XMLHttpRequest call, due to the security rules all browsers enforce on cross-domain scripting.  With a single web/proxy server serving the HTML pages, CSS/JS resources and back-end services, this simply isn't an issue; however, otherwise I would have to use JSONP to make the XHR call.  More on that later.


Here's the (very minimal) nginx configuration I use for testing:


worker_processes  1;
error_log  logs/error.log;
pid        logs/nginx.pid;
events {
    worker_connections  1024;
}
http {
    include       mime.types;
    default_type  application/octet-stream;
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';
    access_log  logs/access.log  main;
    sendfile        on;
    keepalive_timeout  65;
    server {
        listen       8888;
        server_name  localhost;
        location / {
            root   html;
            index  index.html index.htm;
        }
        error_page   500 502 503 504  /50x.html;
        location /mark {
                proxy_pass      http://jersey.mqprichard.cloudbees.net/mark;
        }
        location ~* \.(?:ico|css|js|gif|jpe?g|png|bmp|html) {
                root /Users/markprichard/www;
                expires max;
                add_header Pragma public;
                add_header Cache-Control "public, must-revalidate, proxy-revalidate";
        }
    }
}

The main things to note are: 

  1. I'm using ~markprichard/www as the docroot, with the JS/CSS resources in ~markprichard/www/resources. 
  2. The URL pattern /mark is proxied to  http://jersey.mqprichard.cloudbees.net/mark, which takes care of the XHR call to the cloud web service.  As far as the browser is concerned, this is going to the same host:port as the original request, which avoids the cross-domain scripting problem.

Here is what the final result looks like:


Cheers, Mark