Using Google App Engine to power Web APIs

This post is part of the Who's @ Google I/O, a series of blog posts that give a closer look at developers who'll be speaking or demoing at Google I/O. Today's post is a guest post written by Steven Willmott of 3scale.

Public APIs are a great way to share data and functionality, both for existing applications and as useful resources in their own right. Google Maps, Calendar and many other APIs lead the way in this and are now powering tens of thousands of sites and applications. Flickr, Twitter and others also provide great data and functionality which can be used to enhance and build new applications.

While APIs can provide great value add, it's not always obvious how to make one available if you're not Google or Yahoo. Although serving content via an API is effectively no different than via a user-facing web app - just serving HTTP traffic with XML, JSON or arbitrary data rather than HTML, CSS and Javascript normally destined for a browser - it does come with some unique challenges: authentication works differently, inbound traffic is often automated and response times need to be rapid for the API to be usable within another site - all a bit different from a standard web app.

Since we're in the API business we often get asked about how to quickly get a new API up and running, and one neat new way to do this is using Google App Engine, which provides a lot of the core ingredients needed to launch a new API and help it scale. To see how this works we put together a quick and dirty trial service which exposes App Engine's Image manipulation functions as a new Web API:

1. First we set up a standard App Engine account giving us the standard dashboard and deployment environment. This provides the runtime environment plus a range of tools to monitor the application - all useful to track resource usage which we'll need to worry about later.


2. The service itself in this case is a simple wrap of wrapping App Engine's available image manipulation features which provide standard image manipulation features, but it could easily be more complex.


3. Next up, the functionality is wrapped in a simple RESTful interface. There are multiple ways to do this, libraries such as the App Engine Rest Server and the Django rest interface or a port of the Ning-like REST interface can be used to do most of the heavy lifting. Gae-Rest for example includes indexes and memcache and other features to optimise response times on the API. These give the API standard GET, POST, PUT, DELETE operations we'll need to push images in and out of the application.


4. Finally, while we have the service ready to roll we still need a way for developers to access the service, manage their resource usage and track activity on the API. To do this we can use 3scale's API management platform which provides these features out of the box. Connection works using a standard python plugin and the system creates a configurable management portal for the service which handles signup keys, usage limits, traffic analysis and the documentation needed to support developers using the new API. For heavy users we can also add paid plans to cover any App Engine costs or make a small profit - check out the appspotimage portal here:

The service can now take advantage of App Engine's scalability - even under the constraints of the plan, your API should comfortably serve up 500,000 - 1 Million hits per day (although note that you need to check other resource limits - not just hits) which would be non-trivial to arrange and potentially high cost on a lot of existing hosting services. If you're serving traffic from appspot.com (for appspotimage we serve the management interface from appspotimage.com and the API traffic from appspot.com) response times from App Engine tend to be very fast (sub 10ms if you're on the west coast US) - which is great for API users.

Although backend APIs are not quite as visible as a new Android or iPhone App, they often provide a great resource and hopefully we'll see more and more datasets and new types of processing on line as APIs.

You can find the service itself here along with a longer tutorial on how we built it. Stay posted for some other services coming soon, as we start to play around with Java language support on App Engine.

We'll be at Google I/O on May 27-28 and look forward to talking APIs and App Engine, so stop by the Developer Sandbox. See you there!

Guest Post by Steven Willmott of 3scale