Posts Tagged ‘rest’

REST Compliance Officer

Tuesday, March 17th, 2009

With regard to this blog on REST compliance

Me: The Gliffy API is RESTFul
REST Compliance Officer: Does a “PUT” update the data at the given URL?
Me: Yes.
RCO: Trick Question! It’s “URI”. Is the only way to create a new resource done with a “POST”?
Me: Yes.
RCO: Is there exactly one endpoint, from which any and all resource locators are discoverable?
Me: Um, no, that puts undue burden on the client libraries, and over-complicates what we were trying to accomp….

Thank GODS that REST doesn’t have a spec. If it did, it would still be in development.

P.S. If you are going to coin a term and you want to bitch about it being misused, maybe calling it a “style” isn’t the best idea.

Test REST Services

Friday, September 12th, 2008

In my reply to a post on Tim Bray’s blog about using RSpec for testing REST services, I briefly described a project I’m working on, based on the work I’ve been doing at Gliffy, which is a testing framework for REST services called, unsurprisingly, RestUNIT.

For Gliffy’s REST-based integration API, I needed a way to test it, and hand-coding test cases using HTTPClient was just not going to cut it. Further, requests to Gliffy’s API require signing (similar to how Flickr does it), and our API was going to support multiple ways of specifying the representation type as well as tunneling over POST.

So, it occured to me that there was a lazier way of doing this testing. All I really needed to specify was the relative URL, parameters, headers, method, and expected response. Someone else could do the signing and re-run the tests with the various options (such as specifying the MIME Type via the Accept: header, and then again via a file extension in the URL).

I ended up creating a bunch of text files with this information. I then used a Ruby script to generate two things: an XML file that could be deserialized into a java object useful for testing, and a PHP script to test our PHP client API.

The Ruby script would also do things like calculate the signature (the test text files contained the api and secret keys a Gliffy user would have to use the API) and generate some derivative tests (e.g. one using a DELETE, and another tunneling that over POST). The testing engine could generate some additional derivative tests (e.g. GET requests should respond to conditional gets if the server sent back an ETag or Last-Modified header). All this then runs as a TestNG test.

The whole thing works well, but is pretty hackish. So, RestUNIT was created as a fresh codebase to create a more stable and useful testing engine. My hope is to specify tests as YAML or some other human-readable markup, instead of XML (which is essentially binary for any real-sized data) and to allow for more sophisticated means of comparing results, deriving tests, and running outside a container (all the Gliffy tests require a specific data set and run in-container).

The test specification format should then be usable to generate tests in any other language (like I did with PHP). I’m working on this slowly in my spare time and trying to keep the code clean and the architecture extensible, but not overly complex.

Schema for REST services

Thursday, September 11th, 2008

I’m currently working the integration API for Gliffy, which is a REST-based service. The API is fairly stable and we’re readying a few ancillary things for release. One of those is the documentation for the API. I found it quite difficult to completely describe the REST services and ultimately ended up creating something that lists out “objects” and “methods”, even though the API is not really object-based. For example, the object “Diagram” has a “method” called “list”; to “call” it, you do an HTTP GET to accounts/your account name/diagrams.

The original spec I created to work against (and thus, our initial draft of API documentation) was basically a list of URLs and the HTTP methods they responded to. Not very easy to navigate or understand on a first sitting. Some sort of schema to describe the REST API would have been really helpful (along the lines of an XML Schema). Such a schema could facilitate documentation, testing, code generation.

As an example, consider some features of the Gliffy API: you can list the users in an account, list the diagrams in an account and reference an individual diagram via id. Here’s a YAML-esque description of these services:

  kind: literal
  desc: "Reference to all accounts"
    desc: "Creates a new account"
        - account_name
            required: true
            desc: "Name of the account you want to create"
        - admin_email
            required: true
            desc: "Email address of an administrator for the new account"
       kind: variable
       desc: "The name of your account"
         desc: "Returns meta-data about the account"
            - show_users
              required: false
              desc: "If true, users are included, if false, they are not"
           kind: literal
           desc: "All diagrams in the account"
             desc: "Creates a new diagram"
               - diagram_name
                 required: true
                 desc: "Desired name for this diagram"
               - template_id
                 required: false
                 type: numeric
                 dsec: "If present, the id of the diagram to copy, instead of using the blank one"
             desc: "Gets a list of all diagrams in this account"
               kind: variable
               type: numeric
               desc "The id of a particular diagram"
                 desc: "Gets the diagram; the requested encoding type will determine the form"
                   - version:
                     desc: "The version to get, 1 is the original version.  If omitted, current version is retrieved"
                     required: false
                     type: numeric
                   - size:
                     desc: "For rastered formats, determins the size
                     type: enumeration
                       - L
                       - M
                       - S
                 desc: "Deletes this image"
            kind: literal
            desc: "All users in the account"
              desc: "gets a list of all users in the account"

Since “accounts” is the only top-level element, we are saying that every request to this service must start with accounts/. It has one child, which is a variable value for the account name. It is untyped, so any potential string is allowed. That element has two possible children: diagrams and users. diagrams indicates that it responds to the HTTP methods POST and GET. A POST requires the parameter diagram_name, while the parameter version is optional.

A standard format like this could easily be used to generate documentation, expectations, test cases, and even stub code. This format could even be delivered by an OPTIONS call to a resource. I realize there is not much standardization around how to design and implement a REST service, but something like this could at least be a stake in the ground and support a specific method.

REST Security: Signing requests with secret key, but does it work?

Monday, April 21st, 2008

Both Amazon Web Services and the Flickr Services provide REST APIs to their services. I’m currently working on developing such a service, and noticed that both use signatures based on a shared secret to provide security (basically using a Hash Message Authentication Code).

It works as follows:

  1. Applications receive a shared secret known only to them and the service provider.
  2. A request is constructed (either a URL or a query string)
  3. A digest/hash is created using the shared secret, based on the request (for Flickr, the parameter keys and values are assembled in a certain way, so that Flickr can easily generate the same string)
  4. The digest is included in the request
  5. The service provider, using the shared secret, creates a digest/hash on the request it receives
  6. If the service provider’s signature matches the one included in the request, the request is serviced

It’s actually quite simple, and for one-time requests, is effective. The problem, however, is that anyone intercepting the request can make it themselves, without some other state being shared with the client and service provider. Consider a request for an image. The unsigned request might look like:

The signed request, would look like so:

So, anyone can then take that URL and request the resource. They don’t need to know the shared secret, or the signature algorithm. This is a bit of a problem. One of the advantages of REST is that URLs that request resources are static and can be cached (much as WWW resources are). So, if I wish to protect the given URL, how can I do so?

HTTP Authentication

The usual answer is HTTP Authentication; the service provide protects the resource, and the client must first log in. Login can be done programmatically, and this basically accomplishes sending a second shared secret with the request that cannot be easily intercepted. HTTP Auth has its issues, however, and might not be feasible in every context.

Another way to address this is to provide an additional piece of data that makes each request unique and usable only once. To do so requires state to be saved on the client and the server.

Negotiated One-time Token

Authentication can be avoided by using the shared secret to establish a token, usable for one request of the given resource. It would work like this:

  1. Client requests a token for a given resource
  2. Service Provider creates a token (via some uuid algorithm ensuring no repeats) and associates it with the resource
  3. Client creates a second request, as above, for the resource, including the token in the request
  4. Service Provider checks not just for a valid signature, but also that the provided token is associated with the given resource
  5. If so, the token is retired, and the resource data is returned

Here, the URL constructed in step 3 can be used only once. Anyone intercepting the request can’t make it again, without constructing a new one, which they would be unable to do without the shared secret. Further, this doesn’t preclude caching. The main issue here is that since two requests are required, simultaneous access to one resource could result in false errors: if Client A acquires a token, and Client B requests one before Client A uses the token, Client A’s token could be squashed, resulting in an error when he makes his request. The service provider can alleviate this by allowing the issuance of multiple active tokens per resource.


A disadvantage to the One-Time Token method is that it requires two requests of the service provider for every actual request (one to get the token and one to request the resource). A way around that is to include a timestamp in the request. This would work as follows:

  1. Client creates request, including the current time. This request is signed as per above procedure
  2. Service provider validates the request and compares it’s time with the given timestamp.
  3. If the difference in the service provider’s time and the client’s provided time is within some tolerance, the request is serviced

This obviously requires the two clocks to be vaguely in sync. It also allows the resource to be requested by anyone within the timespan of the tolerance. But, it does save a second request to the client.

Self-created One-time Token

This is an amalgam of the Timestamp solution and the Negotiated One-time Token solution. Here, the client creates its own token, as a simple integer of increasing value. The server maintains the last requested value and accepts only requests with a higher number:

  1. Client creates request, using a global long-lived number
  2. Client signs requests and sends it to the service provider
  3. Service provider validates the signature and compares the provided numeric token with the one last used (the tokens can be globally scoped, or scoped for a given resource)
  4. If the provided numeric token is greater than the previous, the request is serviced
  5. The Client increments his numeric token for next time

As with the Timestamp solution, only one request is required. As with the negotiated one-time token solution, the URL can never be used twice. The main issue here is if the client forgets its numeric token. This could be addressed with an additional call to re-establish the token, made only when the Client has determined it no longer knows the last used value.

Unfortunately, this is much more susceptible to race conditions than the Negotiated one-time token. Since the service provider doesn’t know what tokens to expect (only that they should be greater than the last requested one), the client has to ensure that the “create request, submit request, receive response, update local numeric token” cycle is atomic. That is not straightforward.

Update Got another idea from a co-worker

Session Token

When a user access the system that uses the REST API, they get issued a token (via the REST API). This token is just like a session token, with an inactivity timeout and so forth. The token can be manually invalidated via the API, so that when a user logs out or completes some logical task, the token can be invalidated.

This suffers none of the problems of the other solutions, though it isn’t the most secure. However, the security problem it has (using the valid URL before the session times out) is fairly minor, and the tradeoff of getting one request per actual request and no race conditions makes it probably the best way to go.