Posts Tagged ‘Gliffy’

Git, GitHub, forking: the new hotness

Thursday, February 5th, 2009

While working on my Gliffy Ruby Client, I decided I wanted a better way to describe the command line interface. Finding nothing that was any good, I whipped up GLI and refactored my Giffy command line client to use it. While doing that, I finally got annoyed at technoweenie’s version of rest-client, and also noticed that the original author’s had totally changed interfaces. So, clicked the nice “Fork” button on GitHub to get my own copy and fixed the issues. But that’s not the cool part. The cool part is that I can change my Gliffy gem to depend on my rest-client implementation and, viola! No special instructions, no hacks, no nothing. This is a really cool thing that would be difficult with Subversion, impossible without RubyGems, and downright painful without GitHub.

Command line interface for Gliffy

Wednesday, January 14th, 2009

My command line interface for Gliffy is relatively complete. It works pretty well, though the error handling isn’t very clean. It’s written in Ruby (RDoc here) and can be used as a Ruby client for Gliffy.

I decided on Ruby since that would be the most fun and didn’t require learning a new programming language. I initially tried to make an ActiveRecord-style domain-based interface, but it was just too difficult and it was hard to see the real benefit. At the end of the day, I think integrating Gliffy into another application is a relatively simple thing, and a procedural interface would probably be the easiest way to do that. So, I modeled it after the PHP client library, more or less.

The command line interface uses the Ruby client library and provides just the basic functions I need:

> gliffy ls
321654 Some Diagram
987654 Some Other Diagram
> gliffy edit 321654
# Takes you to your browser to edit the diagram

I live on the command line, so this is much more expedient than logging into Gliffy and navigating the UI to edit a diagram.

I’m already feeling like providing access to the folders via the command line would be helpful (they are exposed in the Ruby client of course). Not sure how much the API will ultimately change (it’s in private beta now), but hopefully not too much.

Gliffy API private beta: what should I do?

Friday, December 12th, 2008

Gliffy hooked me up with access to the private beta of their API (which I helped design and implement). I create a PHP client and experimental MediaWiki plugin to validate the API while working for them, and now I want to get something else going in my spare time.

My first thought was to make a Ruby client, because I think it would be fun and relatively easy. But, I have to admit that a Wordpress plugin would be more useful to me personally. That being said, A Trac extension would be useful at work, since we are using Trac (which is python based, and I can’t say I’m too interestedin Python at the moment). I think if GitHub allowed git access to project wikis, it would be cool to allow easier integration of Gliffy diagrams to GitHub project wikis.

At any rate, I don’t have tons of time outside of work, so I want it to be something easily achievable, and also something Chris and Clint are not likely to work on themselves….

Gliffy updated their site!

Tuesday, December 9th, 2008

Though I’m no longer working for Gliffy, I’m excited to see that they updated their site with some of what I worked on! Awesome!

Specifically, I worked on the folder organization system that they added to replace the tagging system. This fell out of the API work that I did (which I’m assuming is in private beta right now, but I’m not sure). I also worked on a feature that, while painful as a developer, is my favorite new thing about Gliffy: the basic account no longer has a five-diagram limit! That means for free, you can create unlimited diagrams. The catch is that your diagrams are all public, but I think it’s a great way to enhance the functionality while subversively getting their name out to more people.

Are you emailing yourself your log errors? You should be.

Friday, September 26th, 2008

Time and time again, users complain about an application crashing on them or otherwise not working. They don’t provide you any info and it’s hard to repeat. You check out the log, but there’s thousands (or millions) of entries and you have no clue where their error occured. Worse, if you are deploying a RIA, the log may be on their computer and not available.

On my last project we experienced this scenario so much that we instituted two things

  • All messages logged with Level.ERROR in log4j would be emailed to us
  • All exceptions caught on the client would be packaged and sent back to the server and logged at Level.ERROR level (thus emailing them to us

After the initial deluge of emails, we found a lot of bugs. I mean a lot of bugs. The annoying, intermittent kind that are hard to reproduce. Further, by judicious use of logging, we discovered a lot of mis-configured environments and other problems without having to get users to mail us their logs.

At Gliffy, they are doing the same thing. Right now, we’re testing a bunch of new features and the stage instance just sent me a bunch of emails, all indicating configuration problems, which is the exact kind of thing that can be hard to track down.

Setting it up using log4j is dead simple:

log4j.appender.mail=org.apache.log4j.net.SMTPAuthenticateAppender
log4j.appender.mail.SMTPHost=@SMTP_HOST@
log4j.appender.mail.UserName=@SMTP_USER@
log4j.appender.mail.Password=@SMTP_PASS@
log4j.appender.mail.Authenticate=true
log4j.appender.mail.From=errors@gliffy.com
log4j.appender.mail.To=@SMTP_LOGGER_FAILURE@
log4j.appender.mail.Subject=Errors from @SMTP_DESC@
log4j.appender.mail.BufferSize=1
log4j.appender.mail.Threshold=ERROR
log4j.appender.mail.LocationInfo=true
log4j.appender.mail.layout=org.apache.log4j.PatternLayout
log4j.appender.mail.layout.ConversionPattern=%d %p%n%t%n%c:%M:%L%n---%n%m%n---%n%n

In my previous job I even created a customized layout to format the emails in such a way that our code was highlighted and GMail didn’t compress things into threads.

If you aren’t doing this, you should be. Now.

Getting Rake’s PackageTask to depend on generated files

Wednesday, September 17th, 2008

Been playing with Rake lately and decided to use it to package up the PHP Client Library for the Gliffy integration API. Didn’t seem to make sense to use ant for something that amounts to creating a tarball. make would be appropriate here, too, but I figured it would be cool to use Rake and there’s not really much harm in doing so.

A large annoyance is Rake::PackageTask. This is a seemingly handy task that creates tars, zips, etc. and is pretty useful. It’s not a task in and of itself, but it creates the :package task:

Rake::PackageTask.new("gliffy-php-client",GLIFFY_VERSION) do |p|
    p.need_tar = true
    p.need_zip = true
    p.package_files = SRC_FILES + EXAMPLE_FILES + DOC_FILES
end

Unfortunately, this doesn’t do what it seems to do. DOC_FILES is the list of documentation files output by phpDocumentor and are not checked into version control. The syntax of the PackageTask makes it appear that the code in the block will run when the :package task executes, however this is not the case. This code is initialization code. So, I tried:

# DOC_DIR is the dir generated by phpdoc, a task elsewhere
# uses this to kick off phpdoc
task :package => DOC_DIR

The result is that the tarball and zip files are created and then the documentation is generated. The reason is that PackageTask.new creates a set of tasks and the actual creation of the tarball/zip file is done via a file task, upon which :package is dependent. So, the real dependency I created was:

task :package => "gliffy-php-client.zip" "gliffy-php-client.tgz" DOC_DIR

Examining the source code, a task named for the directory created by :package is created. This task is dependent on the package_files set up in the constructor. So this is the task I need to use:

# Have to keep a reference to the PackageTask object
package_task = Rake::PackageTask.new("gliffy-php-client",GLIFFY_VERSION) do |p|
    p.need_tar = true
    p.need_zip = true
    # Executed BEFORE any other tasks; DOC_FILES don't exist  yet
    p.package_files = SRC_FILES + EXAMPLE_FILES
end

file package_task.package_dir_path => DOC_DIR

file DOC_DIR => SRC_FILES + EXAMPLE_FILES do |t|
    system("phpdoc #{PHP_DOC_ARGS}");
    doc_files = FileList.new(DOC_DIR + "/**/**");
    # Have to add these files to the package_task file list
    package_task.package_files = package_task.package_files + doc_files
end

This is definitely a hack, because I’m depending on the internal implementation of the PackageTask. It really needs a facility for including generated files. In make, I could just send the directory DOC_DIR to tar and it would pick up everything. In Ant, I’d probably have to spawn another ant, since ant sets all property vaules at startup time.

Test REST Services

Friday, September 12th, 2008

In my reply to a post on Tim Bray’s blog about using RSpec for testing REST services, I briefly described a project I’m working on, based on the work I’ve been doing at Gliffy, which is a testing framework for REST services called, unsurprisingly, RestUNIT.

For Gliffy’s REST-based integration API, I needed a way to test it, and hand-coding test cases using HTTPClient was just not going to cut it. Further, requests to Gliffy’s API require signing (similar to how Flickr does it), and our API was going to support multiple ways of specifying the representation type as well as tunneling over POST.

So, it occured to me that there was a lazier way of doing this testing. All I really needed to specify was the relative URL, parameters, headers, method, and expected response. Someone else could do the signing and re-run the tests with the various options (such as specifying the MIME Type via the Accept: header, and then again via a file extension in the URL).

I ended up creating a bunch of text files with this information. I then used a Ruby script to generate two things: an XML file that could be deserialized into a java object useful for testing, and a PHP script to test our PHP client API.

The Ruby script would also do things like calculate the signature (the test text files contained the api and secret keys a Gliffy user would have to use the API) and generate some derivative tests (e.g. one using a DELETE, and another tunneling that over POST). The testing engine could generate some additional derivative tests (e.g. GET requests should respond to conditional gets if the server sent back an ETag or Last-Modified header). All this then runs as a TestNG test.

The whole thing works well, but is pretty hackish. So, RestUNIT was created as a fresh codebase to create a more stable and useful testing engine. My hope is to specify tests as YAML or some other human-readable markup, instead of XML (which is essentially binary for any real-sized data) and to allow for more sophisticated means of comparing results, deriving tests, and running outside a container (all the Gliffy tests require a specific data set and run in-container).

The test specification format should then be usable to generate tests in any other language (like I did with PHP). I’m working on this slowly in my spare time and trying to keep the code clean and the architecture extensible, but not overly complex.

Schema for REST services

Thursday, September 11th, 2008

I’m currently working the integration API for Gliffy, which is a REST-based service. The API is fairly stable and we’re readying a few ancillary things for release. One of those is the documentation for the API. I found it quite difficult to completely describe the REST services and ultimately ended up creating something that lists out “objects” and “methods”, even though the API is not really object-based. For example, the object “Diagram” has a “method” called “list”; to “call” it, you do an HTTP GET to accounts/your account name/diagrams.

The original spec I created to work against (and thus, our initial draft of API documentation) was basically a list of URLs and the HTTP methods they responded to. Not very easy to navigate or understand on a first sitting. Some sort of schema to describe the REST API would have been really helpful (along the lines of an XML Schema). Such a schema could facilitate documentation, testing, code generation.

As an example, consider some features of the Gliffy API: you can list the users in an account, list the diagrams in an account and reference an individual diagram via id. Here’s a YAML-esque description of these services:

accounts:
  kind: literal
  desc: "Reference to all accounts"
  POST:
    desc: "Creates a new account"
    parameters:
        - account_name
            required: true
            desc: "Name of the account you want to create"
        - admin_email
            required: true
            desc: "Email address of an administrator for the new account"
  children:
     account_name:
       kind: variable
       desc: "The name of your account"
       GET:
         desc: "Returns meta-data about the account"
         parameters:
            - show_users
              required: false
              desc: "If true, users are included, if false, they are not"
       children:
         diagrams:
           kind: literal
           desc: "All diagrams in the account"
           POST:
             desc: "Creates a new diagram"
             parameters:
               - diagram_name
                 required: true
                 desc: "Desired name for this diagram"
               - template_id
                 required: false
                 type: numeric
                 dsec: "If present, the id of the diagram to copy, instead of using the blank one"
           GET:
             desc: "Gets a list of all diagrams in this account"
           children:
             id:
               kind: variable
               type: numeric
               desc "The id of a particular diagram"
               GET:
                 desc: "Gets the diagram; the requested encoding type will determine the form"
                 parameters:
                   - version:
                     desc: "The version to get, 1 is the original version.  If omitted, current version is retrieved"
                     required: false
                     type: numeric
                   - size:
                     desc: "For rastered formats, determins the size
                     type: enumeration
                       - L
                       - M
                       - S
               DELETE:
                 desc: "Deletes this image"
          users:
            kind: literal
            desc: "All users in the account"
            GET:
              desc: "gets a list of all users in the account"

Since “accounts” is the only top-level element, we are saying that every request to this service must start with accounts/. It has one child, which is a variable value for the account name. It is untyped, so any potential string is allowed. That element has two possible children: diagrams and users. diagrams indicates that it responds to the HTTP methods POST and GET. A POST requires the parameter diagram_name, while the parameter version is optional.

A standard format like this could easily be used to generate documentation, expectations, test cases, and even stub code. This format could even be delivered by an OPTIONS call to a resource. I realize there is not much standardization around how to design and implement a REST service, but something like this could at least be a stake in the ground and support a specific method.