x = Hash.new { |_,_|lambda {}}.tap { |hash|hash[:key] = lambda {}}x[attr].callFilthy?I kinda like it

— ❺➠ David Copeland (@davetron5000) May 15, 2012

This may seem crazy, but it’s a really just an enhanced use of what Steve McConnell, in “Code Complete”, describes as Table-Driven Methods. Let’s see what that has to do with my crazy Hash construct.

If you haven’t read “Code Complete”, and you are a relatively inexperienced developer, I highly recommend it. My version has a lot of Pascal in it, and I think the latest might use Visual Basic (!), but it doesn’t matter. There’s a lot of useful gems in there.

One of them is “Table-Driven Methods”, which he describes as

a scheme that allows you to look up information in a table rather than using logic statements to figure it out

In the simplest form, you’d replace a case statement with a table lookup. Consider this method that determines, based on the type of credit card, what countries that card can be used in:

1
2
3
4
5
6
7
8
9
10
11
12
13

def countries_usable_in
  case self.card_type
    when 'discover'
      ['US']
    when 'maestro'
      ['UK']
    when 'visa'
      ['US','UK','IE']
    when 'master_card'
      ['US','UK','IE']
    end
  end
end

This could be easily replaced by:

1
2
3
4
5
6
7
8
9
10

CARD_TYPE_COUNTRIES = {
  'discover'    => ['US'],
  'maestro'     => ['UK'],
  'visa'        => ['US','UK','ZA'],
  'master_card' => ['US','UK','ZA'],
}

def countries_usable_in
  CARD_TYPE_COUNTRIES[self.card_type]
end

This is much less complex, both from a formal perspective, and from a general “what’s going on here?” perspective. We replace a bunch of conditionals with a simple mapping. Enhancing this code is simple: we just add a new entry to the CARD_TYPE_COUNTRIES Hash and we’re on our way.

This has a couple of problems with it. You’ll notice that both “visa” and “master_card” map to the same list. What we really want is to treat “discover” and “maestro” as special, and then for any other card type, return our default list of US, UK, and South Africa.

Ruby’s Hash constructor can be given a block that returns the value to use when a key is missing, so that would seem to be useful:

1
2
3
4
5
6
7

hash = Hash.new { |key,value|
  "FOO"
}
hash[:blah] # => "FOO"
hash[:blah] = "BAR"
hash[:blah] # => "BAR"
hash[:crud] # => "FOO"

Of course, this makes it a bit awkward to populate our Hash with the lookup table, because we lose the literal syntax. We can deal with this by using tap, which passes the object called on it to the block passed to it, executes the block, throws away the block’s return value and returns the object on which we called tap. Whoa. Let’s look at an example.

1
2
3
4
5
6

CARD_TYPE_COUNTRIES = Hash.new { |key,value|
  ['US','UK','IE']
}.tap { |new_hash|
  new_hash['discover'] = ['US']
  new_hash['maestro']  = ['UK']
}

Now, when we call CARD_TYPE_COUNTRIES['visa'], this uses the block we gave to the constructor, but CARD_TYPE_COUNTRIES['maestro'] simply returns the literal array we assigned in tap.

So far so good. Now, suppose we have a new requirement to add American Express. Suppose that American Express isn’t supported in African countries, but works everywhere else. Since we don’t want to hard-code what countries are in Africa, we’ll need to consult the database.

1
2
3
4
5
6
7
8

def countries_usable_in
  countries = CARD_TYPE_COUNTRIES[self.card_type]
  if self.card_type == 'american_express'
    countries - Continent.find("Africa").countries
  else
    countries
  end
end

We’ve re-introduced those pesky control structures we were trying to remove. Why can’t we do this?

1
2
3
4
5
6
7
8

DEFAULT_COUNTRIES = ['US','UK','IE']
CARD_TYPE_COUNTRIES = Hash.new { |key,value|
  DEFAULT_COUNTRIES
}.tap { |new_hash|
  new_hash['discover']         = ['US']
  new_hash['maestro']          = ['UK']
  new_hash['american_express'] = DEFAULT_COUNTRIES - Continent.find("Africa").countries
}

This has two problems:

• The database query is only run on app startup, so any changes won’t affect things until we restart (imagine our COUNTRIES table only having countries we support and not all countries; we want to add new countries without an app restart)
• We are running a database query inside a class definition and we don’t necessarily have a guarantee that the database connection is even established at that point.

What we need is a lookup table that calculates its results on demand. Ruby has a structure for that: lambda

1
2
3
4
5
6
7
8
9
10
11
12
13
14

DEFAULT_COUNTRIES = ['US','UK','IE']
CARD_TYPE_COUNTRIES = Hash.new { |key,value|
  DEFAULT_COUNTRIES
}.tap { |new_hash|
  new_hash['discover']         = lambda { ['US'] }
  new_hash['maestro']          = lambda { ['UK'] }
  new_hash['american_express'] = lambda {
    DEFAULT_COUNTRIES - Continent.find("Africa").countries
  }
}

def countries_usable_in
  CARD_TYPE_COUNTRIES[self.card_type].call
end

I find this to be a pretty clean solution. We have all the benefits of a table-driven approach, but only need to specify special cases (thanks to our default block), and have the ability to calcualte our results on demand, based on the current state of the system (thanks to using lambdas and static values). Not too bad.

Let’s take this concept even further. We often write code using an if..elsif..else..end structure that essentially tries various conditions to find one that holds, and returns a value based on that condition.

As an example, we’ll switch domains to my favorite: command line apps. Suppose I need to determine the size of the user’s terminal so I can properly format output. My algorithm will be:

• If the environment variable COLUMNS is a number, use that
• Otherwise, if the command tput exists, run tput lines and return its output
• Otherwise, if the command stty exists, run stty size and parse its output for the value
• Otherwise, return a sensible default.

How does this apply to our lookup table? Essentially we want a table of conditions and, for the first one that holds, perform the calculation to figure out the size. For the sake of clarity, we’ll assume some helper methods, which gives us this code:

1
2
3
4
5
6
7
8
9
10
11

def terminal_columns
  if ENV['COLUMNS'] =~ /^\s+$/
    ENV['COLUMNS']
  elsif command_exists?("tput")
    `tput lines`.chomp.to_i
  elsif command_exists?("stty")
    parse_stty
  else
    DEFAULT_COLUMNS
  end
end

This is a pretty complex routine. What if we need to add Windows support? Another elsif. Lets use our newfound lookup table powers, but instead of using a static key for lookup, we’ll use a dynamic one, based on our conditions:

1
2
3
4
5
6

TERMINAL_SIZES = [
  { :test => lambda { ENV['COLUMNS'] =~ /^\s+$/ }, :val => lambda { ENV['COLUMNS'] },
  { :test => lambda { command_exists?("tput") },   :val => lambda { `tput lines`.chomp.to_i },
  { :test => lambda { command_exists?("stty") },   :val => lambda { parse_stty },
  { :test => lambda { true },                      :val => lambda { DEFAULT_COLUMNS },
]

Note that we’re using an array to keep things ordered, but we’re using an Array of Hash so that our client code will be fairly readable (we’ll see that in a second).

Recall that we want the first expression that returns true, and to return the value associated with that expression. This is a one-liner, thanks to Ruby’s aweomse collections:

1
2
3

def terminal_columns
  TERMINAL_SIZES.find { |size| size[:test].call }.first[:val].call
end

Not bad.

Now, if you come across something like this, but didn’t derive it as we have done here, is it really better? I would argue that it is, especially if you are comfortable with the general concept of table-driven algorithms. In the case of our credit card example, you can see a clear mapping between special cases and the results. For the terminal lines example here, we again have a clean mapping between test and result, and it’s not muddied up amongst control structures.

These tables are also more easily changed: new cases can be added to our credit card type table, and we can easily re-order our terminal size calculation if we decide on a better strategy.

If you find yourself writing an elsif or a case statement; consider using a table-driven method. Ruby provides excellent tools to make this easy to do for any use case.

I’m ecstatic with the results so far, but I have no idea how they compare.

Our books complement each other quite well, and came out around the same time. He wrote his largely on his own, going the self-publishing route, while I wrote mine with the Pragmatic Programmers. I thought I’d write a similar piece from my experience going the “traditional” route.

Beginning

I wrote an initial draft of “Build Awesome Command-Line Applications in Ruby” over November 2010, as part of PragProWriMo. I wrote almost every day, and had about 170 pages of Markdown written at the end. I hadn’t thought too far ahead, but in January of 2011, I decided to submit it to the Prags and see if they were interested.

To my surprise and excitement they decided to go forward with it! I had no idea what to expect, but figured that since I had a good 75% of a “real” book done, it shouldn’t be too much more work for me to finish up the first draft and clean it up. Boy was I wrong.

I spent the next 11 months on the book, reaching “done” by February of this year. That may seem like a long time (it certainly did to me). The reason it took “so long” was entirely due to working with a publisher and editor, but I don’t mean that as a negative. It actually needed to take that long, and the book I shipped was markedly better than anything I would’ve done on my own.

Developing

The Prags don’t just take your manuscript, spell-check it and ship it. They assign a development editor to each book. I had no idea what this was; I thought editors fixed spelling and grammar mistakes. Instead, John Osborn, my editor, provided deep and insightful feedback on every aspect of the book. Did the sections flow together? Are the titles consistent? Did the examples make sense? Of course, I had created plenty of passive voice, subject/verb disagreement and other “advanced grammar mishaps” to keep John quite busy.

Identifying these issues in your own work is hard. It’s also just not possible to get real, usable feedback on your work. Friends and colleagues just won’t give you the brutal, honest feedback you sometimes need. It had been a long time since I’ve been given this sort of feedback, and it hurt a bit (at first) to read comments like these:

• “The overall premise of the book is not immediately clear.”
• “The audience level remains unclear.”
• “After a while…it just gets wearing. Why should I bother to read yet one more example when I know that I’ll be told this was the wrong way to do it?”

These came after John and I had spent a lot of time working my initial manuscript into something publishable, and had gotten a lot of positive feedback from the tech reviewers. Based on the publisher feedback, we were looking at an almost total rewrite, and I wasn’t too happy about it. But, I really believed in the book and wasn’t about to quit now, so I trusted the publishers. Recognizing their experience, I treated this as a learning opportunity. So we persevered.

Persevering

Three chapters later, it was so clear to me that I had been wrong that I couldn’t believe I had felt otherwise. And, I never would’ve gotten there without such honest feedback and a literal team of professionals¹ working to make my book as good as it could be. In the end, the book bears little resemblance to what I had produced during PragProWriMo, and, while I probably could’ve gotten that into a “sellable” shape, it would not have been nearly as good.

So, a total of 15 months of work, including an almost total rewrite, all on my own time, with no money up front. Was it worth it? Absolutely. But, how has it been selling?

On the one hand, I’m not sure. I don’t know how many copies of my book will “pay for” the development costs, or how many will make the publishers “happy” with the result (I’ve put off asking in case the answer is “way more than you’ve sold”). I do know that I’ve sold more than I thought I would, and although I’d love to sell more, I’m really happy with how it’s done.

Performing

As of this post, I’ve sold around 3,000 copies from the Pragmatic Programmers online store². I don’t yet know how many I’ve sold from other channels such as Amazon or Barnes & Noble – hopefully a lot – but sales from the Prags’ store alone feels like more than I was expecting.

Of course, I don’t make as much per book as Jesse does³; that team I mentioned doesn’t come free. But, being featured on the Pragmatic Programmers’ website is unbeatable promotion. This certainly drove a lot of sales. It’s my hope that sales are higher than they would’ve been had the Prags published a version of the book I had originally written, but there’s no way to know for sure.

Now, I don’t mean to imply Jesse’s book is poorly written; it’s not at all. Would it be better if it were edited professionally? I suppose that depends on what you mean? Would it be better written? Certainly; most writing improves with editing and revising. Would it have sold more? Would it have sold enough to cover the costs and time of editing? Again, there’s no way to know.

So where does this leave us? I know that working with John and the Prags has made me a better writer, but would I be confident enough to “go it alone”? Given my lack of notability, I feel I benefit greatly from having my work published and distributed by the Prags. Further, knowing my writing style and abilities as I do, my work will be much higher quality with a team of professionals in my corner.

That being said, I’d still love to try self-publishing at some point, but I’d really love to read a book by Jesse developed with the Pragmatic Programmers.

Today is its first official release at 1.0. Woot!

I’ve also released a tutorial as an “enhanced” iBook, available now from the iBookstore as a free download. This isn’t just a big ream of code and text, but a step-by-step walkthrough, with screencasts, on how to use Methadone, along with some detailed discussion on some of Methadone’s more useful features. It’s all presented beautifully as a “textbook”-style iBook that looks great on an iPad 1 and stunning on the new iPad, thanks to iBooks Author and the retina display.

Read the tutorial, read the RDoc, install the library, and feel free to submit patches.

I heartily endorse this and highly recommend this approach if you want to “get into Rails”.

When I first heard about Rails, I bought a copy of Agile Web Development with Rails and was instantly confused as to what I was looking at. What were all those @ signs for? Why are there pipes after the do keyword? Where were all these methods coming from?

The answer was: Learn you a Ruby.

I am not the type programmer to just copy/paste code I don’t understand. I don’t like “magic¹”. I want to know, or at least have an inkling of, how the tools I’m using work. And there’s no hope in understanding Rails without understanding Ruby.

I sold the book, and, using a Ruby book available on Safari, started to learn Ruby the language. I ignored Rails, and didn’t even think about it. Instead, all of the command line apps that I would historically write in Perl, I wrote in Ruby. I wrote a lot of Perl in Ruby at first².

Eventually, I figured out how the language worked and, when it came time to do some serious Rails programming, I wasn’t surprised by much. I mostly just had to learn to navigate the documentation. Even when I didn’t know how something worked, I could usually have a guess, and I was usually right. When I wasn’t, I learned something new about Ruby.

It probably wouldn’t have taken so long if I’d taken a more focused approach like what Ruby Off Rails is doing (and, LivingSocial’s Hungry Academy as well, which starts with just Ruby before moving to Rails).

If you’re tired of slinging Java/C#/PHP, but can’t seem to get focused to learn Rails or Ruby, Ruby Off Rails seems like a good way to go. It’s not very expensive, and it’s geared around writing code, not learning API documentation.

Anyone have a (formal / step-by-step) protocol for code reviews?

Yes, I do. At Opower, we were pretty serious about code reviews. We didn’t review every commit, but we did review a lot, and, after a while, fell into a pretty decent routine. This is an adaptation of that that I think is pretty decent.

Prepare

The code authors should send out the diff, either as a pull request, or whatever the equivalent is in the tool of choice. This should include:

• What the purpose of the change is
• Links to any supporting tickets, documentation, etc.
• A Place To Start

With all that, the reviewers should get an email, inviting them to review the code. This is important, because you need to identify the people whose feedback you want. Don’t just email people_who_installed_ruby@theuniverse.com; email people who either should see your change, or whose feedback you want.

A Place to Start?

Depending on the complexity of the change, this could be the name of the class/method where one should start reading, or a full-blown design document explaining the approach and structure of the change. The point is that the reviewers have no clue how your code works, and need somewhere to get going.

Asynchronous Review

This is “review by online tool”, e.g. Github Pull Requests. To make this work, you must have a tool that allows per-line commenting and discussion. The reviewers will start at the Place to Start, review the code, and comment on things. Reviewers should comment on anything they see fit, and they should follow Wheaton’s Rule: Don’t be a dick. The reviewee promises not to take anything personally. This can be hard to do, especially if you are new to the team, or inexperienced. I’d recommend that for first-time employees, skip this and go to the Synchronous Review as it can be less intimidating.

Responding to comments

The reviewee should absolutely respond to comments. I would expect comments to be one of these three responses:

• You are right, I will change that.
• You are wrong, because of some explanation, and perhaps I’ll drop some comments or better-name my variables before I push this to make it clear.
• I’m confused, can you elaborate?

Uploading changes

As important as it is that your review tool allow per-line comments, it’s equally important that your review tool allow new changes to be added to the review without blowing away the comments. Reviewers should be able to tell that you’ve made changes based on their feedback and see if it makes sense

Stylistic Comments

Early reviews at Opower wasted an inordinate amount of time on stylistic things. We eventually adopted a house style and our reviews moved on to important things. You are not a special butterfly and you are not a snowflake who can only express him or herself through your clever variable names and indenting style. Follow the house style, or at least the style of the code you are in so you can get better feedback on your code instead of arguing about where commas should go.

Synchronous Review

If there are too many “I’m confused” comments, or the overall design/approach of the solution seems undesirable, it can greatly help if you hold a meeting with the interested parties to talk out the issues in the review, rather than creating an endless stream of comments in the review tool.

This review should be attended only by those with an interest in the outcome. It should be timeboxed to hopefully 30 minutes, but not more than an hour, and the end result should be a “todo” list for the reviewee to correct the issues. The ability to project the code on the screen is a plus, and the code authors best know how to navigate their codebase. Senior developers have a responsibility to school them on this during the review. It’s the only way you’ll learn. The issues would be of the form:

• Code author didn’t understand the problem, and a larger rework is indicated
• Reviewers didn’t understand the solution, and, after explaining, the author either reworks the code to be more understandable, or provides documentation to clear things up

Completion

When the code is done to the satisfaction of those involved, the final commit should contain a link to the review in the review tool. This is crucial for future archaeologists that might want to know what’s going on; seeing the discussion can be helpful.

Doing this with Github

Github’s pull requests are less than ideal for this, depending on how you work. Primarily, a reviewer wants to see the diff between the current system and how the system would look with the new code applied. Last I checked, this view in Github doesn’t allow per-line commenting, making it almost useless.

What I’d recommend is to squash the commit onto a branch specifically for the review (e.g. reviews/TICKETNUM-DESCRIPTION). When the code author needs to add changes in response to the review, just add those diffs to the branch. When everything is done, squash all that, and merge it. One diff, one thing to deal with and understand.

If you really want the sausage-making to be part of mainline history, then merge your changes plus one additional change representing all changes you made based on the review, with the message being a link to the review branch in Github.

Doing this with Crucible

Crucible was tailor-made for this, and all you need to do is choose changesets and have at it. You can add diffs as needed, and the commenting system is rich. Crucible is not as polished and slick as Github, but it works great.

Conclusion

This may sound like a lot, but it’s really lightweight, once you start doing it, and it’s way better than zillions of emails or long, horrible meetings. Just try it.

• have a clear and concise purpose
• be easy to use
• be helpful
• play well with others
• delight casual users
• make configuration easy for advanced users
• install and distribute painlessly
• be well-tested and as bug free as possible
• be easy to maintain

In this post, I’ll illustrate each of these facets (along with a test of the tenth chapter on color and formatting), via a code walkthrough of a simple command-line app I created for work.

LivingSocial (where I work) processes thousands of credit card transactions per day, across a highly distributed, asynchronous system. When things go wrong, the log files are the first place I look to find answers. This means that grep is my go-to tool for analysis. Even though grep can highlight search terms in output, with long and complex log lines, it can be hard to pick out just what I’m looking for. I needed a tool to just highlight text, but not actually “grep out” non-matching lines.

To the command-line!

So, in just a few short hours, hl was born. I wrote it using TDD, and, even though it’s barely 100 lines of code, it hits all the notes of an awesome command-line app (if I do say so myself :). Let’s go through all nine of our “facets of an awesome command-line app” and see what the fuss is about.

Have a Clear & Concise Purpose

The best way to have a clear & concise purpose is to do one thing, and one thing only. hl highlights search terms in any output to assist with visual scanning of output. It doesn’t highlight multiple terms, and it doesn’t remove non-matching lines. It just highlights terms. One thing, and one thing only.

Be Easy to Use

This is a big topic, but here’s an example of using hl:

1

$ grep 987876736 my_logs.log | hl credit_card_token

hl does what it’s asked, by default, without a lot of fuss, just like any other UNIX command. It has options, but you never need to worry about them in most cases. Of course, if you are curious about those options, that leads to our next facet.

Be Helpful

hl is based on methadone, which is a proxy to OptionParser, which is the tool to use for parsing the command-line in Ruby. It’s very powerful, and generates a canonical, documented UI for your app:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

$ bin/hl --help
Usage: hl [options] [search_term] [filename]

Highlight terms in output without grepping out lines

v1.0.0

Options:
    -c, --color COLOR                Color to use for highlighting
                                     (red|green|yellow|blue|magenta|cyan|white)
                                     (default: yellow)
    -b, --[no-]bright                Use bright colors
    -n, --[no-]inverse               Inverse highlight
    -u, --[no-]underline             Underline highlight
    -p, --regexp PATTERN             Search term as explicit option
    -i, --[no-]ignore-case           Ignore case in match
        --version                    Show help/version info

Default values can be placed in the HL_OPTS environment variable

Note how much OptionParser gives us:

• Ability to describe our app, its version, and basic invocation syntax
• Nicely formatted list of options and descriptions
• Ability to accept “negatable” options (we’ll talk about that in a second)

Further, I’ve gone to the trouble to make sure that --color clearly indicates the acceptable values as well as the default. Finally, I’ve made sure that all options are available in short-form (for easy typing on the command line) and long-form (for clarity when scripting and configuring our app).

Here’s the code that makes this happen (if you aren’t familiar with methadone, the method on behaves almost exactly like the on method in OptionParser):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

#!/usr/bin/env ruby

require 'optparse'
require 'methadone'
require 'hl'

class App
  include Methadone::Main
  include Methadone::CLILogging

  main do |keyword,*filenames|
    # main logic here
  end

  description "Highlight terms in output without grepping out lines"

  options[:color] = 'yellow'
  colors = ['red', 'green', 'yellow', 'blue', 'magenta', 'cyan', 'white']
  on("-c COLOR",       "--color","Color to use for highlighting",colors,"(#{colors.join('|')})")
  on("--[no-]bright",     "-b",  "Use bright colors")
  on("--[no-]inverse",    "-n",  "Inverse highlight")
  on("--[no-]underline",  "-u",  "Underline highlight")
  on("--regexp PATTERN",  "-p",  "Search term as explicit option")
  on("--[no-]ignore-case","-i",  "Ignore case in match")

  arg :search_term, :optional
  arg :filename, :optional

  version Hl::VERSION

  defaults_from_env_var 'HL_OPTS'

  go!
end

Methods like arg, version, and description are helpers from methadone (see the intro for more), but note how little code it takes just to make a great and polished UI.

The second part of a helpful app is to include more detailed documentation. For a command-line app, this is expected to be in the form of a man page. If you installed hl with RubyGems, try this:

1

$ gem man hl

You should see a nicely formatted man page (which also happens to be the README for the github project)! Creating a man page is extremely simple thanks to ronn. ronn converts Markdown to troff, the format used by the man system. Just add this to your Rakefile:

1
2
3
4
5
6
7
8
9
10

require 'methadone'
require 'fileutils'

include Methadone::SH
include FileUtils

task :man do
  sh 'ronn --markdown --roff man/hl.1.ronn'
  mv 'man/hl.1.markdown','README.md'
end

And, your gemspec just needs:

1
2

  s.add_development_dependency('ronn')
  s.add_dependency('gem-man')

You’ll also need to include the generated file man/hl.1 in your files in your gemspec, but if you’re using the gemspec created by Bundler, this happens automatically as long as the file is in source control.

That’s it. Now your app has a great UI and a man page, and all you had to do was drop a few lines of code and write a short Markdown file (which you’d write anyway, since you are making a README, right?).

In addition to being helpful to humans, awesome command-line apps should be helpful to other commands.

Play well with others

An app that “plays well with others” on the command line, basically means that it acts as a filter. Text comes in, gets processed, the processed text goes out. The expectation is that text from any other “well playing” program can be input into our program, and that our program’s output can be piped into another program as input.

Since the purpose of our app is to add ANSI escape codes to the output for assistance with human visual scanning, we can’t claim that our output plays well with others; it’s not designed to. But, we can still play well with the output from other apps.

We saw that hl was designed to take input from a tool like grep. hl can also highlight terms from any number of files given to it on the command line. You can do this transparently in Ruby using the awesome ARGF, however Methadone doesn’t support ARGF (a sad fact I learned while writing this app, and something I’ll address in the near future), so here’s how did it (a few comments added to indicate what’s going on):

1
2
3
4
5
6
7
8
9
10
11
12
13
14

# filenames is a possibly empty list of strings
files = if filenames.empty?
          [STDIN]
        else
          filenames.map { |_| File.open(_) }
        end
# files is now an Array of open IO objects
begin
  # highlighting code
ensure
  # we close the files since we didn't open them in "block" form; closing STDIN is OK to do
  # since we know our app will soon exit
  files && files.map(&:close)
end

Again, ARGF handles this transparently, but the point is, we want the standard input and a provided list of files to be treated the same by our program, and this is how I did it.

Since our app is similar in concept to grep, I thought it would be nice if users familiar with grep could be instantly familiar with hl.

Delight Casual Users

This is a “level up” from “being easy to use”. The idea behind the term “delight” is to provide a level of polish and attention to detail that your users will appreciate if they’re observant, but hopefully not even notice, because your app “just works”.

Since hl, like grep, is used for filtering and examining text files, I chose my command-line options to match grep’s where i could. Initially, I had the short-form of --inverse as -i. When I later added the ability to do a case-insensitive match, I realized that -i is the option to grep for “case-insensitive”. I quickly changed --inverse to have -n as its short-form, and made -i and --ignore-case the options for case-insensitivity. These are the same values that grep uses, so a user who might subconciously type hl -i expecting a case-insensitive match will get it.

Further, I allowed the user to specify the search term either as a command-line argument, or as the argument to -p or --regexp, which are the option names grep uses. It’s a basic principle of design that things that are the same should be exactly the same, so I used grep as my guide when hl implemented similar features.

Of course, power users love to customize things.

Make Configuration Easy

In the book, I talk about using YAML as a configuration format for an .rc file. This can be very useful for complex apps, but another technique that’s handy is to allow an environment variable to hold default options. grep does this via GREP_OPTS and if you were paying attenion, you noticed this line in bin/hl:

1

  defaults_from_env_var 'HL_OPTS'

This tells methadone to look at the environment variable HL_OPTS (as well as the command line) for any options. These options are placed first in ARGV, essentially like so:

1
2
3

String(ENV[@env_var]).split(/\s+/).each do |arg|
  ::ARGV.unshift(arg)
end

(Note the use of String to make sure that nil gets turned into the empty string, saving us an if statement). Methadone does this before parsing ARGV. Using unshift means that any options the user specifies will come after those in HL_OPTS and therefore take precendence:

$ export HL_OPTS=--color=cyan
$ grep foo some_log.txt | hl --color=magenta

This is the same as

$ grep foo some_log.txt | hl --color-cyan --color=magenta

This is also why I’ve provided the “negatable” forms. Suppose you generally wanted inverse:

$ export HL_OPTS=--inverse

If you wanted to run hl without inverse, but there was no negatable option, the only way to turn it off would be to unset the environment variable. With the negatable forms, it’s simple:

$ grep foo some_log.txt | hl --no-inverse

Since the user’s command-line options take precedence, things work out, but you can still configure your defaults.

Finally, I’d recommend that you use the long-form options in your configuration. In other words, if you prefer bright and inverted highlights, do this:

$ export HL_OPTS='--inverse --bright'

As opposed to

$ export HL_OPTS=-nb

The second form is more compact, but your configuration is going to be read more than written, and, 6 months from now when you are going through your .bashrc, you’re going to appreciate seeing things spelled out; you’ll know instantly what the configuration does and don’t have to wonder about what -n means.

Distribute Painlessly

RubyGems:

$ gem install hl
$ hl --help

That is all.

Be well-tested

I wrote hl entirely using TDD and entirely using aruba. Here’s a sampling:

1
2
3
4
5
6

  Scenario: Highlights with case insensitivity
    Given a file named "test_file" with the word "FOO bar foo" in it
    When I successfully run `hl -i foo ../../test_file`
    Then the entire contents of "test_file" should be output
    But the word "foo" should be highlighted in yellow
    And the word "FOO" should be highlighted in yellow

It was very easy to do this, although aruba could use a man page for easier reference. I had to jump into its source too many times to get reminded of the syntax of the steps it provides. Aruba also strips out ANSI escape sequences, which made testing hl a bit tricky. There appears to be an option to prevent this, but I couldn’t get it to work, so I just used Aruba’s internal API:

1
2
3
4
5
6

Then /^the word "([^"]*)" should be highlighted in (.*$)$/ do |keyword,color|
  # #color is provided by rainbow, which we'll talk about in a bit
  expected = keyword.color(color.to_sym)
  # assert_partial_output and all_stdout are provided by aruba
  assert_partial_output(expected,all_stdout)
end

I still recommend aruba and cucumber, as it forces you to think about how users will use your app first, not how to implement it. In fact, my initial implementation was a big hacky mess of stuff inside the main block. Once the tests were in place, I refactored it to be a lot cleaner.

Be Easy to Maintain

As I just mentioned, I was able to use my tests to refactor my code. As such, the main block of hl is pretty simple:

1
2
3
4
5
6
7
8
9
10
11
12
13

main do |keyword,*filenames|
  if options[:regexp]
    Array(filenames).unshift(keyword)
    keyword = options[:regexp]
  end

  exit_now! 'search term or --regexp/-p required' if keyword.nil?

  keyword = keyword.dup
  highlighter = Hl::Highlighter.new(options)

  puts highlighter.highlight(filenames,keyword)
end

This is the sort of logic you want in your main block:

• Handling the keyword-from-argument and keyword-from-command-line-option case
• Simple error checking
• Duping the keyword (since it comes in frozen)
• Calling our Highlighter class to do the real work

We defer all non-UI logic to the Highlighter class. I decided to make each instance of the class able to highlight any files repeatedly based on a configuration, so the constructor takes in the formatting options, and the method highlight takes the list of filenames and the search term.

The actual highlighting is made possible via lots of list comprehension:

1

files.map { |_| _.readlines}.flatten.map { |_| highlight_matches(regexp,_) }.join("")

If you aren’t comfortable with this use of chained calls, it can be very powerful. What this does is:

• Map each file to an array of its contents as lines. [foo,bar] becomes [ ['first line of foo\n','second line of foo\n'],['first line of bar\n'],['second line of bar\n']]
• Flatten that array of arrays to just one list of all lines of all files. Our example array becomes: [ 'first line of foo\n','second line of foo\n','first line of bar\n','second line of bar\n']
• map those lines to the lines with the search term highlighted. Supposing we wanted to highlight the word “line”, our array becomes: [ 'first \e[33mline\e[0m of foo\n','second \e[33mline\e[0m of foo\n','first \e[33mline\e[0m of bar\n','second \e[33mline\e[0m of bar\n']
• join them all together into one big string "first \e[33mline\e[0m of foo\nsecond \e[33mline\e[0m of foo\nfirst \e[33mline\e[0m of bar\nsecond \e[33mline\e[0m of bar\n"

Granted, this approach will probably have trouble with extremely large input, but hl was designed to work with the output of grep, so hopefully we won’t have too much (I’ve already decided I need it to work with tail ).

Breaking the rules

Color and formatting are not typically associated with awesome command-line apps; too much of it makes an app hard to use with other apps. But, the whole purpose of hl is to colorize output, so for that, I used rainbow, which is a pretty simple enhancement to String that allows coloring and formatting. We can see it in action in the highlight_string method of Highlighter:

1
2
3
4
5
6
7

def highlight_string(string)
  string = string.color(@options['color'].to_sym)
  string = string.inverse if @options[:inverse]
  string = string.bright if @options[:bright]
  string = string.underline if @options[:underline]
  string
end

Each method called on string is a method provided by Rainbow. These methods return a new string with the appropriate ANSI escape codes added.

In Conclusion

Hopefully, you’ve seen that it’s really not that hard to make an awesome command-line app. I was able to write hl in just a few hours, using TDD and the end result is a highly polished, well-documented, easily installable and maintainable piece of software that will be a part of my command-line arsenal for quite a while. You can do this, too. There’s a lot more detail and in-depth explanations in my book, which you should buy right now :)

Here’s a link to download from my talk on JRuby and Threads.

If you just want the “exercises”, here they are:

Echo Server

• Listen on a port
• Respond to each request in a new Thread
• Extra Credit: Record stats on requests in a shared data structure

Connection Pool

• Allow N clients to access X shared instances of, say, Redis (where N > X)
• Clients “check out” a connection and get exclusive access
• Clients “check in” when done
• Instances get re-used

It’s a nonsensical journey through the computer, ending on the command-line. It’s got drama, special effects, and a stern soundtrack :)