RubySpec: Bringing Ruby Test Suites Together

Hooray! The RubySpec Project, a collection of runnable specifications for Ruby 1.8.6ish behavior, has graduated into its own domain. Finally there's a lively, fast-moving, independent project to create a Ruby specification and test kit. And it's already well on its way.

Better documentation on how to pull the specs, update them, and use them for your own Ruby implementation (you do have a Ruby implementation, don't you?) are still being ironed out, but the repository is already available at the RubySpec github address, so you can pull them and start reading and running them. Also see the MSpec github for a lightweight (lighter than RSpec) tool to run the specs with.

But this post is not just about the the RubySpec project...Brian Ford is putting together an official announcement for that as we speak. This post is a call to action.

JRuby currently encompasses something like 6 separate test suites:

• Our old JRuby test suite using "minirunit", a small runit clone no longer in wide use (the camelCase.rb tests at that URL)
  
• Ryan Davis and Eric Hodel's "BFTS" suite, a narrow but deep set of tests for a few core classes
• Our newer set of JRuby tests using test/unit (the underscore_case.rb tests at that URL)
  
• MRI's own set of tests, from the Ruby 1.8 repository (link is to our somewhat out-of-date copy)
  
• A test/unit port of the Rubicon test suite, originally written by Dave Thomas while writing the Pickaxe books
• The ruby_test test suite, a suite of tests created by Daniel Berger for his projects (link is to our out-of-date copy)
  

We don't want to run these tests forever...we would rather just run the RubySpec. So this is where we need help.

Much of these tests are already encompassed in the RubySpec specs. BFTS, for example, focuses only on a very few core classes, which have been heavily covered in RubySpec. In many cases, these test suites even overlap each other, meaning that our 3 minute test run could probably be a lot shorter. If we could just replace our test suite with the RubySpec (modulo JRuby-specific bits like Java integration), we'd be very happy.

But we can't afford to do that unless we know we're not throwing away good tests. The RubySpec is a work in progress, and there are always going to be gaps. It would be folly to throw away our tests without consideration. So that's where you come in.

We need to start at A and work our way through Z, porting over any test cases that aren't covered in the RubySpec.

I started the process tonight, adding a number of missing cases from our test_array.rb script and deleting everything I ported and everything that was already covered. It took perhaps an hour to go through, and it was of a reasonable size. Many other scripts will be much smaller, some will be larger.

The benefits extend far, far beyond JRuby of course. By adding missing test cases, we're going to ensure that all new implementations have a complete spec to go on. We're going to make sure there aren't a lot of incompatibilities you users have to deal with. And we're going to show all those other languages (who are still laughing at our lack of a spec) that we can do this in our own Ruby way.

So what are you waiting for? Contact Brian Ford and get access to the specs (perhaps after paying a one-patch toll)...have a look at the JRuby test repository...pick a file, and start comparing. Tell your friends, email your favorite Ruby list, blog and reblog this effort. The time is now to pull together all the disparate suites into one. RubySpec is ready!