Saturday, June 23, 2007

Camping at O'Reilly

Rumors of my demise were greatly exaggerated!

I've been mostly MIA the past couple weeks, largely due to the Ruby Kaigi and related events in Tokyo and a short family trip this past week to Lake Michigan's eastern shore (warm sandy beaches and a much-welcomed rest). But I figured I'd check in so everyone knows I'm still around and the wheels are still turning.

I'm at Foo Camp this weekend, and so far it's been a great time. Lots of good discussions, games of Werewolf, and staying up too late (I think I got about 3.5 hrs of sleep...more than enough!) Today I'm going to try to talk about something outside my usual subjects, mostly as a discussion facilitator, but with a bit of opinion tossed in for good measure. Here's the description I posted to the Foo calendar:

The Transformation Age

This is a collective presentation. There are no attendees; only nodes in the matrix. Expect to share.

Part 1: Individuals to collectives, isolated to connected, The Next Evolution.

As the exchange of information has become progressively more rapid, the communication boundaries more transparent, we are transforming into a new kind of collective entity. The swing from isolated individuals or isolated communities with their own distinct stores of knowledge toward a single global community with all stores of knowledge available to everyone is more than just an interesting trend...it represents a new evolutionary direction humanity is taking. It is the dawn of the transformation age for humankind.

We will explore what it means to be more and more a collective being in this new era, what it will mean for our children and grandchildren and nth grandchildren as pervasive interconnection becomes the norm. As all minds become so intimately connected that being disconnected from the global consciousness is like losing an arm...or one's own identity. And we'll look at how this new evolution is affecting and has affected successful and unsuccessful technologies, business models, and governments over the past several decades.

Part 2 (time permitting): Knowledge represented as transformation of information rather than as desperate attempts to snapshot or categorize information at a given point in time.

The idea of categorizing information has served us fairly well when information itself was slow to change, slow to be communicated, and largely static. Books can be sorted in a categorization scheme because the information they contain does not change over time; the categories remain as valid as when they were assigned. But what happens when the entirety of humanity's knowledge is now not only instantly available, but rapidly changing and evolving along with us? Is not categorization of changing information inherently flawed when information snapshots are almost immediately out of date? What can we do to allow everyone in this age of transformation to participate in information sharing in scalable way?

I would propose that when information itself has become so fluid it defies static categorization, that the transformation of knowledge is the new information we need to track. Already you see the signs of this:

  • Wikis are updated far more often than they have completely new entries added; the success of wiki is in transformation of knowledge over time, and in continuous evolution of the information contained therein.
  • Scientists in all fields build new ideas upon old; there are no new ideas that don't synthesize existing parts, transforming our understanding of those parts into a new entity.
  • Agile development emphasizes small, rapid, interactive transformations of data (a software algorithm + user interfaces on the large) rather than as big bang snapshots and releases.
  • Open source licenses reduce the barriers to accessing data effectively to zero explicitly to allow for rapid transformation of that data into today's entity. "Build from trunk" is becoming a normal recommendation for rapidly transforming projects, since snapshots are immediately out of date.

We'll talk about whether this all makes sense, whether moment-in-time snapshots of information are becoming more or less relevant than the changes over time to that information, and what can be done to both facilitate transformation and adapt our traditional information management ideals to this constantly changing information sea.

If you're interested in hearing more about all this, about how the session goes, or just would like to talk sometime along these lines, gimme a shout.

Sunday, June 10, 2007

JRuby 1.0 Released!

We have finally released JRuby 1.0, based on the last release candidate, RC3. And what more is there to say? Not really a whole lot...It's almost entirely RC3, with one or two minor fixes added in. But it's really turned out to be an outstanding release, and already reports are coming in of folks trying it out en masse. We're very happy.

So I'll do a little recap here. JRuby 1.0 was focused almost entirely on one goal: Ruby 1.8.x compatibility. To that end, we are now the only alternative Ruby implementation that can reasonably claim we're "compatible". It's no longer a question of whether we can run Ruby applications or not...we've proven that again and again. The issues people run into now are those requiring minor behavioral tweaks, minor parser tweaks, and occasionally exploration of some peculiar threading or memory concerns. It's been a long time coming, but the compatibility issue is largely answered.

Now we start looking toward the future. Once you have a working Ruby implementation, what next? I believe we've shown that the correct way to approach Ruby is to get it right first. The next step is making it run as well as possible. Ok, so we've cheated a bit along the way, introducing interpreter optimizations and a JIT compiler, but that's all fun stuff. The heavy lifting for performance and scalability are coming up, and there's a lot of low-hanging fruit we can start to tackle with 1.0 safely behind us. So action item #1 for the future of JRuby is plucking that low-hanging performance fruit.

The second item for any working implementation would have to be platform integration. Our platform is Java (and of course we've cheated a bit here by doing additional work to make Java integration really useful and usable), so integration involves making Ruby a better citizen of the Java platform. In this area, expect to see us further reduce the disconnect between Ruby and Java, allowing you to construct Ruby objects directly from Java code, define real Java classes using Ruby (classes you can then compile Java code to call), and minimizing to as large an extent possible the performance impact of calling from dynamically typed Ruby code into statically typed Java code. Java integration is action item #2.

And then what? Well, there are many options, all very attractive. I personally would like to see time spent implementing potential Ruby 1.9/2.0 features, to provide a second testbed where people can try those features out. I would like to find ways to share code with Rubinius (beyond tests), either by implementing just enough Rubinius logic to run its pure-Ruby core-class implementations or by implementing a full Rubinius kernel on the JVM. I would like to see JRuby expand to the rest of the Java world, bringing the Ruby Way to Java EE. But most importantly, I want to reach out to the Ruby community and find ways for them to be a part of the JRuby process.

Up to now, JRuby has really existed in its own, separate world. There was the Ruby community and also the JRuby community, and although members might identify themselves with both, there was always this perceived boundary.

We need to break that boundary down...not only for JRuby but for the other Ruby implementations as well.

Ruby is coming of age. Multiple implementations shows that Ruby has really matured as a language...and also shows it has a lot of maturing left to do. Ask Evan (of Rubinius) or me how we feel about retry behavior or block argument processing or thread event processing or SAFE levels and tainting and you'll start to understand some of the ugly, hidden bits of Ruby. We need to make sure that Ruby development proceeds as a whole...not necessarily as a single project or a single codebase, but as an open, direct exchange of concerns, complaints, solutions and ideas. We need to start treating the Rubinius community and the JRuby community and any other implementations' communities as part of the whole...different facets of the same gem.

If you have never tried an alternative implementation of Ruby: do it today. Pick out your favorite app, library, or framework and build your next app using something other than Matz's Ruby. Start running your continuous integration tests against JRuby trunk (or Rubinius trunk, if it runs your code ok; those guys sorely need more CI hits). Make sure your gem releases work on JRuby too, and if you have native (i.e. C) code, explore what it would take to do a JRuby port. Show that you welcome diversity into the Ruby world...that you recognize that diversity is an essential part of language evolution.

JRuby has always been a community project, and only by absorbing the JRuby community into the Ruby community will JRuby continue to be successful. If we can make that happen, I see wonderful things in Ruby's future...regardless of which implementation you use.

Monday, June 4, 2007

A Response to Ola's IronRuby Post

I'm not up for creative titles tonight. Hopefully you'll see this in your feed reader and click for a second opinion. Granted, I agree with Ola's IronRuby post on most points, but I disagree on a few key items. So let's dive in, shall we?

You managed to blog this viewpoint before me, Ola, but you know I agree with almost everything. However, to pour a bit more oil on the fire, I'll take it a step further.

Having run the Ruby gauntlet and brought JRuby from not running anything to running Rails almost 100% perfectly (in just over a year, I might add), I will confidently say there's no way with current specs and tests that anyone could create an implementation of Ruby from scratch that will run Rails unless they can look at the existing implementations. I simply do not believe it's possible.

And I'll throw another couple curve balls too:
  • I don't believe tests and specs will be where they need to be within the next 6-12 months unless there's a major effort put behind them. Even if that effort happens, I still don't think we'll ever have specs enough for someone to implement Ruby "in the dark" for at least a year, if it ever happens.
  • IronPython did a great job getting pretty close to 100% compatible. But Jim Hugunin had implemented an almost 100% Python-compatible implementation in Jython before going to Microsoft; he didn't need to look at the Python source. I don't believe John Lam has the same level of experience with Ruby, so he's at a severe disadvantage (and to John: I really feel for you, man...this has got to be difficult).
  • This is a good friend's belief, but he's won me over: we don't believe Microsoft would ever willingly allow IronRuby to get to the point of running Rails, since that would directly compete with their ASP.NET server, software, and tool offerings. What would be the benefit to them of a free runtime running a free language implementation that runs a free web framework? Probably zero. And as Martin Fowler and others have blogged about NUnit, Microsoft hasn't exactly been lovey-dovey with OSS projects that impact (or are perceived to impact) their bottom line.
  • Given these facts and the current situation, I'd say it's a better bet for us as a community to get behind the Gardens Point Ruby.NET Compiler project, which is already much farther along than IronRuby...plus it's real open source (you can contribute) and they can look at Ruby's source (and have admitted to doing so for at least the parser). I was wary/skeptical of Ruby.NET last year, but they now seem like the current best hope for Ruby on the CLR. That link again is Gardens Point Ruby.NET Compiler. And to the QUT guys working on Ruby.NET, you need to do three things right now to save your project from historical obscurity: set up a public source repository, accept a few external contributors, and start blogging and emailing up a freaking storm.
That about sums up what I have to say about IronRuby, Microsoft, and the future of Ruby on the CLR. The rest of Ola's points I agree with...diversity is important, a spec and test kit are needed immediately, and Microsoft needs to change their OSS attitude pretty quick or risk becoming irrelevant.

Sunday, June 3, 2007

JRuby 1.0.0RC3 Released - And This Is It!

Tom posted the announcements already, but JRuby 1.0.0RC3 is out in the wild! This release is our most important yet, because we intend for this release to become JRuby 1.0. The only things that will change from now until a 1.0 final release later this week would be any showstopping bugs that are extremely low-impact to fix. In general, RC3 should be nearly identical to 1.0 final.

So what does this mean for JRuby? We've taken the approach of calling the JRuby 1.0 release the "Ruby compatible" release. Basically, all known application bugs caused by JRuby incompatibilities with Matz's Ruby (MRI) have been resolved. This doesn't mean there aren't more compatibility bugs; there probably always will be, and we have probably 50-80 in the bug repository right now (and another 80 bugs or enhancements that are JRuby-specific). But it does mean that most applications should "just work" out of the box, and any application failures we knew of have been resolved. In general, the remaining post 1.0 bugs are edge cases and minor correctness issues, like obscure forms of core methods or missing error conditions.

We've tried very hard to make this both a solid release and a stepping stone to JRuby's future. With this release, we can confidently say that we're Ruby 1.8.5 compatible. Any additional fixes we need to make (like the known issues) are in the last 1% of Ruby features we can potentially support. We also recognize that this is a big moment for JRuby. People can start counting on JRuby to run their Ruby applications correctly. Of course many in the JRuby community have already been doing this for many months, but the 1.0 moniker says to the world we feel like we're ready for prime time.

So what comes after 1.0? That breaks down into a few areas:
  1. Performance. JRuby's performance has come a long way in the past year. We've increased speed by at least an order of magnitude, and have enabled the JIT compiler for the 1.0 release. This means that for many cases where JRuby can compile Ruby code, it will perform faster than Ruby 1.8.5. The general case is still a little muddy, though. We've had anecdotal reports that JRuby on Rails in a Java application server performance extremely well, and other reports that general Ruby applications perform somewhat poorly. The general performance situation is not well-understood, but we all agree there's a lot more work to be done. And the good news is that we have a big list of optimizations remaining to continue improving JRuby's specific and general performance.
  2. Java Integration. JRuby does an excellent job of fitting into the Java platform. In almost all cases, you can call libraries, implement interfaces, and extend classes with no difficulty. But there are edge cases--usually nonstandard or antipattern Java--that JRuby doesn't behave as nicely. The code to enable calling Java is also far more complicated than we'd like. To solve both issues, we'll be taking the existing Java integration syntax and API and backing it with a redesigned library. Expect to see something of this work in a 1.1 release later this year. Our goal is to achieve excellent integration with Java, on par with the most tightly-integrated JVM languages available today. And we're not far off.
  3. Ruby 2.0 and Rubinius. We intend to start supporting Ruby 2.0 and Rubinius bytecode execution soon, though the Ruby 2.0 work is much farther along. We intend to catch up the Rubinius work as well as to try adopting some of Rubinius's pure Ruby implementations of core class functionality. We also plan to enable the use of Ruby 2.0 features through configuration and command-line switches to JRuby. Specify something like -J-Djruby.string.version=2 and all strings in the system will be treated as Ruby 2.0 strings, with full character semantics. Or optionally turn on experimental Ruby 2.0 syntax for lambdas and named parameters. Basically we want to bring JRuby up to the bleeding edge of Ruby features, to provide another platform where people can get a feel for what Ruby 2.0 might look like. More exposure for these features will help Matz and Co. decide the best way to move forward. And that's good for everyone.
And of course we'll continue fixing compatibility bugs, Java integration bugs, and expanding JRuby's reach on the Java platforms with more DSLs and API wrappers around all your favorite Java APIs.

But there's one more area I should talk about: how you can get involved.

JRuby is a community project. The core committers play traffic cops as much as developers, routing patches, examining bugs, documenting features. The only way a project like JRuby succeeds is through mass community involvement. We've been extremely lucky to have a constant flow of interested developers into our community, even when OSS community attrition has been very high. The community may look completely different from month to month, but the flow of patches and bugs has steadily increased.

For folks just getting started, I've written a few articles on the JRuby wiki that should help you understand the development process. They're linked from the front page under "Getting Involved".

For easy bugs, I'd recommend looking in the JRuby JIRA for bugs with "rubinius" or bugs reported by Daniel Berger. The Rubinius bugs are almost all problems we've had running Rubinius's excellent specs, and usually represent minor incompatibilities with C Ruby. Daniel's tests are a lot of those edge cases I mentioned...he's running through his own test suite and reporting any incorrect behavior that's come up. Another option would be to just pick your favorite Ruby app and start running its test cases with JRuby. You're sure to find something interesting to report, and it may be an easy fix.

For non-Java-coders (or folks afraid of hacking on JRuby), stop by the RubySpec Wiki and update or author an article. RubySpec is an effort to build a community-driven specification of Ruby that all users and implementers can freely reference. It is linked from the RubyDoc site and is fast becoming a standard way for the community to record language and library behaviors. I believe this is the best and fastest way for us to form a complete specification of Ruby's behavior...and I believe such a specification is becoming extremely important, what with there now being 5-10 different implementations of Ruby, all guessing at what "correct" is. Have you written RubySpec today?

Well that's about it for this release report. If all goes as well as I think it will, JRuby 1.0 final should be released some time this week. Give it a try, I think you'll be pleasantly surprised!

Friday, June 1, 2007

Creating a Field-Initializing 'new' Method

One thing often touted as a missing feature in Ruby is the lack of a constructor form that initializes fields. A few other languages have this feature, including for example Groovy, another JVM dynamic language. The general idea is that if you want to construct an object and initialize a number of fields, you often want to do it in one shot. Rather than modify the class to have additional initializers for all the fields you want to set, there's another option.

Because Ruby is so cool, you can add this feature yourself to all classes at the same time.

class Class
def new!(*args, &block)
# make sure we have arguments
if args && args.size > 0
# if it's not a Hash, perform a normal "new"
return new(*args, &block) unless Hash === args[-1]

# grab the last arg in the list
last_arg = args.pop

# make sure all fields actually exist
last_arg.each_key {|key|
unless public_instance_methods.include?("#{key}=") do
raise ArgumentError.new(
"No attr setter for name: #{key}")
end
}

# create the object and set its fields
new_obj = new(*args, &block)
last_arg.each {|key, value|
new_obj.send "#{key}=", value
}
else
# no args, just do a normal "new" with any block passed
new_obj = new(&block)
end
new_obj
end
end

So with such a simple piece of code, we now have a new! method on all classes that accepts a final parameter--a hash of field names and values--that can be given using Ruby's named-parameter-like syntax. Given a simple class, like the following:

class MyObject
attr_accessor :foo
attr_accessor :bar

def initialize(msg)
puts msg
end
end

No additional work is needed to use our new! method:

x = MyObject.new!("yippee",
:foo => "hello", :bar => "goodbye")
=> "yippee"
p [x.foo, x.bar]
=> ["hello", "goodbye"]
y = MyObject.new!("blah", :yuck => "baz")
=> error: "No attr setter for name: yuck"

The reason this works is that all classes are instances of the Class class. So the MyObject class definition above is roughly equivalent to saying:

MyObject = Class.new {
# class def logic here
}

This means that instances of Class, like MyObject, inherit methods defined on Class, like new!. Since all classes in the system are Class objects, all classes instantly gain a new! method.

This is a perfect example of why Ruby is such a powerful language, and why it's so easy in Ruby to use the coolest metaprogramming tricks. And it's a primary reason why frameworks like Rails have been able to do such amazing things. With a language that's this powerful and this easy, you can imagine what else is possible.

Are we having fun yet?