Sunday, June 1, 2008


Of course anyone who reads my blog expected I'd have something to say about Maglev once it was made public. I've previously performed what I thought was a fair analysis of the various Ruby implementations, and Maglev was mostly a sidebar. With their coming out at RailsConf, they're now fair game for some level of analysis.

Avi Bryant and Bob Walker talked about Maglev, a new Ruby VM based on Gemstone's Smalltalk VM, at RailsConf this weekend. And there's been an explosion of coverage about it.

First off, they demonstrated its distributed object database automatically synchronizing globally-reachable state across multiple VMs. It's an amazing new idea that the world has never really seen...

except that it isn't. This is based on existing OODB technology that Gemstone and others have been promoting for better than a decade. It's cool stuff, no doubt, but it's been available in Gemstone's Smalltalk product and in their Java product for years, and hasn't seen widespread adoption. Maybe it's on the rise, I really don't know. It's certainly cool, but it's certainly not new.

The duo eventually moved on to show off some performance numbers. And please pardon me if I don't have these numbers exactly right. They showed our old friend fib running something like 15x faster. Method dispatch something like 30x faster. While loops 100x faster. Amazing results.

Except that these are results reported entirely in a vacuum. Whether this is fib following the "rules" of Ruby is entirely an open question. Whether this is method dispatch adhering to Ruby's call logic is entirely an open question. Whether this is a while loop using all method calls for its condition and increment steps is an open quesetion. Because the Maglev guys haven't started running Ruby tests yet. Is it Ruby?

I don't want to come off as too defensive here, and I don't want to appear as though I'm taking shots at another implementation. I've certainly launched my share of controversial commentary at Rubinius and IronRuby over the past few months, and while some of it may perhaps have slipped over the edge of polite commentary, I always thought I was being at least honest.

But there's an entirely new situation with Maglev. Maglev has begun to publish glowing performance numbers well in advance of actually running anything at all. They haven't started running the RubySpecs and have no compatibility story today. You can't actually get Maglev yet and run anything on it. It's worse than Vaporware, it's Presentationware. Go to Gemstone's site and download Maglev (you can't). Pull the source (you can't). Build it yourself and investigate what it does (you can't). You start to understand what I mean. And this is what the "Ruby media" is calling the most disruptive new Ruby technology. Dudes, come on. Were you born yesterday?

It's time for a confession. I've been too hard on IronRuby and Rubinius. Both teams are working really hard on their respective implementations, and both teams have really tried to stay true to Ruby ideals in everything they do. Guess what...IronRuby runs Rails. Rubinius runs Rails. And if they're not production ready now, they will be soon. And that's a good thing for Ruby. Sure, I still believe both teams may have made unreasonable claims about what they'd be able to accomplish in a given period of time, but we've all made those claims. If they haven't delivered on all milestones, they've delivered on most of the important ones. And it's those milestones I think deserve some credit now.

My sin is pride. I'm proud of what we've accomplished with JRuby. And when new implementers come along saying they're going to do it in half the time, I feel like it belittles the effort we've put in. IronRuby has done it. Rubinius has done it. And while I've occasionally lashed out at them as a result, I've always been right there trying to help them...answering questions, contributing specs, suggesting strategies and even committing code. In the end it's the cockiness...the attitude...the belief that "I know better than you do" that irritates me, and I'm too sensitive to it. Color me human. But it's time for me and others to understand another side of IronRuby and Rubinius in light of this new contender.

Rubinius and IronRuby teams have always considered compatibility as the primary goal. If you can't run Ruby apps, you're not Ruby, right? And so every step of the way, as they published performance results AND compatibility metrics, they've always been honest about the future.

IronRuby has managed to get great performance on several benchmarks by leveraging the DLR and the excellent language implementation folks on the DLR and IronPython teams at Microsoft. So if nothing else, they've proven many of the "fast-bootstrapping" claims they've made about the DLR. And they've always been balanced in reporting results...John Lam has shown a couple slow benchmarks along with fast benchmarks at every talk, not to mention showing spec results with pass/fail rates clearly spelled out. That honesty has not gone unnoticed, and it shows a realism and humility that will ensure IronRuby's future; a realism that will ensure Ruby users who really want or need a .NET implementation will receive an excellent one.

Rubinius has taken an entirely new approach to implementing Ruby by attempting to write as much as possible in Ruby itself. Maybe they have a lot of C/C++ code right now, but it's not that big a deal...and I was perhaps too pedantic to focus on this ratio in previous posts. What's important is that Rubinius has always tried to be an entirely open, community-driven project. Their successes and failures are immediately accessible to anyone who wants to pull the source; and anyone who wants to pull the source can probably become a Rubinius contributor within a short amount of time. They've had performance ups and downs, but again they've been honest about both the good and the bad. And like IronRuby, if they haven't trumpeted the bad side of things, it's because they're already proving that the Ruby-in-Ruby approach absolutely can work. The bad side will lessen over time until it completely disappears.

Then there's Maglev. Like the other impls, I'm excited that there's a new possibility for Ruby to succeed. A high performance, "scalable" Ruby implementation is certainly what this community needs. But unlike most of the other implementations, it seems like Maglev is pushing performance numbers without compatibility metrics; marketing before reality. Am I far off here?

Let's take a step back. Maglev will probably be amazing. It will probably be fast, maybe on some order approaching the numbers they've reported. Maybe this will happen some day along with support for existing Ruby code. And hell, maybe I'll use it too...I want to be able to write applications in Ruby and have insane performance so I can just write code the way I want to write code. So do you.

But we're talking theory here. So let's do an experiment using JRuby briefly.

Maglev published fib numbers as being around 15x MRI performance. That's very impressive. So let's check MRI perf on my machine (keeping in mind, as I've stated previously, that fib is far from indicative of any real-world performance):

Ruby 1.8.6, fib(34), best of 10: 6.56s

Now let's try stock JRuby, with full compatibility:

JRuby 1.1.2, fib(34), best of 10: 1.735s (3.8x faster)

Not bad, but certainly not up to Maglev speeds, right? Well...perhaps. JRuby, like IronRuby and Rubinius, has always focused first on compatibility. This means we're bending over backwards to make normal Ruby code run. So in many cases, we're doing more work than we need to, because compatibility has always been the primary goal. IronRuby and Rubinius will report the same process. Make it work, then make it fast. And both IronRuby and Rubinius are now starting to run Rails, so I think we've proven at least three times that this is the right approach.

But let's say we could tweak JRuby to run with some "future" optimizations, optimizations that might not be quite "Ruby" but which would still successfully run these benchmarks.

First, we'll turn off first-class frame object allocation/initialization, since it's not needed here:

JRuby 1.1.2, fib(34), no frames: 1.273s (5.15x faster than MRI)

Now we'll turn off thread checkpointing needed to implement operations like Thread#kill and Thread#raise, as well as turning off artificial line-position updates:

JRuby 1.1.2, fib(34), -frames, -checkpoints, -positions: 1.25s (5.24x faster)

Now we'll add in some fast integer operations like Ruby 1.9 includes, where Fixnum#+, -, etc are specially-handled by the compiler. And we'll simultaneously omit some last framing overhead that's still around to handle backtrace information:

JRuby 1.1.2, fib(34), "fastest" mode: 0.984s (6.67x faster)

So just by tweaking a few things we've gained another 3x performance over MRI. Are we having fun yet? Should we extrapolate to optimizations X, Y, Z that bring JRuby performance another half-dozen times faster than MRI? If we can run the benchmarks, it shouldn't matter that we can't run Ruby code, right?

The truth is that not all of these optimizations are kosher right now. Removing the ability to override Fixnum#+ certainly makes it easier to optimize addition, but it's not in the spirit of Ruby. Removing frames may be legal in some cases (like this one) but it's not legal in all cases. And of course I've blogged about how Thread#kill and Thread#raise are broken, but we have to support them anyway. On and on we can go through lots of optimizations you might make in the first 100 days of your implementation, only to back out later when you realize you're actually breaking features people depend on.

This all adds up to a very different picture of Ruby implementation. Rather than wishing for a rose-colored world where anyone with a new VM can swoop in and post magic performance numbers, perhaps we as Ruby community members should be focusing on whether this is going to help us actually run today's apps any better; whether these results are repeatable in ways that actually help us get shit done. Perhaps we should be focusing on the compatibility story over bleeding-edge early performance numbers; focusing on tangible steps toward the future rather than the "furs and gold rings" that David warned about in his keynote. Maybe we should think more about the effect that broadcasting vaporware performance numbers will have on the community, rather than rushing to be the first to republish the latest numbers on the latest slides. Maybe it's worth taking all this microbenchmark nonsense with a grain of salt and trying it out ourselves (if, of course, that's even possible) before serving as the mouthpiece for others' commercial ventures.

Am I wrong? Am I being unfair? Am I taking an unreasonable shot at Maglev?


  1. Wasn't there... mostly agree with the Doc, but can see the frustration Charles has. Personally it's just a little buzz to spice things up, when Monday rolls around we'll all be going back to our Ruby impls with no sign of MagLev.

    At any rate, it's been MONEY WELL SPENT for Gemstone!!! I had never heard of them until MagLev!

    One more thing: isn't it funny how a community that says things like "it's fast enough" would be soo interested in an implementation focused on performance? :)

  2. I want to point out that at RailsConf 2007... Avi got on stage and talked about how the Smalltalk VM was what Ruby in the future would look like...

    And he challenged everyone to plop Ruby on top of the Smalltalk VM... And now he says he's done it (or doing it).

    Avi is a smart man... Charles, you're very intelligent as well... Why not give the guy the benefit of the doubt.

    I like your honesty... it makes me feel better about what you and Sun are trying to do. Keep it up.

  3. Ruby on the JVM is appealing because of what that opens up in a deployment scenario that needs other "on-JVM" resources. The same is true for Ruby on the DLR: easy interoperability with other resources in the same environment. As a developer comfortable with Ruby, I love that Ruby can be a "glue" language because the need for something like that comes up often. The work that you and Tom and all the other JRuby developers do is so important to the big picture for the language. If the work that Gemstone is doing with MagLev materializes into a tool that has appeal for certain deployment scenarios, then all the better for software developers like me who need powerful tools and deployment options to provide value to our own customers.

    The world is vast, and the world of problems that can be solved with software needs lots of options.

  4. No attempt to cheap-shot on Avi, but ... isn't he a smalltalk guy that dislikes Ruby?

    Personally I rather look at Rubinius community than to rely on anyone that spends more time in C and Smalltalk than in Ruby.

  5. What I don't like very much with MagLev (I was not at RailsConf, but I visited their web site), is the fact that they say "We’re working on a new Ruby VM which is built for scale and performance from the ground up, with some features we guarantee will blow your mind. Come see how far we’ve gotten", and they show absolutely NOTHING on their website, except "leave your email address". It sounds only marketing to me. I don't like to see people who say "I've done this, it's awesome", and not show anything. Come to this, I also could prepare a few slides and say that I've designed the ultimate Ruby VM. But of course I have done nothing. As long as they continue to show nothing to us, I would consider MagLev as having done nothing.

  6. Excellent post, Charles. It really disturbs me how many Rubyists are just wowed by the glitz and don't stop to think about the deeper issues at play here. Do you really want to get stuck another vendor lock-in situation?

    Even if you were to go with the Microsoft solution (who have in the past been the vendor-lock-in king) you still have an option to deploy on free software with the Mono stack. But with Gemstone, you're out of luck. And we know how well the proprietary implementation route worked out for Smalltalk in the long run. =\

    It's really time for Rubyists to consider what they stand for as hackers and what they'll put up with as a community. Are you really so willing to give up the ability to dive into the source of your implementation to fix bugs or add features? Many of us have been there, and it's so antithetical to what being a hacker is all about... This whole situation makes me very nervous. I don't want to have to choose between performance and freedom, but if it comes down to it I know which one I'll pick.

  7. Maybe this is my Rubinius bias showing, but I think this is the best post about implementations you have made.
    Well done, and thank for the praise. I'll try to be less hard on JRuby in the future myself.
    I guess you guys aren't killer robots from the future. :)

  8. Did JRuby run Rails after 3 months?

    Did Rubinius?

    Did IronRuby?

    Sheesh, man, I'd almost guarantee that you guys were in the same shape after 3 months. To be honest, when you have to reassure your readers that you're not being defensive, you might want to go back and make sure that you have the right perspective on the issue. I'm glad JRuby is around along with all the others, but there's plenty of room for other VM's (even those that might be better than yours).

  9. I think the tone of this post speaks for itself.

    You have a history of writing posts that are generally negative towards any Ruby implementation other than JRuby. So did you come off as defensive? Yes, very much so.

    It's almost like you need to have a bad guy, for reasons that escape me. No one ever seems to measure up to the JRuby greatness. First it was time to rip on Rubinius and call for the "Ruby in Ruby" meme to die. Now that we have a new bad guy, Maglev, suddenly Rubinius is making great strides, so what if they're using a little C/C++, etc.

    It sounds to me like you're worried that Maglev, when it's finished, might turn out to be such a phenomenal implementation of Ruby that no one will have a need for JRuby. Or maybe you're just jealous that Maglev can already do things JRuby can't. Maybe you can't believe that anyone doing honest work could make the kind of progress they've made in 3 months. So, whatever they have must be "Presentationware", because JRuby wasn't this far along in 3 months. If they've made this kind of progress, and we didn't, that might mean that *gasp* they're smarter than us! I'm grasping at straws here because I honestly can't figure out why your posts about Ruby implementations other than JRuby need to be so negative.

    What's wrong with focusing on the positive? Maglev is ultimately good for the Ruby community. In the words of DHH, wipe the wah-wah tears off your chin.

  10. Anonymous: He never denied that MagLev is positive for Ruby. He simply said that there's going to be a lot of hype around it, and that we shouldn't necessarily believe the benchmarks shown.

    I don't care if they have worked on it for 3 days, you shouldn't claim such performance improvements if your implementation doesn't actually run Ruby code.

  11. I walked in toward the end of the Maglev presentation. However, based on what I heard, I walked out of that talk somewhat.... unsettled.

    I believe that my sense was a result of the relative haughtiness that Bob Walker portrayed when speaking on if/when Gemstone would open source Maglev ( to paraphrase from memory, "in a world full of possibilities, it's a possibility" and "software is an ongoing process so we don't know yet" respectively).

    Now knowing that they're not running Ruby specs, I generally agree with you, headius. Also considering that there is no concrete information out there about Maglev makes their abrupt presentation of performance statistics spurious.

    Why did Gemstone jump the gun and present Maglev at RailsConf? I don't get it. It seems premature.

  12. anonymous -- no, Avi is an old Ruby hacker who grudgingly went to SmallTalk. He's a polyglot and a good guy. He might not be a perfect ambassador to bring to RailsConf, but he should get more credit than you appear to give him.

  13. I take Charles's primary point to be about Science! (capital S) He's not trying to tear down MagLev, particularly he's trying to build up Openness and Transparency. And his primary point is that Presentationware is the antithesis of Openness and Transparency, because opens not just room for differences of opinion, but complete and utter lack of certainty regarding how the presentationware relates to open baselines. In order to compete fairly, they should be playing by the same rules used by everyone else (Publishing how they're doing in the rubyspecs would be a great start).

  14. I may be missing something, but couldn't you do everything Gemstone does with Terracotta and JRuby?

    A quick google shows a lot of people already have this working.

  15. Since you have no idea how Maglev works, isn't this wild speculation? Is it really impossible to have such good performance & run standard Ruby code? I've seen no evidence that Maglev doesn't run standard Ruby code. Time will tell.

    is it possible that you're lashing out at this project because you couldn't make JRuby run standard Ruby code and have similar performance? That would fit with your anecdotes about bashing other implementations.

  16. After reading Avi and Charles' posts... well...

    I agree with Charles about the open and visible comments.

    I agree that Avi is a wicked smart guy and good at pulling rabbits out of hats.

    I agree that building a compatible Ruby is harder than it looks. I know Avi has done a fair amount of Ruby, but I'm not sure he has the why/Charles level of intimacy with the Ruby runtime.

    The object paging model that Avi describes is like what Terracotta does on the JVM... so creating a similar demo for JRuby would take a half a day... it's not magic.

    Plus, I'm not keen on the Lisp/Smalltalk "we're not dead yet" vendors who have proprietary stuff. Yeah, people are in business to make money, but there are plenty of folks who have demonstrated how you can make money in open source-land.

    So... Charles, you are one of the best diplomats I've met in this industry... you might occasionally lash out, but in my opinion, you're entitled given your amazing contribution and your general calmness.

    Rock on!



  17. > Did JRuby run Rails after 3 months?
    > Did Rubinius?
    > Did IronRuby?

    @mrneighborly: The huge difference here is that we've always been able to see for ourselves exactly how compliant every other Ruby implementation ever begun has run.

    If Sun or MS had come out saying, "Hey, we've got this awesome fast ruby impl." but didn't show us the source, you can bet they'd face a lot more hostility than Maglev has brought on themselves.

  18. If you need an open source, battle-tested distributed object database in a Ruby-like language today, you could consider using the ZODB in Python. It's been around for around 10 years too.

  19. Although this was a somewhat fair analysis of MagLev, it's missing the point. The shared global object space was the paradigm shifting part of the presentation, not the performance benchmarks. The real buzz these days seems to be around cloud computing, object databases and map reduce. Although MagLev/Gemstone is a really cool implementation of accessing an underlying object data store, it is really just one of the first implementations of what will probably become a common computing paradigm in the years to come.

  20. Just a thought, but it seems to me that many of those optimizations can be done by a compiler with sufficient information about the expression context. Perhaps MagLev actually performs such optimizations, and so constructing all the extra info is simply optimized away when it's not actually used.

  21. I would expect a Smalltalk implementation to beat the others. Smalltalk companies have years of optimizing a dynamic language very similar to Ruby. You do amazing work, but I would expect at some point they will have better performance than the current JRuby on the current JVM.

  22. I'm not interested as much in te performance, for me even MRI is _fast_enough_, but in the replicated OODB. So far none of the big players have take an interest in Ruby really so it would be really nice if that were really true. And no, I don't mind if it is closed source but it works as long as it is affordable. I'm willing to give them some time as long and hope it really works. I won't hold my breath but will (im)patiently wait for it.

  23. The "dynamic-optimized VM" is not a silver bullet.

    "Smalltalk/Ruby can be faster on a native VM because the VM pritimives align better with the operations that a dynamically typed object language needs."

    I don't know but Ruby/DLR isn't 15 faster than JRuby.

    And with the Smalltalk VM 5x slower than the Java VM, it would need to be 45x faster than Java for Ruby to be 15x faster than JRuby. And Ruby on .NET surely isn't 45x faster.


  24. Hey Charlie -- I'm even more skeptical than you are! As the corollary to Murphy's Law goes, "Nature always sides with the hidden flaw." I see lots of hidden flaws in Maglev.

  25. Last time I looked at JRuby, it was a "paraphrase" of the Matz VM but in Java, not C. You won't get the full potential of the speed of the JVM until you compile Ruby down to Java bytecodes. But running that way would lose some of the dynamic capability of Ruby. If they are compiling Ruby to Gemstone Smalltalk bytecodes, then they will have the full potential of that VM. And since Smalltalk is very dynamic, they will retain the full dynamic capabilities of the language.

  26. You aren't being unfair at all- you and Sho are both 100% correct. People like wearing rose-colored glasses, and for some reason I think the rails community is more guilty of that than most others.

    For now it's just marketing and hype, in the end they may well come up with something great (I hope they do) but, for now, it's just presentationware. For people to be fawning over it and freaking out at this stage is silly.

  27. What does the DLR have to do with anything? It's pretty much just a runtime code gen library. The .NET CLR has many of the same limitations as the JVM here, where a hosted language is limited to using memory-safe constructs which the host VM understands. This is too restrictive to obtain optimal performance for a language too dissimilar to the host language, as it requires embedding primitives as host language constructs, forcing superfluous allocations and indirections. A well-designed native VM will always perform better than a well-designed hosted VM, unless the hosted VM is assembly (ala proof-carrying code).

    The only "advantage" the CLR may have is its support for value types, which can reduce GC pressure if you can somehow take advantage of the stack allocation.

  28. To understand the kind of speed that Gemstone can achieve one has to imagine a JVM optimized to run Ruby instead of running Java.

    This is different from a Ruby implementation running on top of a JVM optimized to run Java.

    The later case has some impedance mismatch that hurts speed.

    Look backward at the progress of the JVM in terms of speed, including JIT and other recent advanced optimizations. They are impressive. Gemstone are almost at the same level. But for Ruby code instead of for Java code.

    Now that the JVM source code is open source, I wonder why the JRuby team does not look at it and see how to optimize things for Ruby? Or do they?

    That there is more distance between Java and Ruby than between Smalltalk and Ruby is a clear sign that Gemstone claims are probably valid.

    I wonder why this did not happened years before, it was so obvious to me that Ruby had all that Smalltalk was missing to become a commercial success...

  29. I think "presentationware" is a bit harsh. Avi and gang seem to be doing pretty well, and we have also had some discussions on #rubinius that indicate they have some degree of real infrastructure backing the fib.

    I am personally concerned about the non-openness aspect of it, partial as it may be. I would certainly expect a substantial amount of giving back to the Ruby world or risking a blowback.

    The one bit most interesting to me is what you seem to dismiss as an OODB. While certainly true that it has not made much of an impact thus far, you must also remember that Ruby and Rails have popularised, re-popularised or simply dusted off technologies that were not being adopted on their merits before. So perhaps this will be another one of those generational changes where the already existing technology meets the critical mass of developers ready to embrace it.

  30. For god's sake, stop whining!

  31. @anonymous re: 'unlimited object space' - that's exactly what terracotta does, except it's open source and has been going for years.

  32. I recall seeing benchmarks for rubinius and jruby before they were able to run rails. So a little of this may be the pot calling the kettle black? I do ahgree that benchmarks at this stage are irrelevant for maglev, and would be more meaningful if they could show rails running. so i throw out all the sillyness with the performance claims. My takeaway is the seamles sintegration of distributed cache and object store. I see maglev as upping the ante for rails VM's. Afterall if jruby or rubinius had a built in distributed object store, would the maglev announcement have made any splash whatsoever?

  33. I had a chance to exchange some emails with Charles. He intrigued me. Why? Because he was calm, patient and humble. I was asking pretty basic questions and the guy who did JRuby from scratch was answering me like he was really interested in them.

    Some suggestions for Charles: It is not a sin to be pride. You should not be humble. There is a very big difference between being humble and being arrogant. I believe you can be not-humble and not-arrogant at the same time.

    JRuby is faster than Python, Jython, Rubinius, etc. It only loses for C, Java and PyPy/Psyco.

    So you should be very proud of what you did. Let the competition come, the talk, the criticism, etc. This is like gravity, you cannot fight with this. Use it in your favor.

    Watch the movies below and sleep well: