Friday, October 12, 2007

Performance Update

As some of you may know, I've been busily migrating all method binding to use Java annotations. The main reasons for this are to simplify binding and to provide end-to-end metadata that can be used for optimizing methods. It has enabled using a single binding generator for 90% of methods in the system (and increasing). And today that has enabled making some impressive perf improvements.

The method binding ends up looking like this:
@JRubyMethod(name = "[]", name2 = "slice",
required = 1, optional = 1)
public IRubyObject aref(IRubyObject[] args) {
...

This binds the aref Java method to the two Ruby method names [] and slice and enforces a minimum of one argument and a maximum of two. And it does this all automatically; no manual arity checking or method binding is necessary. Neat. But that's not the coolest result of the migration.

The first big step I took today was migrating all annotation-based binding to directly generate unique DynamicMethod subclasses rather than unique Callback subclasses that would then be wrapped in a generic DynamicMethod implementation. This moves generated code closer to the actual calls.

The second step was to completely disable STI dispatch. STI, we shall miss you.

So, benchmarks. Of course fibonacci numbers are indicative of only a very narrow range of performance, but I think they're a good indicator of where general performance will go in the future, as we're able to expand these optimizations to a wider range of methods.

JRuby before the changes:
$ jruby -J-server -O bench_fib_recursive.rb
1.039000 0.000000 1.039000 ( 1.039000)
1.182000 0.000000 1.182000 ( 1.182000)
1.201000 0.000000 1.201000 ( 1.201000)
1.197000 0.000000 1.197000 ( 1.197000)
1.208000 0.000000 1.208000 ( 1.208000)
1.202000 0.000000 1.202000 ( 1.202000)
1.187000 0.000000 1.187000 ( 1.187000)
1.188000 0.000000 1.188000 ( 1.188000)

JRuby after:
$ jruby -J-server -O bench_fib_recursive.rb
0.864000 0.000000 0.864000 ( 0.863000)
0.640000 0.000000 0.640000 ( 0.640000)
0.637000 0.000000 0.637000 ( 0.637000)
0.637000 0.000000 0.637000 ( 0.637000)
0.642000 0.000000 0.642000 ( 0.642000)
0.643000 0.000000 0.643000 ( 0.643000)
0.652000 0.000000 0.652000 ( 0.652000)
0.637000 0.000000 0.637000 ( 0.637000)

This is probably the largest performance boost since the early days of the compiler, and it's by far the fastest fib has ever run. Here's MRI (Ruby 1.8) and YARV (Ruby 1.9) numbers for comparison:

MRI:
$ ruby bench_fib_recursive.rb
1.760000 0.010000 1.770000 ( 1.813867)
1.750000 0.010000 1.760000 ( 1.827066)
1.760000 0.000000 1.760000 ( 1.796172)
1.760000 0.010000 1.770000 ( 1.822739)
1.740000 0.000000 1.740000 ( 1.800645)
1.750000 0.010000 1.760000 ( 1.751270)
1.750000 0.000000 1.750000 ( 1.778388)
1.740000 0.000000 1.740000 ( 1.755024)

And YARV:
$ ./ruby -I lib bench_fib_recursive.rb
0.390000 0.000000 0.390000 ( 0.398399)
0.390000 0.000000 0.390000 ( 0.412120)
0.400000 0.010000 0.410000 ( 0.424013)
0.400000 0.000000 0.400000 ( 0.415217)
0.400000 0.000000 0.400000 ( 0.409039)
0.390000 0.000000 0.390000 ( 0.415853)
0.400000 0.000000 0.400000 ( 0.415201)
0.400000 0.000000 0.400000 ( 0.504051)

What I think is really awesome is that I'm comfortable showing YARV's numbers, since we're getting so close--and YARV has a bunch of additional integer math optimizations we don't currently support and thought we'd never be able to compete with. Well, I guess we can.

However a more reasonable benchmark is the "pentomino" benchmark in the YARV suite. We've always been slower than MRI...much slower some time ago when nothing compiled. But times they are a-changin'. Here's JRuby before the changes:
$ time jruby -J-server -O sbench/bm_app_pentomino.rb

real 1m50.463s
user 1m49.990s
sys 0m1.131s

And after:
$ time jruby -J-server -O bench/bm_app_pentomino.rb

real 1m25.906s
user 1m26.393s
sys 0m0.946s

MRI:
$ time ruby test/bench/yarv/bm_app_pentomino.rb

real 1m47.635s
user 1m47.287s
sys 0m0.138s

And YARV:
$ time ./ruby -I lib bench/bm_app_pentomino.rb

real 0m49.733s
user 0m49.543s
sys 0m0.104s

Again, keep in mind that YARV is optimized around these benchmarks, so it's not surprising it would still be faster. But with these recent changes--general-purpose changes that are not targeted at any specific benchmark--we're now less than 2x slower.

My confidence has been wholly restored.

9 comments:

  1. What's the bottleneck with JRuby on Rails in development mode. With today's trunk build of JRuby, production mode seems to be reasonably snappy, but development mode seems to lag far behind MRI, to the point where it's painful to use. How far are we from catching up in this aspect. Obviously reaching par with MRI in this regard would be much beneficial as it would greatly enhance development of Rails appplications on JRuby.

    ReplyDelete
  2. Anonymous: see my recent post on the matter at http://ola-bini.blogspot.com. There are many numbers, but the Rails performance is awful. We don't know the reason yet, but we're working on it.

    ReplyDelete
  3. Brad C: That would be great, wouldn't it? Unfortunately annotations don't allow arrays. The only values you can put into an annotation must be constant literal values, classes, or enums. I'd love to have a better option for binding multiple names, but this will have to do for now.

    ReplyDelete
  4. Actually, they do allow arrays: it's really cool! Check out the definition of the @Target annotation (the meta-annotation used whenever you're creating your own annotation). It takes an ElementType[] and you can use @Target with one parameter:

    @Target(ElementType.FIELD)

    or multiple parameters:

    @Target({ ElementType.FIELD, ElementType.METHOD })

    So I gather your @JRubyMethod definition would look something like:

    @Target(ElementType.METHOD)
    @Retention(RetentionPolicy.RUNTIME)
    public @interface JRubyMethod {
    String[] name();
    int required();
    int optional();
    }
    plus any default values that you have set.

    Neat stuff!

    ReplyDelete
  5. anonymous: Can you report that as a bug? I can't reproduce it here on OS X.

    Brad C: We'll I'll be darned. Looks like I'll be migrating the annotations to arrays!

    ReplyDelete
  6. glad to be of some small help :-)

    ReplyDelete
  7. We didn't see you at Minnedemo last night-- were you there?

    ReplyDelete
  8. This is so great Charles, congratulations!

    ReplyDelete
  9. Is there a graph somewhere showing the performance numbers over time? I would love to see that STEEP climb!

    ReplyDelete