But not by much. Depending on your definition of "on par", I'd say we're safely in the range of being on par with Ruby 1.8.6 performance, at least for these too-short, too-small measurements.
Alioth benchmarks are regularly panned by language enthusiasts...or at least enthusiasts on the "slow" end of the benchmarks. In the case of JVM-based language implementations, the problem is the same old excuse: not enough time for the JVM JIT to work its magic. This case isn't all that different, but it's fair to say that Alioth isn't testing straight-line performance in a hot application, but getting from point A to point B without a running start. And in that domain, it provides a reasonable window into implementation performance.
So then, the JRuby 1.0 numbers. You can click the link to see them, but they break down about like this:
Performance:
- Four tests are equal or faster in JRuby--usually not more than 2x faster, but "4.8x" faster in one case. That fastest one involves "concurrency", and I haven't studied it enough to know whether it's meaningful or not.
- Eight tests are less than 2x slower in JRuby.
- The remaining four tests are greater than 2x slower in JRuby.
- Startup time is considerably worse; it's listed as 305x slower for JRuby (!!!) but it's not a particularly useful ratio. We take a couple seconds to get going compared to Ruby's hundredths. That's life.
- All except one worse in JRuby
- Most more than 2x worse in JRuby
- Several more then 10x worse in JRuby
- Does anyone care?
So I guess that's all I've got to point out. We've been working on performance since 1.0, and there's a number of major improvements planned for the 1.1 release. And considering that "beating Ruby performance" wasn't a primary goal for JRuby 1.0, I think our "roughly on par" numbers here are pretty damn good. Granted, Ruby 1.8.x isn't the fastest implementation in the world, but we're pretty happy to have improved performance by an order of magnitude in the past year.
Now, onward to the future!
The concurrency benchmark is complete bullshit. All of the fastest programs are actually just running in one thread.
ReplyDeleteanonymous: You're right about the one-thread thing. I just had a look at the concurrency test for Ruby and there's nothing concurrent about it. Ruby doesn't implicitly parallelize block invocations (heck, Ruby 1.8 doesn't even support multiple threads) so there's nothing "concurrent" about this test. However, it does demonstrate a recursive mix of method invocation and block invocation, so I don't know if I'd say it's complete bullshit when comparing implementations of the same language.
ReplyDeleteIt just doesn't measure anything to do with concurrency in Ruby.
Re memory: I think quite a few people care. The difference between paging and not-paging is huge, and memory, not CPU, is typically the limiting resource on 32-bit systems nowadays.
ReplyDeleteI'd be careful about the memory as well as cpu cycles. Java has an ugly, and well deserved IMO, reputation as a RAM/CPU pig.
ReplyDeleteI won't exactly be keen to stick JRuby applications on an app server if I know it's going to bring the system to a crawl and/or consume so much RAM that we can't put anything else on it.
I've already seen what WebLogic can do to a system. Let's not repeat that.
As far as the memory issue goes, I care. :-)
ReplyDeleteI recently tried to run Mingle on a system with 256 MB of RAM and a 1.7 Ghz CPU. To say the least, it didn't work well. Just loading up the first configuration page took over 10 minutes. Later config steps took over an hour *per page*.
Mingle's a pretty typical JRuby on Rails app too. I've tried other Rails applications on JRuby. And while the problems certainly aren't as dramatic as with Mingle, they are approx. 20-30 times slower just due to memory problems (watching top tells me as much).
I love what you guys are doing with JRuby, but can you look at the memory issues sometime? :-)
With respect to both memory and cpu optimizations, we are also someone, that does not care that much.
ReplyDeleteWhat I mean is, that there are many apps and usage scenarious for which it is of no concern. Completeness and correctness on the other hand, now that is important :-)
That said, if taking the low-hanging fruits means beating the hell out of other implementations due to the excellent JVM, that could be a selling point to many.
anonymous wrote The concurrency benchmark is complete bullshit. ...
ReplyDeleteWhat does concurrency mean on a single processor?
Charles Oliver Nutter And considering that "beating Ruby performance" wasn't a primary goal for JRuby 1.0, I think our "roughly on par" numbers here are pretty damn good.
ReplyDeleteI think it's good that so many of the programs run unchanged - unlike ruby 1.9.0
Isaac: thanks for your comments. On your questions about startup time for JRuby:
ReplyDeleteStartup time is worse for JRuby than for plain old Java because we have the additional overhead of initializing Ruby core classes and JRuby runtime structures. So there's a double cost to starting up a Ruby program under JRuby, and that makes a significant difference on many benchmarks. But we don't whine about it, and most of the time when we run these sorts of benchmarks we include startup time anyway.
Incidentally, the programs have been re-measured - instead of using
ReplyDelete-J-server -J-Djruby.jit.threshold=0 -O
most of the programs are now run with a more normal Java command line
-J-server -J-Xbatch -O
The result is that some programs became ~20% faster, some became ~10% slower.
The exception is nsieve which became 100% slower, so for nsieve we still do that ole jit.threshold trick.
isaac: Thanks for the update. Yes, in JRuby 1.0 the -Djruby.jit.threshold=0 line helps a lot of programs, since many will call a target method that performs a lot of work only once, which doesn't trigger JRuby's JIT.
ReplyDeleteIn JRuby 1.1, probably to be released by the end of October, all code should be compilable. Once we've reached that point, most scripts will just be compiled outright on load, with JIT reserved for methods generated at runtime via eval and other tricks. At that point, setting the threshold will be much less necessary.
Actually, already on JRuby trunk nsieve is looking very good, and compiles everything except the "break" in the "downto" loop. But of course that prevents the whole script from compiling, so JRuby falls back on JIT. We're close.