Friday, January 8, 2010

Busy Week: JRuby with Android, Maven, Rake, C exts, and More!

(How long has it been since I last blogged? Answer: a long time. I'll try to blog more frequently now that the (TOTALLY INSANE) fall conference season is over. Remind me to never have three conferences in a month ever again.)

Hello friends! It's been a busy week! I thought I'd show some of what's happening with JRuby, so you know we're still here plugging away.

Android Update

Earlier this week I helped Jan Berkel get the Ruboto IRB application working on a more recent JRuby build. There were a few minor tweaks needed, and Jan was interested in helping out, so I made the tweaks and added him as a committer.

The Ruboto IRB project repository is here: http://github.com/headius/ruboto-irb

Lately my interest in JRuby on Android has increased. I realized very recently that JRuby is just about the only mainstream JVM languge that can create *new* code while running on the device, which opens up a whole host of possibilities. It is not possible to implement an interactive shell in Groovy or Scala or Clojure, for example, since they all must first compile code to JVM bytecode, and JVM bytecode can't run on the Dalvik VM directly.

As I played with Ruboto IRB this week, I discovered something even more exciting: almost all the bugs that prevented JRuby from working well in early Android releases have been solved! Specifically, the inability to reflect any android.* classes seems to be fixed in both Android 1.6 and Android 2.0.x. Why is this so cool? It's cool because with Ruboto IRB you can interactively play with almost any Android API:



This example accesses the Activity (the IRB class in Ruboto IRB), constructs a new WebView, loads some HTML into it, and (not shown) replaces the content WebView. Interactively. On the device. Awesome.



I am trusting you to not go mad with power, and to use Ruboto only for good.

RubyGems/Maven Integration

JRuby has always been a Ruby implementation first, and as a result we've often neglected Java platform integration. But with Ruby 1.8.7 compatibility very solid and Ruby 1.9 work under way, we've started to turn our attentions toward Java again.

One of the key areas for language integration is tool support. And for Java developers, tool support invariably involves Maven.

About a year ago, I started a little project to turn Maven artifacts into RubyGems. The mapping was straightforward: both have dependencies, a name, a description, a unique identifier, version numbers, and a standard file format for describing a given package. The maven_gem project was my proof-of-concept that it's possible to merge the two worlds.

The maven_gem repository is here: http://github.com/jruby/maven_gem

The project sat mostly dormant until I circled back to it this fall. But once I got the guys from Sonatype involved (purveyors of the Nexus Maven server) things really got interesting.

Thanks to Tamas Cservenak from Sonatype, we now have something once thought impossible: full RubyGems integration of all the Maven artifacts in the world!

The Nexus RubyGems support repository is here: http://github.com/cstamas/nexus-ruby-support/

~/projects/jruby ➔ gem install com.lowagie.itext-rtf
Successfully installed bouncycastle.bcmail-jdk14-138-java
Successfully installed bouncycastle.bcprov-jdk14-138-java
Successfully installed bouncycastle.bctsp-jdk14-138-java
Successfully installed com.lowagie.itext-rtf-2.1.7-java
4 gems installed
Installing ri documentation for bouncycastle.bcmail-jdk14-138-java...
Installing ri documentation for bouncycastle.bcprov-jdk14-138-java...
Installing ri documentation for bouncycastle.bctsp-jdk14-138-java...
Installing ri documentation for com.lowagie.itext-rtf-2.1.7-java...
Installing RDoc documentation for bouncycastle.bcmail-jdk14-138-java...
Installing RDoc documentation for bouncycastle.bcprov-jdk14-138-java...
Installing RDoc documentation for bouncycastle.bctsp-jdk14-138-java...
Installing RDoc documentation for com.lowagie.itext-rtf-2.1.7-java...


Here's a an example of a full session, where I additionally install Rhino and then use it from IRB: http://gist.github.com/271764

I should reiterate what this means for JRuby users: as of JRuby 1.5, you'll be able to install or use in dependencies any Java library ever published to the public Maven repository. In short, you now have an additional 60000-some libraries at your fingertips. Awesome, no?

There are some remaining issues to work through, like the fact that RubyGems itself chokes on that many gems when generating indexes, but there's a test server up and working. We'll get all the issues resolved by the time we release JRuby 1.5 RC1. Jump on the JRuby mailing list if you're like to discuss this new capability.

Rake? Ant? Why not Both?

Another item on the integration front is Tom Enebo's work on providing seamless two-way integration of Rake (Ruby's build tool) and Ant. There's three aspects to Rake/Ant integration:
  • Using Rake tasks from Ant and Ant tasks from Rake
  • Calling Rake targets from Ant and Ant targets from Rake
  • Mixed build systems with part in Rake and part in Ant
Tom's work so far has focused on the first bullet, but the other two will come along as well. You'll be able to translate your Ant script to Rake and have it work without modification, call out to Rake from Ant, include a Rakefile in Ant and use its targets, and so on.

Here's an example of pulling in a build.xml file, depending on its targets, and calling Ant tasks from Rake:
require 'ant'

ant.load 'build.xml' # defines tasks :mkdir + :setup in ant!

task :compile => [:mkdir, :setup] do
ant.javac(:srcdir => src_dir, :destdir => "./build/classes") do
classpath :refid => "project.class.path"
end
end

Ideally we'll cover all possible integration scenarios, and finally blur the lines between Rake and Ant. And we'll be able to move JRuby's build into Rake, which will make all of us very happy. Look forward to this in JRuby 1.5 as well.

The C Extension Question

One aspect of Ruby itself that we've punted on is support for Ruby's C extension API. We haven't done that to spite C extension users or developers--far from it...we'd love to flip a switch and have C extensions magically work. The problem is that Ruby's C API provides too-invasive access to the internals of objects, and there's just no way we can support that sort of access without incurring a massive overhead (because we'd have to copy stuff back and forth for every operation).

But there's another possibility we've started to explore: supporting only a "safe" subset of the Ruby C API, and providing a few additional functions to replace the "unsafe" invasive bits. To that end, we (as in Wayne Meissner, creator of the FFI implementations for JRuby and Ruby) have cleaned up and published a C extension API shim library to Github.

The JRuby C extension shim library repository is here: http://github.com/wmeissner/jruby-cext

Last night I spent some time getting this building and working on OS X, and to my surprise I was able to get a (very) trivial C extension to work!

Here's the part in C. Note that this is identical to what you'd write if you were implementing the extension for Ruby:
#include <stdio.h>
#include <ruby.h>

VALUE HelloModule = Qnil;
VALUE HelloClass = Qnil;

VALUE say_hello(VALUE self, VALUE hello);
VALUE get_hello(VALUE self);

void
Init_defineclass()
{
HelloClass = rb_define_class("Hello", rb_cObject);
rb_define_method(HelloClass, "get_hello", get_hello, 0);
rb_define_method(HelloClass, "say_hello", say_hello, 1);
}

VALUE
say_hello(VALUE self, VALUE hello)
{
return Qnil;
}
VALUE
get_hello(VALUE self)
{
return rb_str_new2("Hello, World");
}

Here's the little snippit of Ruby code that loads and calls it. Note that the ModuleLoader logic would be hidden behind require/load in a final version of the C extension support.
require 'java'

m = org.jruby.cext.ModuleLoader.new
m.load(self, "defineclass")

h = Hello.new
puts "Hello.new returned #{h.inspect}"
puts "h.get_hello returns #{h.get_hello}"

Among the C API pieces we probably won't ever support are things like RSTRING, RARRAY, RHASH that give direct access to string, array, and hash internals, anything dealing with Ruby's threads or runtime, and so on. Basically the pieces that don't fit well into JNI (the Java VM C API) would not be supported.

It's also worth mentioning that this is really a user-driven venture. If you are interested in a C extension API for JRuby, then you'll need to help us get there. Not only are we plenty busy with Java and Ruby support, we are also not extension authors ourselves. Have a look at the repository, hop on the JRuby dev list, and we'll start collaborating.

JRuby in 2010

There's a lot more coming for JRuby in 2010. We're going to finally let you create "real" Java classes from Ruby code that you can compile against, serialize, annotate, and specify in configuration files. We're going to offer JRuby development support to help prioritize and accelerate bug fixes for commercial users that really need them. We're going to have a beautiful, simple deployment option in Engine Yard Cloud, with fire-and-forget for your JRuby-based applications (including any non-Ruby libraries and code you might use like Java, Scala or Clojure). And we're going to finally publish a JRuby book, with all the walkthroughs, reference material, and troubleshooting tips you'll need to adopt JRuby today.

It's going to be a great year...and a lot of fun.

17 comments:

  1. As I'm looking into Android dev, this is pretty interesting. I'd heard reports that while precompiled Clojure works (as long as you don't use eval), the boot times were lousy, making it an iffy choice for Android. This actually had me curious about the use of Duby instead, since presumably JRuby would have the same boot time issues, though if that's not true I'd be thrilled. Have you tried Duby on dalvik yet?

    ReplyDelete
  2. Wow, impressive work Charles! Great to see JRuby improving its integration with the Java world.

    ReplyDelete
  3. I haven't had time to explore jruby more than in IRB...to take an explicit example: Will it be posible to run - say - HornetsEye ruby gem on jruby + android in the near future?

    ReplyDelete
  4. @Charles: OK, fascinating. The reason I ask is that I might do my master thesis this spring on augmented reality, and I elaborated with the idea of using jruby on android. =) Follow-up question: Is there any roadmap for getting jruby + android rock 100%? With that I mean "stable enough for development at least".

    ReplyDelete
  5. grimen: In my head, there's a roadmap...but finding time and resources to work on it is rather tricky. There's a lot of stuff happening with JRuby right now, and Android is one of many projects. I really need more help to flesh it out and get it ready for prime time. Maybe you can help? :)

    ReplyDelete
  6. @Charles: More specifically...what would you want help with? I'm experienced in Ruby (and Java programming...well haven't touched in 2 years now), but if we talking c coding my passion is sort of non-existing. =)

    P.S. I'm busy now with Rails, but when it's time for master thesis (in Mars at earliest) I need to do some research on android/jruby so maybe I ould help out somehow by then.

    ReplyDelete
  7. grimen: I think it would be useful to start playing with JRuby's current compiler and try to precompile Ruby code and get it to load on Android. There are some challenges, like the extra code it generates for method bindings, but I don't think there's anything we can't overcome. If you started playing with it and hit specific roadblocks, it would help me prioritize some of the heavy lifting on the JRuby side.

    ReplyDelete
  8. @grimen: HornetsEye makes full use of the Ruby C-extension API (including memory allocations with 'xmalloc'). Furthermore it makes use of libJIT and Video4Linux. I'm quite sure that the current version of HornetsEye will not run on Android without modifications.

    ReplyDelete
  9. Jan: That is unfortunate. When will they learn!

    ReplyDelete
  10. @Jan: OK, thanks for the heads up. Will have to use some java library I guess.

    ReplyDelete
  11. ..and yea, thanks for the Marvin link, it might become very useful.

    ReplyDelete
  12. Jan: Perhaps you would be interested in helping us with the C extension shim for JRuby?

    Phoet: Yes, we know of buildr, and it's a great tool if you want to (and are able to) follow the maven conventions. But it does not provide full two-way integration between ant and rake, which we would like to have out of the box for JRuby. I could see buildr being reworked to run atop JRuby's rake/ant integration in the future, or JRuby integrating maven logic more directly, but our current intention is to provide the lowest levels of integration without imposing project structure on users.

    ReplyDelete
  13. @charles I was afraid someone would say that ;)
    I would be interested in getting HornetsEye running with JRuby. I'll put it on my TODO list. At the moment however I want to implement lazy computation in order to get sufficient performance for doing real-time feature matching and hopefully 3D object recognition in the end.

    ReplyDelete
  14. Could you expand on this: "JRuby is just about the only mainstream JVM languge that can create *new* code while running on the device"

    Is jruby doing something different to jython and groovy here?

    ReplyDelete
  15. Nice work, I managed to get Ruboto IRB working perfectly on the emulator.

    Sorry if I'm missing something obvious, but is it possible to execute ruby scripts with the application or are we limited to running in interactive mode?

    ReplyDelete
  16. comment_bot: Neither Jython nor Groovy nor Clojure nor Scala have an interpreted mode, which means all code they execute has to be compiled to JVM bytecode. Since the Dalvik VM on the device does not run JVM bytecode (it precompiles to Dalvik bytecode when installing) none of those languages can create new code while on the device.

    Rhino (JavaScript) does have an interpreter, but I'm not sure anyone's using it for Android stuff.

    ReplyDelete
  17. Rake/Ant integration and Maven gem can simplify several hybrid Ruby/Scala/Java tools I work on. Definitely worth having around. Many thanks!

    ReplyDelete