Wednesday, February 18, 2009

JRuby 1.2: Coping With Bugs

We get a lot of bug reports on the JRuby project. No, it's not because JRuby's a buggy POS. It's because we're implementing the core of an entire platform. It's hard.

Over the past year or so we kinda let the bug tracker go a little wild. We've never had more than two full-time people on the project, and community contributions wax and wane with the seasons, so we've had to choose our battles. For some amount of time, we would focus on performance, since without a reasonably-performing implementation, nobody would choose to use JRuby. Then we'd cycle back to fixing bugs, usually in the last couple weeks before a release. So the flow of bug fixes waxed and waned with our release cycles.

The problem with this approach was that we really couldn't keep up with all the reports coming in and continue to work on the "unreported issues" of JRuby like performance. So something had to give, and it was sometimes the bugs.

Roughly around September, we crossed an important threshold: 500 open bugs. They dated as far back as 2006 (we moved to Codehaus in 2006), and across the entire release sequence of JRuby; there were even some bug reports against pre-1.0 versions. From September until about mid-January, we worked to keep the number of open bugs below 500. Sometimes it would peek above, but we were generally successful.

About mid-January, I figured it was time to really bite the bullet, and I started walking from the oldest bug forward. It has been a long slog, but I'm making great progress.

So far, I've managed to whittle the total number of bugs down from 506 to 326, and that's with a couple dozen more filed in the past two weeks. The majority of these were either already fixed or invalid/incomplete bugs, but I've also made a concentrated effort to focus on and fix as many as possible. At the very least, I've made an attempt on every single one.

During that process, I learned a bit about how my brain works. During the past year, if I didn't know immediately how to fix a bug I often walked away from it and worked on something fun like compiler optimization. But during this latest bug hunt, because I did not want to just punt all those bugs one more time, I did something a little different.

Whenever I encountered a bug that seemed difficult or that I simply didn't want to fix, I would do one of two things:
  • Take a break. Often this meant I had reached my limit and needed a breather. A little Wii, a glass of scotch, a movie...anything to take my mind of it for a bit.
  • Count to ten. Sometimes, after reading through the bug, all I really needed was a moment for my brain to start "wondering" about the solution. Once the wheels started turning, I was almost compelled to dig deeper.
As a result, many long-standing bugs--including bugs I thought we would never fix--are now resolved and ready for the 1.2 release. I'm going on three weeks now of 12-hour days trying to fix what seems like an endless stream of bugs, but the end is in sight. And I want to stress that this level of effort has never happened in a single release. There's as much work in JRuby 1.2 as in multiple 1.1.x releases combined. I'll be blogging more of these posts over the next couple weeks.

For your amusement, here is the complete list of bugs we have resolved during the past nine weeks, since shortly after the 1.1.6 release. These aren't all fixes, but even the "already fixed", "won't fix", "not a bug", or "incomplete" ones were reviewed, tested, and considered. A couple of these I just applied patches or marked resolved based on someone else's work. And of course we're not done with JRuby 1.2 yet. Just you wait and see :)

Bugs Resolved (Fixed, Won't Fix, Not A Bug, or Incomplete Information), Past Nine Weeks:

(in the order they were resolved)

JRUBY-2538

File#chmod always returns -1, fails rubyspec

JRUBY-1505

Solidify public APIs separate from those that might change

JRUBY-3208

ant task api-docs runs out of memory, patch included

JRUBY-3221

Rubinius -bad instanceof tests in RubiniusMachine for attempt at fast path evaluation

JRUBY-3232

Zlib::GzipReader.each_line yields compressed data

JRUBY-3244

[PATCH] Support atime updates via utime

JRUBY-3249

Testing errors in test_java_mime.rb

JRUBY-3250

test_java_pkcs7.rb - Problem coercing Bignum into class java.math.BigInteger

JRUBY-2992

Bad cast in jruby-openssl X509Utils

JRUBY-3204

IO.read and IO.readlines do not take a block

JRUBY-3251

ConcurrencyError bug when installing Merb on JRuby

JRUBY-3248

jruby can't load file produced with jrubyc

JRUBY-3257

RaiseException#printStrackTrace doesn't deal well with nil messages

JRUBY-3259

sqlite db file on MRI is put in db dir. We put it in CWD.

JRUBY-3260

:date is stored as integer (not date) in sqlite3

JRUBY-2425

NPE at Connection Shutdown

JRUBY-2329

java.lang.NullPointerException when trying to connect to a Oracle XE database

JRUBY-3023

Renaming indexed columns fails

JRUBY-2384

Rails migrations fail with: undefined method `create_database' for class `#'

JRUBY-2400

ActiveRecord/JRuby Memory leak in update_attributes!()

JRUBY-2040

new samples/jnlp dir: includes both unsigned and signed examples that work in Java 1.5 and 1.6

JRUBY-3270

using activerecord 'to_xml' method crashes JRuby (and debugger)

JRUBY-3277

Constant lookup should omit topmost lexical scope; not bail out on first Object-hosting lexical scope

JRUBY-3278

Array#reduce is undefined

JRUBY-3187

JRuby command line scripts do not work with spaces in the path

JRUBY-3267

patch attached to update eclipse .classpath to refer to bytelist-1.0.1.jar

JRUBY-3280

patch attached to update eclipse .classpath to add new jars in build_lib/

JRUBY-3276

New IO#readpartial and IO#seek spec failures

JRUBY-3256

jruby -S script/console fails in Rails application

JRUBY-3199

IllegalArgumentException in X509V3CertificateGenerator when executing Hoe's rake gem running on SoyLatte JDK

JRUBY-3275

New ARGF spec failures

JRUBY-3287

Add public RubyInstanceConfig#setCompatVersion

JRUBY-3171

loading rake breaks import

JRUBY-3290

Float range to restrictive in FFI code

JRUBY-2826

Array#pack('w') is broken (exceptions and wrong results)

JRUBY-3116

ArrayIndexOutOfBoundsException in tmail

JRUBY-2522

Can not execute JRuby 1.1.1 "ant test" with IBM JDK

JRUBY-2426

JRuby trunk does not build with IBM JVM 1.5

JRUBY-3042

undefined method `quote' for Regexp:Class (NoMethodError) when runs gem list on AIX with IBM JAVA 1.5

JRUBY-1950

'-i flag not supported

JRUBY-3302

Doesn't biuld on windows for ffi problem

JRUBY-3303

[1.9] splat of array at front of assignment broken (same for other grammatical constructs)

JRUBY-3307

tempfile.rb sucks in JRuby

JRUBY-3305

Fixnums & Floats should be coerced to the narrowest java.lang.Number when passed to a java method that expects Object

JRUBY-2768

jdbc_mysql.rb should not enforce the utf8_bin collation on table creation

JRUBY-2115

Error doing add_column with Derby and jdbc-adapter-0.7.2

JRUBY-2967

[PATCH] Derby handling of binary/date/time fields. hsqldb handling of binary fields.

JRUBY-2647

[activerecord-jdbcsqlite3 adapter] Date/Datetime/Time fields simplification to INTEGER sqlite storage type is not correctly handled

JRUBY-3073

JRuby script engine does not load on IBM JDK

JRUBY-3109

Update stdlib to an appropriate 1.8.6 patchlevel

JRUBY-2541

Time assumes UTC timezone

JRUBY-3254

Make JRuby's Time#local behave more like MRI

JRUBY-2740

BigDecimal#precs fails two new rubyspec tests

JRUBY-3298

Kernel.srand calls #to_i on number

JRUBY-2987

Kernel#system does not work correctly

JRUBY-3242

Performance degradation on write operations

JRUBY-2128

error accessing JRuby DRb server from MRI DRb client

JRUBY-3268

JRuby DRb client cannot connect to MRI DRb server

JRUBY-3274

Many new Array#pack failures in updated RubySpecs

JRUBY-3147

Time#_load RubySpec failures

JRUBY-3146

Time#_dump RubySpec failures

JRUBY-3148

Array#fill spec failures

JRUBY-3142

Dir.pwd correctly displays dirs with unicode characters in them

JRUBY-2609

spec --line does not work with JRuby

JRUBY-3317

require "tempfile"; Dir.tmpdir fails

JRUBY-2364

The private keyword doesn't change visiblity of previously called methods with same send/call site

JRUBY-2796

Missing constant in Process::Constants on Mac OS X (darwin)

JRUBY-3226

Including more than one java interface stops Test::Unit execution

JRUBY-3035

Get Rails tests running in a CI server somewhere before 1.1.6 release, to ensure no future breakages

JRUBY-3318

If 'jruby' executable is a relative symlink, JRuby won't start

JRUBY-3328

Incorrect behavior for case/when when true or false are the first conditions

JRUBY-3269

SystemStackError with YAML.dump

JRUBY-3229

Javasand does not compile with JRuby 1.1.5

JRUBY-3333

idea

JRUBY-3332

ConcurrentModificationException In Application Startup

JRUBY-3329

Major divergence in threading behaviour between Ruby 1.8.6 and JRuby 1.1.6

JRUBY-3308

Deadlock between RubyModule.includeModule() and RubyClass.invalidateCacheDescendants()

JRUBY-3324

Error in string encoding conversion

JRUBY-3340

String#unpack('m') fails with ArrayIndexOutOfBoundsException

JRUBY-3341

The body of POST requests is not passing through jruby-rack

JRUBY-3312

JRuby can double slash ("//") the directory given by __FILE__ when "everything" suits this thing to happen. :-)

JRUBY-2884

jdbc_postgre.rb issue handling nil booleans

JRUBY-2208

Memory Leak in ActiveRecord-jdbc or JRubyEngine used with BeanScriptingFramework

JRUBY-2897

jdbc_postgre.rb needs microsecond support

JRUBY-2995

activerecord jdbc derby adapter should quote columns called "year"

JRUBY-2949

jdbcderby adapter should ignore :limit for INTEGER

JRUBY-2718

jdbcsqlite3 at JRuby on Rails do not work properly

JRUBY-3344

Update H2 to a later version

JRUBY-3342

Wrabler escapes config.ru twice

JRUBY-3282

Upgrade to derby 10.4.2.0 to allow unique constraints with nullable columns

JRUBY-2767

Mistake in selecting identity with H2/HSQLDB

JRUBY-2433

java.lang.IllegalArgumentException in debug mode when using CLOB

JRUBY-3310

Race condition in JRuby when Active Support is overriding Kernel.require

JRUBY-3127

Inconsistent DynamicScope instances in evalScriptlet

JRUBY-3339

java.lang.ArrayIndexOutOfBoundsException in RubyArray.java

JRUBY-3351

Tempfile defaults to a directory that does not exist on Windows

JRUBY-3353

String#unpack invalid output

JRUBY-3361

Compiled case/when blows up with arrays as when clauses

JRUBY-3362

attr_accessor, attr_reader, attr_writer, and attr not obeying visibility

JRUBY-1034

Post parse processing of the result comment placing as example

JRUBY-48

Ability to write JUnit4+ tests in JRuby

JRUBY-1572

Allow visible fields from inherited Java classes show up as instance variables (@foo)

JRUBY-2108

IOWaitLibrary ready? returns nil when ready

JRUBY-1895

JRUBY-1890 IO.popen does not support "-"

JRUBY-852

jirb exits on ESC followed by any arrow key followed by return

JRUBY-2015

"RAILS_ROOT" cannot be put on a place different from "Context root" in Goldspike.

JRUBY-2139

jruby.compile.frameless=true cause "Superclass method 'initialize' disabled."

JRUBY-2174

Method#to_proc.call invocations much slower than normal Method#call invocations

JRUBY-2170

Very unverbose message when problem with native library exists

JRUBY-3338

jIRB cannot get any input with Polish characters (utf-8) on Mac OSX

JRUBY-2252

Kernel#exec should not raise SystemExit

JRUBY-1855

EOF handling in case of reading program from standard input

JRUBY-1997

System.gc() is extremely expensive when calling Java code from JRuby

JRUBY-821

getting in $SAFE level 4 freezes

JRUBY-1770

weakref-based singleton class "attached" can blow up in certain unusual circumstances

JRUBY-1555

DST bug in second form of Time.local

JRUBY-1872

next statement should return the argument passed, not nil

JRUBY-2214

Pure-Java version of lstat doesn't fail on non-existing file

JRUBY-1901

standardize and publish public API for embedders

JRUBY-2211

SNMP Requests does not timeout properly

JRUBY-2298

We should generate method binding code with APT

JRUBY-2305

File output cuts off at a certain point?

JRUBY-2306

NPE in org.jruby.RubyIO.flush

JRUBY-2288

UDP locking problems when receiving from multiple senders on same port

JRUBY-2333

usage of unexistant variable in REXML::SourceFactory

JRUBY-2346

Nailgun does not compile under HP-UX

JRUBY-2353

Process.kill breaks up JRuby, while works in MRI on Windows

JRUBY-2369

'ant spec' failures on on MacOS

JRUBY-2168

test_io (test_copy_dev_null) fails under WinXP

JRUBY-2410

Multiple assignment could be made faster by not returning unused array

JRUBY-2389

java.util.ConcurrentModificationException

JRUBY-929

Java input/output streams from Ruby string's backing byte array

JRUBY-1681

Pathological slowdown in Array#insert

JRUBY-1470

FileTest methods only work with string arguments

JRUBY-1576

multi-byte character encoding probrem

JRUBY-2119

Jruby shell script don't work on KSH shell

JRUBY-3360

Pure-Java tempfile does not use a #make_tmpname callback

JRUBY-2518

Dir["some glob"] doesn't work for files inside a jar

JRUBY-3367

Dir.glob processes nested braces incorrectly

JRUBY-3366

Faulty marshalling error blows up in ActiveSupport

JRUBY-2027

Problems with the H2 jdbc adapter, schema metadata behavior

JRUBY-2417

Make fields immutable where possible

JRUBY-1565

ReWriteVisitor transforms Fixnums into base 10 despite their original format

JRUBY-2917

Missing peephole optimizations in parser, compiler

JRUBY-2542

DATA.flock causes ClassCastException

JRUBY-2545

Recursively locking on a mutex results in deadlock

JRUBY-2285

Regression: rubyspec run with -V option breaks JRuby

JRUBY-2120

Wrong stacktraces from exceptions about wrong number of arguments

JRUBY-1639

Exception when stopping WEBrick running the new rails application

JRUBY-2534

jdbcpostgresql adapter inserts invalid number literals into generated SQL, causes exceptions

JRUBY-3375

'attr' implementation is incompatible with MRI 1.9.1 behavior

JRUBY-3365

Objects with custom to_yaml methods break collection to_yaml

JRUBY-2603

Couple of new rubyspec failures for URI library

JRUBY-2465

2 digit year resolves to 1st century rather than 20th when using postgres timestamp with timezone + activerecord-jdbcpostgresql-adapter

JRUBY-2672

BasicSocket#close_read and #close_write not working

JRUBY-2655

TCPSocket.new('localhost', 2001) connects on the wrong interface

JRUBY-2759

Time Zone "MET" maps to Asia/Tehran

JRUBY-2758

Class Socket is missing #bind

JRUBY-2761

Thread#wakeup fails when sleep is called with a parameter

JRUBY-2772

Zlib::Deflate doesn't appear to deflate

JRUBY-3289

ArgumentError: dump format error() when unmarshalling Rails session and EOFError when unmarshalling hash with containing value of type ActionController::Flash::FlashHash

JRUBY-3228

Some error saving to a Rails session [Marshalling bug?]

JRUBY-2792

Net::HTTP spec failures in get_print spec; IO not getting expected "print" call

JRUBY-2804

IO.popen of 'yes' process hangs on close

JRUBY-2755

Precompiled scripts do not set initial position information correctly

JRUBY-2484

unicode symbol not supported as in MRI

JRUBY-2817

File.chown works with 64bit JDK, but raises NotImplementedError on 32 bit JDK

JRUBY-2912

MS SQL-Server - Create/Drop Database bug

JRUBY-2815

System triggered from inside of tracing behaves oddly

JRUBY-3069

dbd-jdbc: Support Ruby DBI Statement.fetch_scroll

JRUBY-3235

dbd-jdbc: Add helper functions to wrap already existing jdbc Connection and Statement objects

JRUBY-3068

dbd-jdbc: Override Ruby DBI's default database return value coercion for Date / Time / Timestamp columns

JRUBY-3067

dbd-jdbc: Register a handler for DBI 0.4.x's default parameter binding type coercion

JRUBY-1345

Files added to a module via Kernel.autoload are not automatically loaded if the class they contain is extended

JRUBY-2727

ActiveScaffold 1.1.1 fails with JRuby 1.1.2, probable path problem

JRUBY-2699

If user's PATH contains '.' and 'jruby' resolves to './jruby', JRUBY_HOME is incorrectly set to '.', leading to 'NoClassDefFoundError'

JRUBY-2820

Most Etc methods behave diferently on Windows under x32 and x64 JVMs

JRUBY-2841

String#<<>

JRUBY-2844

Memcache session store does not work across multiple application servers

JRUBY-2845

'-S switch breaks if you have directory in pwd matching name of file in bin/PATH

JRUBY-2849

CCE using proc{} in AR callbacks

JRUBY-2862

Improve JRuby::Commands' handling of bindir

JRUBY-2861

Cannot call super inside a method that overrides a protected method on Java base class

JRUBY-2868

ThreadService.getRubyThreadFromThread supports only one non-rubynative thread at a time.

JRUBY-2782

"No such file or directory (IOError)" when JAR file named in java.class.path but not active filesystem

JRUBY-2901

JRuby does not support options in shebang line

JRUBY-2846

gem generate_index broken

JRUBY-2832

Rails static page caching won't generally work with Java ServletFilter unless some serious hack with the cache directory is found

JRUBY-414

Serialization of wrapped Java objects could be handled as user-marshalled data by Marshal

JRUBY-2921

Newly overridden methods are not called if the old superclass method has already been called

JRUBY-2661

specs hang under Apple Java 5

JRUBY-2933

Ruby runtimes are pinned to memory due to ChannelStream

JRUBY-1578

Incomplete HTTS response from Microsoft IIS servers using net/https

JRUBY-2941

popen IO lockup with multiple threads

JRUBY-2925

Java Integration can now instantiate Java classes with no public constructor

JRUBY-2802

JRuby wrapped Java objects formerly exposed readers for non-public fields

JRUBY-2504

JAR URLs are inconsistently supported by APIs that access the filesystem

JRUBY-1846

Kernel::fork should use forkall(2) on solaris instead of fork(2)

JRUBY-2395

Have to call setPaint instead of assigning to g.paint

JRUBY-1401

Pathname#realpath fails for Windows drive letters

JRUBY-2296

convertToRuby does not work if to_s does not return a String (but for example nil)

JRUBY-2904

Need a better solution for tracking Java object wrappers

JRUBY-1327

When define_method replaces an inherited Java method, the ruby_case_version is not replaced

JRUBY-2951

repeated popen call may throw NULL pointer exception

JRUBY-2974

multi-byte exception message turned into garbled characters when Java Exception nested NativeException.

JRUBY-2980

String#<<>

JRUBY-2981

alias_method_chain pattern is extremely show on jruby

JRUBY-2982

Unicode regular expressions by UTF-8 don't work

JRUBY-2988

jirb does not echo characters to the terminal after suspend and resume in the shell

JRUBY-2997

Dir[] results in NPE depending on permissions

JRUBY-3000

integration issues with Java classes containing inner classes

JRUBY-3007

Message method value of Exception is missing in Java when exception raised in ruby

JRUBY-3016

Nested Class/Enum issue with JRockit

JRUBY-3011

java classes with non-public constructors are incorrectly instantiated

JRUBY-2999

Regression: Inheriting method with same name from two Java interfaces causes Java classloader error

JRUBY-2700

jdbcsqlite3 adapter does not respect default parameter

JRUBY-3026

[Derby] Allow select/delete/update conditions with comparison to NULL using '='

JRUBY-3288

jruby-openssl: SSLSocket.syswrite error for big (>16k) data

JRUBY-3017

DRb "premature header" error on JRuby client

JRUBY-2766

Create caching mechanism for all our many calls to callMethod from Java code

JRUBY-3200

insert statement returns null from the postgresql jdbc adapter

JRUBY-2996

Migration step with create_table with string column with :default => "" fails due to missing default value in SQL

JRUBY-2184

ActiveRecord is slower on JRuby than on MRI

JRUBY-3202

url style Postgres configuration is broken in rake.

JRUBY-1854

def is almost 3x slower in JRuby

JRUBY-2618

Table does not exist error possibly caused by upper case table name

JRUBY-3003

ArrayStoreException in compiler when calling Array.fill

JRUBY-2180

break performance is slower than MRI, sometimes slower than interpreted

JRUBY-2304

active_scaffold - No such file or directory

JRUBY-3386

Array#eql? rubyspec failure

JRUBY-3387

Array#== rubyspec failure

JRUBY-3054

Thread#status is "run" when thread is blocking on condition variable

JRUBY-3390

RubySpec: Numeric#coerce calls #to_f to convert other if self responds to #to_f

JRUBY-3055

permission javax.management.MBeanServerPermission "createMBeanServer";

JRUBY-3072

Using a ruby Numeric to set a parameter with a signature of java.lang.Integer causes a java.lang.ClassCastException

JRUBY-3082

1.1.4 unable to use 'logging' gem (works in 1.1.3)

JRUBY-3398

Attributes defined from Java tried to use current frame for visibility

JRUBY-3397

defined? CONST is not following proper constant lookup rules

JRUBY-3089

Digest::Base stores in memory all bytes passed to the digest, causing OutOfMemoryError

JRUBY-3125

Some Socket constants not supported.

JRUBY-3128

Different jruby behaviour that ruby 1.8.6/1.8.7/1.9 when running file that is named as one of required file.

JRUBY-3175

Cloning java byte array returns incorrect object

JRUBY-3177

ConcurrentModificationException

JRUBY-3178

http spec tests fail when webrick starts on address 0.0.0.0

JRUBY-2548

JRuby conversion of String to Java String is arcane

JRUBY-3174

FFI MemoryPointer#get_string(0, x) gets confused by null byte in the buffer

JRUBY-2896

jruby -S picks up script from local directory

JRUBY-3163

Incorrect namespace for CipherError

JRUBY-1853

define_method methods do not display the correct stack trace, may not be using frame safely

JRUBY-2855

Classpath search order is not respected by require when classpath includes both jars and directories

JRUBY-3222

Problems with migrations where direct type specified

JRUBY-2424

running with -rprofile with multithreaded test causes NPE

JRUBY-2691

Update sybase driver to pass simple unit tests with jtds and verify it works with the new dialect keyword

JRUBY-3065

New public LoadService#findFileToLoad method

JRUBY-2261

ThreadContext should use SoftReference instead of WeakReference

JRUBY-2116

import org.eclipse.jface.dialogs.MessageDialog -- causes java.lang.NullPointerException

JRUBY-2898

Exception trying to use BSF (java.lang.NoClassDefFoundError: org/apache/bsf/util/BSFEngineImp)

JRUBY-2003

rexml pretty printing wrap() error

JRUBY-752

undef'ing and redef'ing methods should give disabled message?

JRUBY-2262

Java toString() mapping to invoke(... to_s ...).toString() doesn

JRUBY-3121

Glassfish Gem 0.9.0 does not work with JRuby 1.1.5

JRUBY-551

JRUBY-549 bfts test_file_test failures

JRUBY-2248

Problem with active_scaffold defaults defined in application.rb

JRUBY-2446

httprequest error for a SOAP request to a Rails ActiveWebService

JRUBY-3005

Stomp gem hangs in Stomp::Client#subscribe

JRUBY-3180

Class in default package can't be reopened under Java namespace

JRUBY-3047

IO.select sometime reports Unable to establish loopback

JRUBY-2851

JDBC adapter in JAR file not being picked up by java.lang.Class.forName in 1.1.3 (worked in 1.1.2)

JRUBY-2313

Automatically convert RubyFile to java.io.File

JRUBY-2328

Overriding require causes eval to give wrong __FILE__ in certain circumstances

JRUBY-2794

Failure in String#crypt specs, Mac OS X, Soylatte

JRUBY-2137

ENV[] is case sensitive on Windows

JRUBY-2797

File.expand_path rubyspec failure on Mac OS X, Soylatte

JRUBY-2608

Examine whether -Xrs flag to JVM could be used to improve our signal-handling logic

JRUBY-2675

jirb + jline do not work in cygwin

JRUBY-2521

Compilation Warning for Sun proprietary API using

JRUBY-2263

Running merb under a concurrent load causes ArrayIndexOutOfBoundsExceptions

JRUBY-2706

File.stat('/home') treated as a relative path

JRUBY-2557

NPE when yielding method

JRUBY-2860

EOFError while consuming REST web service from glassfish v3

JRUBY-2835

rails image_tag generates url get 404

JRUBY-3095

chmod_R broken

JRUBY-2830

String#unpack with 'w' is completely unimplemented

JRUBY-3038

Unable to insert YAML into t.text field (behavior different from Ruby 1.8.6)

JRUBY-3004

Excess warnings when using rake TestTask

JRUBY-1606

File.expand_path does not work as expected

JRUBY-750

defineAlias reports original name and not aliased one in stack traces

JRUBY-1201

TCPSocket.new(dest_adrr, dest_port, src_addr, src_port) missing from JRuby

JRUBY-3160

REXML returns error for add_attribute

JRUBY-2732

org.jruby.util.RubyInputStream is unused

JRUBY-2455

"jruby" script failed to find JRUBY_HOME leads to class not found

JRUBY-3190

method `module_eval' for main:Object should be undefined?

JRUBY-2950

Net::IMAP#authenticate stalls / blocks in recent versions of JRuby

JRUBY-1542

1.1b1 doesn't build with gcj 4.2

JRUBY-2788

Make Time.now monotonically increasing

JRUBY-3296

Etc.getpwuid should raise TypeError if invalid type

JRUBY-3167

loadpath and classpath problem

JRUBY-2983

JRuby crashes after running 110 RSpec examples

JRUBY-3085

Add constant caching for Colon2 and Colon3 (specific module and Object cases)

JRUBY-3207

super error message is inaccurate when superclass doesn't implement method.

JRUBY-2979

File#getc fails on illegal seek when reading from /dev/ttyS0 on Linux

JRUBY-3076

RubyUDPSocket.send tries to do inet address lookup, pauses too long for inet6 failure

JRUBY-2555

CLONE -JIT max and JIT threshold should be adjusted for improvements in JRuby over the past months

JRUBY-3407

FFI Stack Level Too Deep on Library.send(:method, args)

JRUBY-3410

Add -y flag to debug the parser

JRUBY-3103

Compiler closures use two different binding mechanisms, need better construction and caching logic

JRUBY-3179

JRuby can't handle do/end blocks within String-inlined expressions

JRUBY-3316

Group expression class path fails to parse

JRUBY-3151

OpenSSL::Random#pseudo_bytes raises ArgumentError on negative arguments

JRUBY-3088

"RStone" port of PyStone is slower on JRuby than 1.9

JRUBY-3120

Regression: error serializing array of objects with custom to_yaml def to yaml

JRUBY-3211

builder library is slower in JRuby than in MRI

JRUBY-1884

handling of file close while reading differs from mri

JRUBY-3214

load and require don't work correctly for .class files

JRUBY-3218

Import is sometimes confused by unquoted class name

JRUBY-3224

Array#to_java with java object reference doesn't carry over object identity

JRUBY-1079

IO.sysopen not defined

JRUBY-3071

Illegal seek error trying to read() from pipe

JRUBY-3152

Process.times returns invalid values

JRUBY-3064

Process.uid does not work on non-natve platforms (jna-posix)

JRUBY-2757

Compiled argument processing and arity checking are too large, too much bytecode

JRUBY-3061

NKF(unsupported Q encode stream)

JRUBY-3105

Embedding JRuby-Rack (non-war deployment)

JRUBY-3077

Update ruby_test tests and incorporate changes into build

JRUBY-2970

Timeout.rb is not working in special cases (Thread may block whole application).

JRUBY-3253

Lack of default jruby.home in embedded usage causes NPE

JRUBY-3063

File::link using CIFS raises Errno::EEXIST error if the 'old_file' is open while Matz Ruby raises Errno::ETXTBSY

JRUBY-3252

Iterative String concatenation operations in JRuby degrade faster than in MRI.

JRUBY-3255

jruby -S script/server fails in Rails application

JRUBY-3258

after installing minigems (and after minigem setup gems not working on 1.1.6)

JRUBY-1235

DRb hangs when transferring Ruby objects created in Java

JRUBY-3271

Java Stacktrace and exception is swallowed

JRUBY-2063

Wrong monitor handling in o.j.util.ChannelStream.read

JRUBY-2131

Select with nil arrays should throw an exception

Thursday, January 29, 2009

The JRuby Version String

I just love the JRuby version string. I love it so much, I'm microblogging what each piece means.
jruby 1.2.0 (ruby 1.8.6 patchlevel 287) (2009-01-29 rev 8947+3) [x86_64-java]
  • Obviously "jruby 1.2.0" is the name of the impl and the version number. We'll probably modify this to suffix "-dev" on trunk.
  • "ruby 1.8.6 patchlevel 287" roughly identifies which release of Ruby the current mode is intended to be compatible with.
  • "2009-01-29" is the date it was built.
  • The revision number "8947+3" indicates the base SVN revision is 8947 and I'm three commits ahead of it on my local git-svn clone. If I were using straight-up SVN it would just show current revision. Once we move to git it will just show the current hash, plus possibly the hash of origin's HEAD.
  • And finally "x86_64-java" identifies the hardware and platform we report to Ruby programs.
Now what I really like is what happens when you specify the --1.9 flag:
jruby 1.2.0 (ruby 1.9.1 patchlevel 0) (2009-01-29 rev 8947+3) [x86_64-java]
How cool is that?

My Favorite Hotspot JVM Flags

I probably start up a JVM a thousand times a day. Test runs, benchmark runs, bug confirmation, API exploration, or running actual apps. And in many of these runs, I use various JVM switches to tweak performance or investigate runtime metrics. Here's a short list of my favorite JVM switches (note these are Hotspot/OpenJDK/SunJDK switches, and may or may not work on yours. Apple JVM is basically the same, so these work).

The Basics

Most runs will want to tweak a few simple flags:
  • -server turns on the optimizing JIT along with a few other "server-class" settings. Generally you get the best performance out of this setting. The default VM is -client, unless you're on 64-bit (it only has -server).
  • -Xms and -Xmx set the minimum and maximum sizes for the heap. Touted as a feature, Hotspot puts a cap on heap size to prevent it from blowing out your system. So once you figure out the max memory your app needs, you cap it to keep rogue code from impacting other apps. Use these flags like -Xmx512M, where the M stands for MB. If you don't include it, you're specifying bytes. Several flags use this format. You can also get a minor startup perf boost by setting minimum higher, since it doesn't have to grow the heap right away.
  • -Xshare:dump can help improve startup performance on some installations. When run as root (or whatever user you have the JVM installed as) it will dump a shared-memory file to disk containing all of the core class data. This file is much faster to load then re-verifying and re-loading all the individual classes, and once in memory it's shared by all JVMs on the system. Note that -Xshare:off, -Xshare:on, -Xshare:auto set whether "Class Data Sharing" is enabled, and it's not available on the -server VM or on 64-bit systems. Mac users: you're already using Apple's version of this feature, upon which Hotspot's version is based.
There are also some basic flags for logging runtime information:
  • -verbose:gc logs garbage collector runs and how long they're taking. I generally use this as my first tool to investigate if GC is a bottleneck for a given application.
  • -Xprof turns on a low-impact sampling profiler. I've had Hotspot engineers recommend I "don't use this" but I still think it's a decent (albeit very blunt) tool for finding bottlenecks. Just don't use the results as anything more than a guide.
  • -Xrunhprof turns on a higher-impact instrumenting profiler. The default invocation with no extra parameters records object allocations and high-allocation sites, which is useful for finding excess object creation. -Xrunhprof:cpu=times instruments all Java code in the JVM and records the actual CPU time calls take. I generally only use this to profile JRuby internals because it's extremely slow, but it's also much more accurate than -Xprof.
Deeper Magic

Eventually you may want to tweak deeper details of the JVM:
  • -XX:+UseParallelGC turns on the parallel young-generation garbage collector. This is a stop-the-world collector that uses several threads to reduce pause times. There's also -XX:+UseParallelOldGC to use a parallel collector for the old generation, but it's generally only useful if you often have large numbers of old objects getting collected.
  • -XX:+UseConcMarkSweepGC turns on the concurrent mark-sweep collector. This one runs most GC operations in parallel to your application's execution, reducing pauses significantly. It still stops the world for its compact phase, but that's usually quicker than pausing for the whole set of GC operations. This is useful if you need to reduce the impact GC has on an application run and don't mind that it's a little slower than the full stop-the-world versions. Also, you obviously would need multiple processors to see full effect. (Incidentally, if you're interested in GC tuning, you should look at Java SE 6 HotSpot Virtual Machine Garbage Collection Tuning. There's a lot more there.)
  • -XX:NewRatio=# sets the desired ratio of "new" to "old" generations in the heap. The defaults are 1:12 in the -client VM and 1:8 in the -server VM. You often want a higher ratio if you have a lot more transient data flowing through your application than long-lived data. For example, Ruby's high object churn often means a lower NewRatio (i.e. larger "new" versus "old") helps performance, since it prevents transient objects from getting promoted to old generations.
  • -XX:MaxPermSize=###M sets the maximum "permanent generation" size. Hotspot is unusual in that several types of data get stored in the "permanent generation", a separate area of the heap that is only rarely (or never) garbage-collected. The list of perm-gen hosted data is a little fuzzy, but it generally contains things like class metadata, bytecode, interned strings, and so on (and this certainly varies across Hotspot versions). Because this generation is rarely or never collected, you may need to increase its size (or turn on perm-gen sweeping with a couple other flags). In JRuby especially we generate a lot of adapter bytecode, which usually demands more perm gen space.
And there are a few more advanced logging and profiling options as well:
  • -XX:+PrintCompilation prints out the name of each Java method Hotspot decides to JIT compile. The list will usually show a bunch of core Java class methods initially, and then turn to methods in your application. In JRuby, it eventually starts to show Ruby methods as well.
  • -XX:+PrintGCDetails includes the data from -verbose:gc but also adds information about the size of the new generation and more accurate timings.
  • -XX:+TraceClassLoading and -XX:+TraceClassUnloading print information class loads and unloads. Useful for investigating if you have a class leak or if old classes (like JITed Ruby methods in JRuby) are getting collected or not.
Into The Belly

Finally here's a list of the deepest options we use to investigate performance. Some of these require a debug build of the JVM, which you can download from java.net.

Also, some of these may require you also pass -XX:+UnlockDiagnosticVMOptions to enable them.
  • -XX:MaxInlineSize=# sets the maximum size method Hotspot will consider for inlining. By default it's set at 35 *bytes* of bytecode (i.e. pretty small). This is largely why Hotspot really like lots of small methods; it can then decide the best way to inline them based on runtime profiling. You can bump it up, and sometimes it will produce better performance, but at some point the compilation units get large enough that many of Hotspot's optimizations are skipped. Fun to play with though.
  • -XX:CompileThreshold=# sets the number of method invocations before Hotspot will compile a method to native code. The -server VM defaults to 10000 and -client defaults to 1500. Large numbers allow Hotspot to gather more profile data and make better decisions about inlining and optimizations. Smaller numbers reduce "warm up" time.
  • -XX:+LogCompilation is like -XX:+PrintCompilation on steroids. It not only prints out methods that are being JITed, it also prints out why methods may be deoptimized (like if new code is loaded or a new call target is discovered) and information about which methods are being inlined. There's a caveat though: the output is seriously nasty XML without any real structure to it. I use a Sun-internal tool for rendering it in a nicer format, which I'm trying to get open-sourced. Hopefully that will happen soon. Note, this option requires -XX:+UnlockDiagnosticVMOptions.
And finally, my current absolute favorite option, which requires a debug build of the JVM:
  • -XX:+PrintOptoAssembly dumps to the console a log of all assembly being generated for JITed methods. The instructions are basically x86 assembly with a few Hotspot-specific instruction names that get replaced with hardware-specific instructions during the final assembly phase. In addition to the JITed assembly, this flag also shows how registers are being allocated, the probability of various branches being followed (along with multiple assembly blocks for the different paths), and information about calls back into the JVM. Outside the logging options for the final generated assembly (which requires a separate plugin) this is the best tool for discovering what optimizations are actually happening. I use this at least a couple times a week to investigate JRuby performance enhancements.
And So Much More

Hotspot has literally hundreds of different flags (and here's another list specific to Java 6), and dozens of them that might be useful to you. I may add a few more to this post as I remember them, but this list includes all those I use on a regular basis. If you're using JRuby, you can use the -J flag to pass any of these flags through to the JVM, as in -J-XX:+PrintCompilation.

What are some of your favorite Hotspot JVM flags?

Update: Another couple that commenters added or reminded me of:
  • Marcus Kohler commented on -XX:+HeapDumpOnOutOfMemoryError, useful if you have a slow-leaking application you can't pin down. It will dump heap information to disk whenever there's an OutOfMemoryError, allowing you to do offline analysis.
  • j6wbs mentioned that you can send SIGQUIT (or hit Ctrl+Backslash or Ctrl+Break in the console) to dump the current execution stack of all running threads. This is especially nice if you have a runaway app or if an app appears to have frozen.
  • karld offers up -XX:OnOutOfMemoryError="mail -s 'OOM on `hostname` at `date`' whoever@example.com <<< ''" as a way to send out email when there's an OutOfMemoryError. Poor-man's monitoring!
  • I also remembered a very important option for JRuby: -Xbootclasspath specifies classpath entries you want loaded without verification. The JVM verifies all classes it loads to ensure they don't try to dereference an object with an int, pop extra entries off the stack or push too many, and so on. This verification is part of the reason why the JVM is very stable, but it's also rather costly, and responsible for a large part of startup delay. Putting classes on the bootclasspath skips this cost, but should only be used when you know the classes have been verified many times before. In JRuby, this reduced startup time by half or more for a simple script. Use -Xbootclasspath/a: and -Xbootclasspath/p: to append and prepend to the default bootclasspath or -Xbootclasspath: to completely set your own.

Tuesday, December 23, 2008

Irish Java Technologies Conference

I've been a bit quiet on the blog over the past couple months, and for that I apologize. JRuby is moving very quickly now, and we'll have some pretty big announcements soon. For now, I figured I'd let my friends in Europe and the British Isles know I'll be in your back yard again soon!

I'm going to be speaking at the Irish Java Technologies Conference in January. The conference is the 7th and 8th, and I'll be doing two talks.

The first is going to be a talk on JRuby, with all the usual trimmings to show Ruby's power and the strength of the Ruby community and ecosystem at large. I'll give a brief intro to Ruby, and then do a few live-coded demonstrations of Ruby calling Java libraries.

The second talk will be on the future of the JVM and the Java platform in light of this renewed interest in alternative languages. I'll be discussion the challenges of implementing a language like JRuby, walking through some of the work we've done to make JRuby the fastest Ruby implementation. Then we'll dive into the future of languages like JRuby on the JVM, going through upcoming changes in Java 7 and talking about how they'll affect the average developer on the Java platform.

The second talk will be the first of it's kind I've ever presented, but it's a topic very dear to me. I think we have a responsibility to ensure that the JVM, especially OpenJDK, become the VM and platform of choice for all kinds of application development, and our work on JRuby is directly related to that goal. What we've learned from JRuby, and what others have learned implementing their languages, will directly affect the future of the platform.

So, if you're anywhere near Dublin around 7th-8th January, come on by the conference!

Speaker IJTC 2008

Wednesday, December 3, 2008

JavaOne and CommunityOne 2009

It's that time of year again, and the call for papers for JavaOne and CommunityOne is closing soon! We're hoping for a lot more diverse JRuby presentations and speakers this year, so don't think you have to be doing Rails to get in. There's room for GUI dev, Game dev, automation, services, anything you might be doing with JRuby. So don't miss the deadline, and consider submitting a couple talks if you have a couple good topics!

CommunityOne
CommunityOne East - March 18-19, 2009 - New York City
CommunityOne West - June 1-2, 2009 - San Francisco
Deadline to submit speaking abstracts: Dec. 11, 2008
Where to make submissions: Click Here
Event Details (external): Click Here
Questions/Inquiries: communityone-info@sun.com

JavaOne
JavaOne - June 2-5, 2009 - San Francisco
Deadline to submit speaking abstracts: December 19, 2008
Where to make submissions: Click Here
Even Details (external): Click Here
Questions/Inquiries: j1papers@sun.com

Submit early! Submit often!

Sunday, November 23, 2008

Noise Cancelling

Short thoughts on KirinDave's post The Opposite of Momentum, which the anti-Ruby crowd has latched on to as more evidence that Ruby is "falling flat on its face".

Dave's post appears to largely lament the lack of progress in the C implementations, be it their lack of performance, presence of memory leaks, or relatively primitive GC implementations. He briefly mentions alternative implementations, calling Rubinius "the light and hope of the Rubyverse" and summarizing JRuby as "great" before dismissing both.

In Rubinius's case, he may have a point. There have been many setbacks, and after a year of funding six developers it's still not really usable for real apps. The sad news that their team has been cut to two paid developers certainly doesn't help. Most will agree that the Rubinius ideal, that of Ruby implemented in Ruby, is a great goal...perhaps the highest ideal for a Ruby implementation. But in practice, it's turned out to be a lot harder to execute...and impossible to execute in a year with six smart hackers.

Update: My intent here seems to have been missed. I am not in any way claiming "Rubinius sucks", because Rubinius isn't even finished yet. I would not make such a claim about any incomplete project. I mention Rubinius only because Dave dismissed both in the same breath; to address one case I felt I needed to address the other. Rubinius will succeed...of that I have no doubt. And already it has proven many things possible nobody ever expected to see. It's unknown how long it will take to "get there" but it will "get there" eventually, and be a major contender for replacing the standard impls. Dave's use of Rubinius to help prove a lack of momentum is almost as questionable as his out-of-hand dismissal of JRuby. We've started to attempt an integration of Rubinius's kernel into JRuby, and we've added Rubinius's FFI and MVM APIs because they were very well-designed. We continue to contribute to the RubySpecs, and I eagerly offer any technical help I can whenever the Rubinius team has questions about how JRuby solves certain problems. What more good things do you want me to say about Rubinius?

However when we look at JRuby, Dave is entirely wrong to dismiss it in a single sentence. JRuby *is* what he wants, in almost every way.

  • JRuby runs on the best available dynamic language VM, Hotspot. Hotspot has its roots in the Smalltalk world Rubyists hold in such high esteem, and real steps are being taken to make it even better than before for dynamic languages. The end result will be trivially inlinable dynamic calls nearly as fast as static Java code, which no other VM in the world can claim.
  • JRuby does not have GC headaches and memory leaks like the C impls because we use the existing GC options on Hotspot...which are, again, some of the best GC implementations in the world. It's folly to say that an implementation from scratch is going to be able to compete with JRuby on JVM, because the simple truth is that making a world-class VM and/or GC is super dooper hard.
  • Even where JRuby still stumbles a bit, like the classic JVM problem areas of memory use and startup time, we've made huge strides. JRuby starts up no slower than Rubinius at its fastest...in many cases under a second. And memory-wise, we pay little cost beyond the JVM's own 20-30MB memory tax, while still working to reduce our overall consumption with every release.
  • JRuby is also moving faster than any implementation available. A year ago, we were generally a bit slower than Ruby 1.8.6; this year, we're faster in most cases than Ruby 1.9. A year ago, there were a handful of early-adopting production users; this year, dozens of them, ranging from small orgs and agile projects to governments, banks, and telecom. There's no lack of momentum when it comes to JRuby.
  • To top all this off, there's still dozens of folks working to make Hotspot even faster, it's GCs even smarter, and we just come along for the ride. Yes, we've done a lot because we haven't implemented our own VM specifically for Ruby. But that's exactly the point...we've been able to focus on areas that matter most, like actually running apps and running them as fast as possible.

The bottom line, as far as I can see it, is that if the C impls of aren't moving fast enough, and the "light of the Rubyverse" is hitting a few roadblocks, you damn well better give JRuby more consideration than one sentence. I certainly hope this wasn't a reflection of any continuing anti-Java bigotry in the Ruby world, since as I've mentioned previously that attitude will only make things worse. If it reflects simple ignorance as to why JRuby really will be a better Ruby, then we're not doing a good enough job educating...help us do better.

JRuby is continuing to do things no other Ruby has been able to do. We've cracked the Enterprise glass ceiling, with numerous deployments inside some of the most rigidly Java-centric and intransigent organizations around. We've shown it's possible for Ruby to perform extremely well, and we've only scratched the surface of Hotspot's potential. We've implemented most of Ruby 1.9's features in just a few months, where other implementations have barely gotten 1.8 working. And we're just getting started...we have plans upon plans to continue improving compatibility, performance, scaling, and ease-of-use, and the various folks working on the JVM will continue making it a better and better host for dynamic languages. If you haven't used JRuby yet, you're really missing out. It's time to give it a try.

(Footnote: Don't take this as a shot at any of the other impls; it's obvious JRuby had a big head start, and we've been funded longer (albeit less) than most of them. This post is squarely focused on anyone like Dave that thinks if the C impls don't solve all Ruby's implementation woes, Ruby is doomed. The truth is that Ruby will survive despite any lingering issues in the C impls, and JRuby is a crucial part of that survival.)