I imagine 1.9 would perform a great deal better. Since we're scraping entirely on single-CPU machines, the GIL probably wouldn't hurt us much. We rewrote the scheduler and our interface to the HTTP client using Scala, however, so now we're pretty wedded to the JVM.

However, if we were to use Ruby 1.9, we would have to spend more time investigating HTTP libraries. From our experience, it's worth it to have workers threads NEVER block directly on IO. We saw a huge increase in reliability by having worker threads (1) send a request to an event-based client, running in its own thread and (2) block on a future with a set timeout, waiting for the callback. Having individual threads talk directly to the sockets, in either MRI or Java, wasn't a promising approach. If sockets got wedged (and they do, even using very reputable HTTP libraries) the event-based approach keeps on humming, while the blocking approach grinds to a halt.

There exist several event-based HTTP clients for Java that all work well (though some significantly better than others). Compare that with Ruby, where we couldn't find anything that mature.

That's not to mention all the concurrency primitives available in Scala, Clojure, and the underrated java.util.concurrent.

Even if Ruby 1.9 might look much more competitive to JRuby in synthetic tests of concurrent performance, in the real world the Java ecosystem features tons of libraries that helps one write reliable, predictable code quickly.