Beanstalk

Beanstalk + Clojure = Love (and 20x better performance)

Posted on May 29, 2012 by Dima Sabanin Dima Sabanin

I try to only post on the Beanstalk blog when I have something cool to post about and today is the perfect occasion. Yesterday we deployed to production a rewrite of Beanstalk’s caching system written in Clojure. It is 20 times faster than the previous version that was written in Ruby and allowed us to bring the latency between committing a change to a repo and seeing the update in Beanstalk’s UI to an average of 20 ms.

Here I’ll try to explain how and why we did it. But first let me explain the Beanstalk caching architecture.

Beanstalk Caching Architecture

When a commit or a push is made to a repository hosted on Beanstalk it first goes directly to the repository on our high availability storage servers. This way we make sure that your commits are safe and that if any error happens, your version control client will let you know immediately.

The next step is commit caching, when we take the commit information from the repo and put in the database. This is to make sure we don’t need to hit the disks every time we need a tiny bit of information about your files or commits. This greatly improves your general Beanstalk experience — from the web UI to email notifications to deployments.

After the commit is cached it is then transferred further to trigger email notifications, various integrated services, automated deployments and update various stats in your account.

To fetch commit information from the repos in Beanstalk we use several different APIs — for each of the version control systems we support:

  • Official Subversion Ruby bindings (which bind Ruby with Subversion’s C library)
  • Grit for Git (which is a mix of pure Ruby code with bits of Git command line client output parsing)
  • Our own Ruby gem for Mercurial

So What’s The Problem?

The problem was that commit caching was frequently quite slow. For Subversion repositories big commits could easily take 30 seconds to cache. If a new Subversion repository was imported into Beanstalk, it could take up to 10 minutes to cache.

For Git the situation was even worse, since each push can potentially represent thousands of commits it could easily take 30 minutes to cache it. If your Git repo has a lot of branches, then it could take even more time. Now if you imagine that we have to process thousands commits every hour, this could easily become a problem for our servers and sometimes it was.

I probably need to explain why Git is much slower for us — it’s because of the way we represent Git activity timeline. We think that all commits to different branches should be viewable together in a single timeline, so we have to perform an extra bit of caching to get additional information about branches structure and what commit was pushed to which branch first (even though Git doesn’t provide this information easily).

Basically, our problem was caused by the fact that we relied on some not very efficient algorithms to build our caches (due to the API restrictions of SVN bindings and Grit library) and the fact that we used Ruby in such a critical place for Beanstalk’s performance.

Clojure to the Rescue

I’ve been looking for an excuse to use Clojure in a production environment for a while. If you don’t know what Clojure is — it’s a Lisp dialect that runs on a JVM. You get the best of the both worlds - amazing power and clarity of a modern Lisp and huge variety of production ready Java libraries out there.

Clojure is an awesome language that brings Lisp into the 21st century. It provides great support for concurrency out of the box, it’s a functional language with a lot of things done right and it has a very vibrant community with lots of extremely smart people. The fact that it integrates seamlessly with Java allows you to use all the code from the Java world without writing a line of Java.

So the best thing is that with some optimizations (like optional type hints) Clojure can be made to run almost as fast as native Java. But even without any optimizations Clojure code is easily 30-50 times faster than regular Ruby code on artificial benchmarks, and that sounded like a great plan to me!

There are also some amazing libraries for version control system for Java that I could use through Clojure right away:

  • SVNKit for Subversion (Used by Oracle, IBM, Novell, Borland)
  • JGit used by Eclipse

There’s also plenty of great libraries for Clojure and many more appear each day. All this made Clojure a great candidate for a second language to be added to the Beanstalk team’s toolbox.

The Results

The rewrite in Clojure resulted in much cleaner, faster code, totaling at only 700 lines of Clojure code (I don’t have a clear comparison with Ruby code here). Some of the features of our new caching daemon are:

  • It’s very fast. Subversion caching is more than 15-20 times faster than our previous Ruby version. Git caching is 50 times faster than before.
  • It is multi-threaded so it doesn’t spawn processes when new job comes in, so it’s very fast at catching new commits. It’s also more efficient and we’ll be able to process more commits with it.
  • It’s deployed as a single JAR file that we can run on any of our machines that are connected to the HA file storage. That will allow us to scale our commit caching capacity in matter of minutes on spare machines.
  • Very low latency - it starts caching new commits before you press that commit button :).

What Does This Mean To Our Customers?

  • Commits appearing in timeline nearly instantly.
  • Deployments will be triggered faster.
  • Generally better caching as Clojure is not having problems with commits that were “too large” to cache for our Ruby version.
  • More consistent caching results, no unicode issues even in weird cases.

Future Plans

We plan to improve our caching even more by running the caching daemon on our backup storage servers, which will make commit caching even faster as we would bypass NFS latency.

Our experience of using Clojure for real world stuff was a very positive one and we look forward to using more Clojure in Beanstalk infrastructure. Who knows, maybe someday we’ll be able to speed up our deployments even further by using Clojure.

Interested in Clojure?

If you are interested in learning more about Clojure here are some great resources:

Here are also some good books on the subject: Clojure in Action and Joy of Clojure from Manning, Clojure Programming from O’Reilly and Programming Clojure (2nd edition) from Pragmatic Programmers.

TL;DR: Clojure is awesome, Beanstalk commit caching is now 20-40 times faster.

  1. beanstalkapp posted this