Spotlight on: SpringSource

For this week’s user spotlight segment, I’m talking with Doug MacEachern of Hyperic, part of SpringSource, a division of VMware, hoping I got that dependency chain correct. Hyperic builds enterprise systems monitoring and management software and also contributes to a number of open source projects, many of which are built with Hudson.

To date I must say that Doug’s use of Hudson is one of the largest and more impressive installations I’ve seen. I don’t want to spoil the interview, but they’re testing on platforms that don’t even run Java. Madness! If you think you can out-do him, you can find my email information at the bottom of the interview, I’d love to hear about it!

Without further ado, Doug from SpringSource.

DougWe started using Hudson in early 2008 to automate the build and testing of our SIGAR library. The SIGAR API implements a portable interface in C for gathering system information related to memory, processors, file systems, network interfaces, network connection tables, the process table and more. We support dozens of OS + version + architecture combinations, along with several language bindings. SIGAR is a key component of the Hyperic HQ agent and is used in other projects including Hypertable, Terracotta, GridGain and MySQL enterprise.

HudsonWas SpringSource using continuous integration before Hudson? If so, what caused you guys to switch?

DougThe SIGAR project actually started back in late 2002 and our initial CI system for the project was a good old-fashioned Perl script / ssh for-loop. It was good enough to get by in the early years, but a proper replacement was long overdue. We were (and still are) using Bamboo to build and test Hyperic HQ. We looked at using Bamboo for SIGAR, but at the time the "Remote Agent" feature was new to Bamboo and was not in the version we were running. Rather than disrupt HQ’s CI along with taking on an additional licensing cost, we gave Hudson a shot and haven’t looked back.

HudsonMight be a bit of personal bias, but I think you guys made the right choice there! Checking out the public Hudson server, I see that SpringSource is building/testing products on AIX, the BSDs, various flavors of Linux, Solaris, Windows and Mac OS X, what kinds of languages/build systems are being built by Hudson? How varied are the environments that Hudson executes jobs in?

DougAnd HP-UX! The matrix of SIGAR’s supported OS + kernel version + architecture + distribution is north of 100 combinations. So, Hudson is covering a very heterogeneous collection of systems with most jobs tied to a specific node. Our primary focus has been the C API and Java JNI bindings, using an Ant based build system and a JUnit test suite. SIGAR also has language bindings for Perl, Ruby, Python, Erlang, PHP, C# and Lua. So, Hudson is also driving each language’s extension build system of choice, respectively: MakeMaker, Rake, distutils, emake, phpize, Nant and autotools.

HudsonWhat do you consider to be noteworthy about your Hudson implementation? Besides, clearly, that you’re running Hudson agents on just about every OS that will run Java :)

Doug

The majority of our x86/x64 nodes are virtualized on VMware ESX and VMware Server. We also have a fine collection of PPC, PA-RISC and Sparc hardware in house, with IA-64 and s390x hosted elsewhere by third parties. Some of these systems are too old to support Java 1.5 and/or Git. As a simple work-around, the nodes share an NFS workspace where the agent node takes care of 'SCM' and 'Post-build Actions', but the 'Build' step in between is invoked via ssh.

The SIGAR distribution includes about two dozen native binaries that are compatible with most of the supported platform matrix. There’s a Hudson job for each Git branch that rolls these binaries into a release bundle. Another job flavor uses the Hudson URL SCM plugin to download and unit test the binary releases on the rest of the platform matrix. This is key to testing binary compatibility. Similar for the collectd project, each Git branch has a job that runs automake, autoconf, etc. and 'make dist' into the collectd release flavor tarball. So a push to git.verplant.org by octo in Germany triggers an update of the collectd release artifact, which in turn triggers the URL SCM jobs to download the tarball, unpack and build over here at our west coast locations.

We have four Hudson servers in different locations, three of which are managing most of the jobs behind firewalls. Select jobs use the Build Publisher plugin to post the job and its artifacts to our public Hudson server. This makes it easy for us to provide platform specific bug fixes in binary form, share build logs with external projects and host a central repository of artifacts reachable by all of the URL SCM based jobs. Our public Hudson server also provides CI for the HQApi project and jobs to build HQ plugins, again making it easier to distribute patch fixes in binary form between releases. </td></tr>

HudsonI’ve very impressed! I’m glad the fact that Java won’t run on some of the platforms you want to support hasn’t stopped you from testing anyways. Clearly you folks have written some addition tools behind the scenes, mind discussing them a bit?

DougOther than some Hudson plugin tweaks and additions, the Perl script I mentioned earlier was converted to generate the majority of our Hudson jobs and includes a simple templating system. The same script generates jobs to build collectd and a few other projects. We’ve outgrown this flavor of the script and have started working on integrating Opscode Chef to automate our Hudson configuration along with the systems we build and test on. And of course, we’re using Hyperic HQ to monitor our Hudson server instances, agent and node machines.

HudsonBut of course, I’d say dog-fooding is an important part of any continuous testing set up. It appears that SpringSource has bought in pretty deeply to a Hudson-oriented workflow, given the amount of time and resources you all have invested in getting the massive farm set up that you have. That said, on a scale from 1-10, how important would you rate Hudson to your day-to-day workflow?

DougI’d say at least an 8, although my daily workflow doesn’t always directly involve Hudson. Most of those points go to Hudson for automating what otherwise would be interrupting my workflow on a daily basis.

</table>

Editor’s note: Doug was the primary author of mod_perl for many years until he was tricked into "helping out" with a new project. This project turned into Hyperic HQ which shifted his focus to systems and application management for the past ~7 years and counting. He occasionally rambles on Twitter as @dougmaceachern.