Nearly a year ago to the day, my freind and coleague Brian O’Neill blogged about building storm on OSX. I had been through that pain two years ago, and largely forgot about it (once you get 0mq and JZMQ installed you’re largely in the clear). That is until today, when I had to set up a storm development environment on a new laptop…
Things have changed since then. Apple has released OSX 10.9 (Mavericks) and turned over development of the OSX JDK to Oracle, adding a little more salt to the wound.
Hopefully this will spare a few others from that pain.
On a fresh install of OSX 10.9 (no JDK installed), when you
java -version from the command line you will get a prompt to download Java 7 from Oracle. But what if you need JDK 6?
Some Java applications can trigger the install of JDK 6 via software update. In my case it was Intellij IDEA. But one utility that Apple added to OSX was the
/usr/libexec/java_home executable, which will output a path suitable for use as
JAVA_HOME value. Without arguments, it will output the path for the default JDK:
1 2 3 4 5 6 7
You can also specify a certain version (running this should trigger an install of JDK 6 via software update – I’m not sure because Intellij had already triggered it in my situation):
1 2 3 4 5 6 7
So now we have a way to easily switch between JDK 6 and 7.
If you need to download Java 6 manually, there is a
.dmg file available from Apple.
Install Xcode and Command Line Tools
Xcode is easy to install via the App Store. You will also need the command line tools for compiling various things. You can trigger the install of the command line tools by invoking one of the stubs of the tools included, e.g.:
Homebrew is like a package manager for OSX that handles download, compilation, and installation of various tools and libraries. We will use it later to install dependencies.
brew doctor command completes without error, you’re ready to move on.
The Storm documentation recommends using 0mq version 2.1.7, however I’ve found that 2.1.7 will not build on OS X 10.9 (Mavericks). The closest version I’ve found that will build under Mavericks is 2.1.9.
The new/preferred method is to use
brew tap homebrew/versions, however that repository does not have 0mq 2.1.9. The fallback method is to use the
brew versions command to find and checkout a specific version:
1 2 3 4
Install JZMQ (Java bindings for 0mq)
Compiling and instlling JZMQ is where most of pain comes in, and most of the errors that come out of the build process are cryptic.
Follow the steps below exactly and you should be spared that pain:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
libtool are dependencies of the JZMQ build process. After installing those, we switch to JDK 6. Next we symlink the JDK 6 header directory where the JZMQ build process will be able to find it. Without it you will get an error like:
touch src/classdist_noinst.stamp command prevents the following error:
Next, we manually compile the JZMQ java sources. Without that step, you will get the following error:
1 2 3 4 5 6
Finally, we compile and install JZMQ.
Leiningen is the build tool used by storm. It is similar to Maven and Gradle, but much more suited to clojure development.
Installing Leiningen is just a matter of downloading the script and making it executable:
The above commands assume you have a
~/bin directory and it is added to your
PATH. The easiest way to do that is in a
Now we’re finally ready to build storm:
1 2 3
This will build and install the storm jars so they can be used as dependencies in other Leiningen/Maven/Gradle/etc. projects. If you want to create a distribution ZIP archive, run the following script:
Hopefully this will save some future storm contributors some time Googling cryptic errors.