Installing GNU Octave from Source on a Heroku Dyno

This is Part 2 of a set of posts on how to hack together a buildpack for Heroku that runs GNU Octave, the open-source Matlab clone that can solve and simulate medium-scale dynamic macroeconomic models. I’m starting with a Heroku app that runs node.js, and all the terminal commands I list here are run in a one-off web dyno that I accessed via heroku run bash.

In Part 1, I discussed how to install the GNU Compiler Collection, and I will assume here that GCC is already installed and that ../gcc/bin has been added to the PATH environment variable.

Prerequisites

The official list of prerequisites can be found here. Some of these are already installed on a Heroku dyno, and some will not be needed since we are not going to use Octave’s graphics functionality. We will need to install:

• Basic Linear Algebra Subroutine library (BLAS)
• Linear Algebra Package (LAPACK)
• ARPACK, a collection of Fortran77 subroutines designed to solve large scale eigenvalue problems
• Fastest Fourier Transform in the West (FFTW3)
• GNU Linear Programming Kit (GLPK)
• GNU gperf, a perfect hash function generator
• HDF5, a program for reading Hierarchical Data Format files
• Qhull, a computational geometry library
• QRUPDATE, a QR factorization updating library
• SuiteSparse, a sparse matrix factorization library

Folder structure

Some of the libraries above will be needed just for installation, and some will be needed in the final buildpack for Octave to run. The libraries that are needed for installation only can be installed in /app/.heroku, but the libraries that will go in the buildpack should be installed inside of Octave’s install directory (which we have to make). SOmething like this will suffice:

~ $ cd /app/.heroku
~/.heroku $ mkdir octave
~/.heroku $ cd octave
~/.heroku/octave $ mkdir deps

We will use this folder /app/.heroku/octave/deps to install the libraries that should end up in the final buildpack.

BLAS and LAPACK (with ATLAS)

Instead of installing the normal BLAS and LAPACK, we’ll use an accelerated BLAS library (that will hopefully make Octave run a little faster) called ATLAS. This has to be done first, as some of the other libraries will need to refer to BLAS and LAPACK.

We’ll do this inside the ../octave/deps folder. ATLAS is a bit unusual because the install folder must be different than the source folder and the build folder. We must also have a tar-file for LAPACK saved somewhere and tell ATLAS where this is (so that it can build its own LAPACK library).

The ATLAS tar-file can be found here, and the LAPACK tar-file can be found here. The former must be scp’d onto the Heroku dyno, and the latter can be curl’d. Let’s assume that both of these files are in the ../octave/deps folder. We also need to tell ATLAS to build shared libraries with the –shared command.

~/.heroku/octave/deps $ tar -xjf atlas3.10.2.tar.bz2
~/.heroku/octave/deps $ mv ATLAS atlas-src
~/.heroku/octave/deps $ mkdir atlas-bld && mkdir atlas
~/.heroku/octave/deps $ cd atlas-bld
~/.heroku/octave/deps/atlas-bld $ ../atlas-src/configure --prefix=/app/.heroku/octave/deps/atlas --shared --with-netlib-lapack-tarfile=/app/.heroku/octave/deps/lapack-3.5.0.tgz
~/.heroku/octave/deps/atlas-bld $ make && make install

All other libraries that need to find the BLAS or LAPACK libraries should point to a specific s-file: /app/.heroku/octave/deps/atlas/lib/libtatlas.so.

ARPACK

ARPACK is thankfully maintained in GitHub these days and is fairly easy to install. We only need to clone the GitHub repository and point the configure script to our BLAS library that was installed above. The GitHub command line utilities are already installed on a Heroku dyno, making the operation very simple.

~/.heroku/octave/deps $ git clone https://github.com/opencollab/arpack-ng.git
~/.heroku/octave/deps $ mkdir arpack
~/.heroku/octave/deps $ cd arpack-ng
~/.heroku/octave/deps/arpack-ng $ sh bootstrap
~/.heroku/octave/deps/arpack-ng $ ./configure --prefix=/app/.heroku/octave/deps/arpack --with-blas=/app/.heroku/octave/deps/atlas/lib/libtatlas.so
~/.heroku/octave/deps/arpack-ng $ make && make install

Lastly, we must add the lib directory to LDFLAGS and LD_LIBRARY_PATH.

~/.heroku/octave/deps/arpack-ng $ cd ../arpack 
~/.heroku/octave/deps/arpack $ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$PWD/lib
~/.heroku/octave/deps/arpack $ export LDFLAGS="$LDFLAGS -L$PWD/lib"

FFTW

The Fastest Fourier Transform in the West library has a wicked cool name but was a big pain to figure out how to install. This must be installed twice, once with a single-precision/float version (which we will call FFTWF) and once with a double-precision version (which we will call FFTW). FFTWF must be installed in the ../octave/deps folder, while FFTW will be installelib in /app/.heroku as it is not needed in the final buildpack.

For both sets of libraries, we will need to install shared libraries with the –enable-shared option. First, I’ll go over FFTWF.

~/.heroku/octave/deps $ curl ftp://ftp.fftw.org/pub/fftw/fftw-3.3.4.tar.gz -s -O
~/.heroku/octave/deps $ tar -xzf fftw-3.3.4.tar.gz
~/.heroku/octave/deps $ mkdir fftwf
~/.heroku/octave/deps $ cd fftw-3.3.4
~/.heroku/octave/deps/fftw-3.3.4 $ ./configure --prefix=/app/.heroku/octave/deps/fftwf --enable-float --enable-shared
~/.heroku/octave/deps/fftw-3.3.4 $ make && make install

We must add the lib and include directories to LDFLAGS and CPPFLAGS, respectively.

~/.heroku/octave/deps/fftw-3.3.4 $ cd ../fftwf
~/.heroku/octave/deps/fftwf $ export LDFLAGS="$LDFLAGS -L$PWD/lib"
~/.heroku/octave/deps/fftwf $ export CPPFLAGS="$CPPFLAGS -I$PWD/include"

FFTW has a similar procedure, but in a different folder and without the –enable-float option.

~/.heroku $ curl ftp://ftp.fftw.org/pub/fftw/fftw-3.3.4.tar.gz -s -O
~/.heroku $ tar -xzf fftw-3.3.4.tar.gz
~/.heroku $ mkdir fftw
~/.heroku $ cd fftw-3.3.4
~/.heroku/fftw-3.3.4 $ ./configure --prefix=/app/.heroku/fftw --enable-shared
~/.heroku/fftw-3.3.4 $ make && make install

Once again, update the flags.

~/.heroku/fftw-3.3.4 $ cd ../fftw
~/.heroku/fftw $ export LDFLAGS="$LDFLAGS -L$PWD/lib"
~/.heroku/fftw $ export CPPFLAGS="$CPPFLAGS -I$PWD/include"

GLPK

GLPK is a very straightforward install, as you can get it from the GNU FTP server and it needs no special configuration. It can go directly into /app/.heroku. Without further comment…

~/.heroku $ curl ftp://ftp.gnu.org/gnu/glpk/glpk-4.55.tar.gz -s -O
~/.heroku $ tar -xzf glpk-4.55.tar.gz
~/.heroku $ mkdir glpk
~/.heroku $ cd glpk-4.55
~/.heroku/glpk-4.55 $ ./configure --prefix=/app/.heroku/glpk
~/.heroku/glpk-4.55 $ make && make install

Once again, update the flags.

~/.heroku/glpk-4.55 $ cd ../glpk
~/.heroku/glpk $ export LDFLAGS="$LDFLAGS -L$PWD/lib"
~/.heroku/glpk $ export CPPFLAGS="$CPPFLAGS -I$PWD/include"

GNU gperf

GNU gperf is another pretty one to install. We can put it in the /.heroku folder, and it follows the standard configure/make/install procedure. Once it has been installed, add the \bin directory to the path.

~/.heroku $ curl http://ftp.gnu.org/pub/gnu/gperf/gperf-3.0.4.tar.gz -s -O
~/.heroku $ tar -xzf gperf-3.0.4.tar.gz
~/.heroku $ mkdir gperf
~/.heroku $ cd gperf-3.0.4
~/.heroku/gperf-3.0.4 $ ./configure --prefix=/app/.heroku/gperf
~/.heroku/gperf-3.0.4 $ make && make install
~/.heroku/gperf-3.0.4 $ cd ../gperf/bin
~/.heroku/gperf-3.0.4 $ export PATH=$PATH:$PWD

HDF5

HDF5 is another very easy one. The only trick is that it must be installed in ../octave/deps as it needs to be included as part of the final buildpack.

~/.heroku/octave/deps $ curl http://www.hdfgroup.org/ftp/HDF5/current/src/hdf5-1.8.14.tar.gz -s -O 
~/.heroku/octave/deps $ tar -xzf hdf5-1.8.14.tar.gz 
~/.heroku/octave/deps $ mkdir hdf5
~/.heroku/octave/deps $ cd hdf5-1.8.14
~/.heroku/octave/deps/hdf5-1.8.14 $ ./configure --prefix=/app/.heroku/octave/deps/hdf5
~/.heroku/octave/deps/hdf5-1.8.14 $ make && make install 

As before, update the flags.

~/.heroku/octave/deps/hdf5-1.8.14 $ cd ../hdf5
~/.heroku/octave/deps/hdf5 $ export LDFLAGS="$LDFLAGS -L$PWD/lib"
~/.heroku/octave/deps/hdf5 $ export CPPFLAGS="$CPPFLAGS -I$PWD/include"

Qhull

Qhull is one of the trickier libraries to install because it doesn’t follow the standard configure/make/install procedure. Once we unzip the tar file, we’ll need to edit the Makefile to specify our install directory. I hate using whatever editor comes loaded on a Heroku dyno, so I actually did this locally and then scp’d the Makefile back onto the Heroku dyno after it was edited. We can install it in /app/.heroku.

~/.heroku $ curl http://www.qhull.org/download/qhull-2012.1-src.tgz -s -O
~/.heroku $ tar -xzf qhull-2012.1-src.tgz
~/.heroku $ mkdir qhull
~/.heroku $ cd qhull-2012.1

Note that there is no configure script here, which is why we have to edit the Makefile. Edit the Makefile on line 75 so that it reads:

DESTDIR = /app/.heroku/qhull

Now we can actually perform the installation.

~/.heroku/qhull-2012.1 $ make && make install 

In addition to adding the include directory to CPPFLAGS, we must add the lib directory to LDFLAGS and LD_LIBRARY_PATH.

~/.heroku/qhull-2012.1 $ cd ../qhull
~/.heroku/qhull $ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$PWD/lib
~/.heroku/qhull $ export LDFLAGS="$LDFLAGS -L$PWD/lib"
~/.heroku/qhull $ export CPPFLAGS="$CPPFLAGS -I$PWD/include"

QRUPDATE

QRUPDATE cannot be curl’d; it must be scp’d from here. This is another tough one, since it does not use the configure/make/make install procedure. Once you have the source code loaded on the Heroku dyno, you must edit the Makeconf file to point to our BLAS and LAPACK libraries as well as our non-standard install directory (inside of ../octave/deps).

~/.heroku/octave/deps $ tar -xzf qrupdate-1.1.2.tar.gz 
~/.heroku/octave/deps $ mkdir qrupdate
~/.heroku/octave/deps $ cd qrupdate-1.1.2

Edit the Makeconf file to point to the proper directories.

BLAS=/app/.heroku/octave/deps/atlas/lib/libtatlas.so
LAPACK=/app/.heroku/octave/deps/atlas/lib/libtatlas.so
PREFIX=/app/.heroku/octave/deps/qrupdate

Now we can go back into our install folder and tell QRUPDATE to make and install shared libraries.

~/.heroku/octave/deps/qrupdate-1.1.2 $ make solib
~/.heroku/octave/deps/qrupdate-1.1.2 $ make install

The lib directory must be added to LD_LIBRARY_PATH and LDFLAGS.

~/.heroku/octave/deps/qrupdate-1.1.2 $ cd ../qrupdate
~/.heroku/octave/deps/qrupdate $ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$PWD/lib
~/.heroku/octave/deps/qrupdate $ export LDFLAGS="$LDFLAGS -L$PWD/lib"

SuiteSparse

SuiteSparse is the last library to install. We can put it right into /app/.heroku, but it’s a bit of a pain to install (this should sound familiar by now). For starters, we must use a version less than 4.3 as the build configuration changed and will not work with Octave. SuiteSparse also does not follow the configure/make/make install procedure–we must edit a Makefile to point to our BLAS/LAPACK and install directories.

To begin with, download the source and set up the folder structure. Our install folder will be /app/.heroku/SuiteSparse, and we need to create a lib directory and include directory before we actually run the installation scripts.

~/.heroku/ $ curl http://faculty.cse.tamu.edu/davis/SuiteSparse/SuiteSparse-4.2.1.tar.gz -s -O
~/.heroku/ $ tar -xzf SuiteSparse-4.2.1.tar.gz
~/.heroku/ $ mv SuiteSparse SuiteSparse-source
~/.heroku/ $ mkdir SuiteSparse
~/.heroku/ $ cd SuiteSparse
~/.heroku/SuiteSparse $ mkdir lib && mkdir include

We must now edit a the file SuiteSparse_config/SuiteSparse_config.mk and make it point to the right directories.

INSTALL_LIB = /app/.heroku/SuiteSparse/lib
INSTALL_INCLUDE = /app/.heroku/SuiteSparse/include
BLAS = /app/.heroku/octave/deps/atlas/lib/libtatlas.so
LAPACK = /app/.heroku/octave/deps/atlas/lib/libtatlas.so

Now we can get back into our install directory and tell SuiteSparse to build and install its libraries.

~/.heroku/SuiteSparse-source $ make library
~/.heroku/SuiteSparse-source $ make install

Once again, update the flags.

~/.heroku/SuiteSparse-source $ cd ../SuiteSparse
~/.heroku/SuiteSparse $ export LDFLAGS="$LDFLAGS -L$PWD/lib"
~/.heroku/SuiteSparse $ export CPPFLAGS="$CPPFLAGS -I$PWD/include"

Configure and Install Octave

Now that all the required libraries are installed, we can finally install Octave. The source code is hosted on the GNU FTP server and we can use the normal configure/make/install process. We only need to specify where our BLAS/LAPACK libraries are, and we need to tell Octave not to install any of the graphics components (since we will be using this as a back-end computational tool with no graphics required).

We will perfrom the installation in /app/.heroku as follows.

~/.heroku/ $ curl ftp://ftp.gnu.org/gnu/octave/octave-3.8.2.tar.gz -s -O
~/.heroku/ $ tar -xzf octave-3.8.2.tar.gz
~/.heroku/ $ cd octave-3.8.2
~/.heroku/octave-3.8.2 $ ./configure --prefix=/app/.heroku/octave --disable-gui --disable-docs --disable-java --without-opengl --with-blas=/app/.heroku/octave/deps/atlas/lib/libtatlas.so --with-lapack=/app/.heroku/octave/deps/atlas/lib/libtatlas.so
~/.heroku/octave-3.8.2 $ make && make install  

The configure script will take about 10 minutes to run, and the actuall installation will take a few hours. Once it has finished up, make sure to add the bin directory to the PATH environment variable so that the system knows where to look for Octave (since it is installed in a non-standard directory).

~/.heroku/octave-3.8.2 $ cd ../octave
~/.heroku/octave $ export PATH=$PWD/bin:$PATH 

For the purposes of making a buildpack, we will want to compress the installed Octave directory into a tarball and scp it to a permanent storage space. I’ll go over how to turn this into a buildpack for Heroku in the next post.

Discussion