Packaging Varnish VMODs

12.06.2013
Joern Barthel

Varnish (the caching HTTP reverse proxy) had an extension mechanism called Varnish Modules (or VMODs) since it's version 3.0. The official listing on the homepage lists a bunch of production-ready open-source VMODs which are quite useful for a number of situations e.g.

VMODs are typically distributed as source code on github.com and require a compile against the same version of source code the varnishd binary was build from. After a successful make the build files have to be copied into the lib/varnish/vmod directory and can be imported from there using e.g. import throttle; in your configuration files. Since this adds a little bit of complexity when deploying a Varnish configuration which relies on VMODs, we are having a closer look at what exactly is necessary to deliver a VMOD as an easy to install rpm.

Most VMODs follow the structure established by the libvmod-example (demo) project which means we can build a similar template for packaging such VMODs. We're using the fabulous fpm gem for the purpose. The basic steps are (illustrated below with the before mentioned VMODs):

  1. Install some prerequisites (Ruby is assumed to be installed on the build machine though)
  2. Fetch a copy of Varnish from github and pick an appropriate version (tag)
  3. Build Varnish
  4. Fetch a copy of the VMOD from github and pick an appropriate version (tag)
  5. Build the VMOD
  6. Package the VMOD artefacts using FPM
  7. (not depicted here) Upload the package to some storage or repository

Please note that the build script does not cross-compile. In order to build an appropriate RPM you either need to run the build inside a matching virtual machine (we use Vagrant for this purpose) or integrate this into you CI environment of choice (we favor Travis Pro and Atlassian Bamboo). More on that in a later post!

So without further ado, the script for building libvmod-throttle:

VARNISH_VERSION=3.0.3

VMOD_REPOSITORY=https://github.com/nand2/libvmod-throttle.git
VMOD_NAME=libvmod-throttle
VMOD_VERSION='0.1'
VMOD_LICENSE='Simplified BSD License'
VMOD_VENDOR='Nicolas Deschildre'
VMOD_DESCRIPTION='Rate limiting in Varnish, on different time windows and keys'
sudo yum install -y pcre-devel python-docutils
sudo gem install fpm

pushd /tmp

if [ ! -d varnish ]; then
  git clone --progress --recursive git://github.com/varnish/Varnish-Cache.git varnish
fi

cd varnish

git fetch
git checkout varnish-$VARNISH_VERSION

sed -i '' 's/AM_CONFIG_HEADER/AC_CONFIG_HEADERS/g' configure.ac

make clean
nice ./autogen.sh
nice ./configure
nice make

cd /tmp

if [ ! -d $VMOD_NAME ]; then
  git clone --progress --recursive $VMOD_REPOSITORY $VMOD_NAME
fi

cd $VMOD_NAME

git fetch
git checkout master

mkdir -p build

export VARNISHSRC=/tmp/varnish
export VMODDIR=/tmp/$VMOD_NAME/build

sed -i '' 's/AM_CONFIG_HEADER/AC_CONFIG_HEADERS/g' configure.ac

make clean
nice ./autogen.sh
nice ./configure
nice make
make install

cd build

fpm -f \
    -t rpm \
    -s dir \
    --prefix=/usr/lib64/varnish/vmods \
    -n $VMOD_NAME \
    -v $VMOD_VERSION \
    --iteration `git rev-parse --short HEAD` \
    --epoch `date +"%s"` \
    -a noarch \
    -d 'varnish' \
    --license "$VMOD_LICENSE" \
    --vendor "$VMOD_VENDOR" \
    --description "$VMOD_DESCRIPTION" \
    --url "`echo $VMOD_REPOSITORY | sed 's/\.git$//'`" \
    *

popd

mv -f $VMODDIR/*.rpm .

And the (basically) same for libvmod-boltsort:

VARNISH_VERSION=3.0.3

VMOD_REPOSITORY=https://github.com/vimeo/libvmod-boltsort.git
VMOD_NAME=libvmod-boltsort
VMOD_VERSION='1.0.0'
VMOD_LICENSE='Apache License 2.0'
VMOD_VENDOR='Vimeo, LLC'
VMOD_DESCRIPTION='Sort query strings in Varnish'

sudo yum install -y pcre-devel python-docutils
sudo gem install fpm

pushd /tmp

if [ ! -d varnish ]; then
  git clone --progress --recursive git://github.com/varnish/Varnish-Cache.git varnish
fi

cd varnish

git fetch
git checkout varnish-$VARNISH_VERSION

sed -i '' 's/AM_CONFIG_HEADER/AC_CONFIG_HEADERS/g' configure.ac

make clean
nice ./autogen.sh
nice ./configure
nice make

cd /tmp

if [ ! -d $VMOD_NAME ]; then
  git clone --progress --recursive $VMOD_REPOSITORY $VMOD_NAME
fi

cd $VMOD_NAME

git fetch
git checkout master

mkdir -p build

export VARNISHSRC=/tmp/varnish
export VMODDIR=/tmp/$VMOD_NAME/build

sed -i '' 's/AM_CONFIG_HEADER/AC_CONFIG_HEADERS/g' configure.ac

make clean
nice ./autogen.sh
nice ./configure
nice make
make install

cd build

fpm -f \
    -t rpm \
    -s dir \
    --prefix=/usr/lib64/varnish/vmods \
    -n $VMOD_NAME \
    -v $VMOD_VERSION \
    --iteration `git rev-parse --short HEAD` \
    --epoch `date +"%s"` \
    -a noarch \
    -d 'varnish' \
    --license "$VMOD_LICENSE" \
    --vendor "$VMOD_VENDOR" \
    --description "$VMOD_DESCRIPTION" \
    --url "`echo $VMOD_REPOSITORY | sed 's/\.git$//'`" \
    *

popd

mv -f $VMODDIR/*.rpm .
Joern Barthel

Joern entwickelt, debugged und entwirft Software seit über zehn Jahren. Bei den kreuzwerkern ist er für Architektur, neue Technologien und Konzepte zuständig. Das Interesse des Diplom-Informatikers gilt dem Einsatz von funktionalen Programmiersprachen und der Architektur…

Mehr Lesen ...
comments powered by Disqus