Packaging Varnish VMODs

Varnish (the caching HTTP reverse proxy) had an extension mechanism called Varnish Modules (or VMODs) since it’s version 3.0. The official listing on the homepage lists a bunch of production-ready open-source VMODs which are quite useful for a number of situations e.g.

• libvmod-throttle: rate limiting access based on (relatively) arbitrary keys (e.g. derived from IP addresses) and (somewhat) arbitrary windows.
• libvmod-boltsort: sort or ignore the order of arguments in the query string, e.g. treat a request to /action?a=b&c=d the same as /action?c=d&a=b.

VMODs are typically distributed as source code on github.com and require a compile against the same version of source code the varnishd binary was build from. After a successful make the build files have to be copied into the lib/varnish/vmod directory and can be imported from there using e.g. import throttle; in your configuration files. Since this adds a little bit of complexity when deploying a Varnish configuration which relies on VMODs, we are having a closer look at what exactly is necessary to deliver a VMOD as an easy to install rpm.

Most VMODs follow the structure established by the libvmod-example (demo) project which means we can build a similar template for packaging such VMODs. We’re using the fabulous fpm gem for the purpose. The basic steps are (illustrated below with the before mentioned VMODs):

1. Install some prerequisites (Ruby is assumed to be installed on the build machine though)
2. Fetch a copy of Varnish from github and pick an appropriate version (tag)
3. Build Varnish
4. Fetch a copy of the VMOD from github and pick an appropriate version (tag)
5. Build the VMOD
6. Package the VMOD artefacts using FPM
7. (not depicted here) Upload the package to some storage or repository

Please note that the build script does not cross-compile. In order to build an appropriate RPM you either need to run the build inside a matching virtual machine (we use Vagrant for this purpose) or integrate this into you CI environment of choice (we favor Travis Pro and Atlassian Bamboo). More on that in a later post!

So without further ado, the script for building libvmod-throttle:

    VARNISH_VERSION=3.0.3

    VMOD_REPOSITORY=https://github.com/nand2/libvmod-throttle.git
    VMOD_NAME=libvmod-throttle
    VMOD_VERSION='0.1'
    VMOD_LICENSE='Simplified BSD License'
    VMOD_VENDOR='Nicolas Deschildre'
    VMOD_DESCRIPTION='Rate limiting in Varnish, on different time windows and keys'
    sudo yum install -y pcre-devel python-docutils
    sudo gem install fpm

    pushd /tmp

    if [ ! -d varnish ]; then
      git clone --progress --recursive git://github.com/varnish/Varnish-Cache.git varnish
    fi

    cd varnish

    git fetch
    git checkout varnish-$VARNISH_VERSION

    sed -i '' 's/AM_CONFIG_HEADER/AC_CONFIG_HEADERS/g' configure.ac

    make clean
    nice ./autogen.sh
    nice ./configure
    nice make

    cd /tmp

    if [ ! -d $VMOD_NAME ]; then
      git clone --progress --recursive $VMOD_REPOSITORY $VMOD_NAME
    fi

    cd $VMOD_NAME

    git fetch
    git checkout master

    mkdir -p build

    export VARNISHSRC=/tmp/varnish
    export VMODDIR=/tmp/$VMOD_NAME/build

    sed -i '' 's/AM_CONFIG_HEADER/AC_CONFIG_HEADERS/g' configure.ac

    make clean
    nice ./autogen.sh
    nice ./configure
    nice make
    make install

    cd build

    fpm -f \
        -t rpm \
        -s dir \
        --prefix=/usr/lib64/varnish/vmods \
        -n $VMOD_NAME \
        -v $VMOD_VERSION \
        --iteration `git rev-parse --short HEAD` \
        --epoch `date +"%s"` \
        -a noarch \
        -d 'varnish' \
        --license "$VMOD_LICENSE" \
        --vendor "$VMOD_VENDOR" \
        --description "$VMOD_DESCRIPTION" \
        --url "`echo $VMOD_REPOSITORY | sed 's/\.git$//'`" \
        *

    popd

    mv -f $VMODDIR/*.rpm .

And the (basically) same for libvmod-boltsort:

    VARNISH_VERSION=3.0.3

    VMOD_REPOSITORY=https://github.com/vimeo/libvmod-boltsort.git
    VMOD_NAME=libvmod-boltsort
    VMOD_VERSION='1.0.0'
    VMOD_LICENSE='Apache License 2.0'
    VMOD_VENDOR='Vimeo, LLC'
    VMOD_DESCRIPTION='Sort query strings in Varnish'

    sudo yum install -y pcre-devel python-docutils
    sudo gem install fpm

    pushd /tmp

    if [ ! -d varnish ]; then
      git clone --progress --recursive git://github.com/varnish/Varnish-Cache.git varnish
    fi

    cd varnish

    git fetch
    git checkout varnish-$VARNISH_VERSION

    sed -i '' 's/AM_CONFIG_HEADER/AC_CONFIG_HEADERS/g' configure.ac

    make clean
    nice ./autogen.sh
    nice ./configure
    nice make

    cd /tmp

    if [ ! -d $VMOD_NAME ]; then
      git clone --progress --recursive $VMOD_REPOSITORY $VMOD_NAME
    fi

    cd $VMOD_NAME

    git fetch
    git checkout master

    mkdir -p build

    export VARNISHSRC=/tmp/varnish
    export VMODDIR=/tmp/$VMOD_NAME/build

    sed -i '' 's/AM_CONFIG_HEADER/AC_CONFIG_HEADERS/g' configure.ac

    make clean
    nice ./autogen.sh
    nice ./configure
    nice make
    make install

    cd build

    fpm -f \
        -t rpm \
        -s dir \
        --prefix=/usr/lib64/varnish/vmods \
        -n $VMOD_NAME \
        -v $VMOD_VERSION \
        --iteration `git rev-parse --short HEAD` \
        --epoch `date +"%s"` \
        -a noarch \
        -d 'varnish' \
        --license "$VMOD_LICENSE" \
        --vendor "$VMOD_VENDOR" \
        --description "$VMOD_DESCRIPTION" \
        --url "`echo $VMOD_REPOSITORY | sed 's/\.git$//'`" \
        *

    popd

    mv -f $VMODDIR/*.rpm .