User Acceptance Tests from a can

Together with the IDEAS-DevOps we integrated a test tool executing the test suite using docker containers.
Martin Fleischer

Recently Simone wrote a blog post about the Celepedia launch. I participated in the test automation for this project. A large proportion of the tests, especially for the frontend, consists of UATs with Selenium. Since I already wrote a blog post about Selenium tests in general, I’m focusing on the a special part of the test setup here. Together with the IDEAS-DevOps we integrated a test tool executing the test suite using docker containers. As I already said in my former blog post user acceptance tests with Selenium getting slow really quickly. I also remarked that parallelizing the UATs helps solving this problem. Instead of showing parts of the test setup we implemented, I decided to write a tiny project to spare the project specific detail. Also the original project is written in Java, I wanted to to it more kreuzwerer-ish and use ruby instead.

Before implementing a test, there are some things need to be set up. The basic structure of the tests is pretty simple:

├── Gemfile
└── spec
    ├── features
    └── spec_helper.rb

All the dependencies are located within the Gemfile. The test are written using the Capybara framework which provides a nice syntax and neat helpers for testing the user interface. I also added the Rspec testing framework because I like the expectations syntax. The test is using PhantomJS, a headless WebKit browser that renders your Websites much quicker than a real browser. PhantomJS can easily be plugged in into Capybara using the Poltergeist gem. Thus I can use the nice Capybara DSL on top of the headless driver.

source ''

gem 'poltergeist'
gem 'rspec'
gem 'selenium-webdriver'

The features directory is the place where to add the tests. On the same level as the features directory there is the spec_helper which will be required by all of the test specs. The helper loads the frameworks and libraries I mentioned before and configures the used driver. By default capybara uses the poltergeist driver in this setup. Setting the environment variable DRIVER switches the driver slightly.

require 'rspec'
require 'capybara/rspec'
require 'capybara/poltergeist'

driver = ENV['DRIVER'] ? ENV['DRIVER'].to_sym : :poltergeist

Capybara.configure do |config|
  config.default_driver = driver
  config.javascript_driver = driver
  config.run_server = false

Now its time to write the first test. Due to my poverty of imagination I took an example from the Selenium page but implemented the example in Capybara style. It just goes to, types ‘Cheese!’ into the search bar which has the attribute name with value ‘q’ and hits the search button with name ‘btnG’. After that it ensures that the title includes ‘Cheese!’ within the next 10 seconds.

require 'spec_helper'
feature("On Google startpage") do
  scenario("user searches for Cheese") do
    visit ''
    fill_in 'q', :with => 'Cheese!'
    find(:xpath, "//'[@name='btnG']").click()
    using_wait_time 10 do
      expect(page).to have_title('Cheese!')
    sleep 1

Clearly legible in my opinion.

Now it comes to the exciting part. First I added a Dockerfile. I don’t want to go into detail on docker to much. In short docker bundles software in lightweight linux containers and makes them easily shippable. “Dockerized” applications could be imagined as software running in a virtual machine, but container virtualization boots much faster and is more resource efficient than emulating a complete OS.
The Dockerfile is kind of a blueprint for a docker image. Let’s just see what it looks like.

FROM ubuntu:14.04
MAINTAINER Martin Fleischer

configure packages using default values

ENV DEBIAN_FRONTEND noninteractive

make sure the package repository is up to date

RUN apt-get update

Install required packages

RUN apt-get install -y wget parallel ruby ruby-dev build-essential fontconfig

install phantomjs

RUN mkdir drivers
RUN wget -q -P drivers
RUN tar -C drivers -xjf /drivers/phantomjs-1.9.7-linux-x86_64.tar.bz2
RUN rm -Rf /drivers/phantomjs-1.9.7-linux-x86_64.tar.bz2
RUN ln -s /drivers/phantomjs-1.9.7-linux-x86_64/bin/phantomjs /usr/bin/phantomjs
RUN chmod 755 /usr/bin/phantomjs

install the required gems

RUN gem sources -a
RUN gem install bundler --no-ri --no-rdoc --pre
ADD ./Gemfile Gemfile
ADD ./Gemfile.lock Gemfile.lock
RUN bundle install

add the whole project

ADD . uats

The Dockerfile is based on the latest ubuntu version. First the system is gonna be updated and packages like ruby and build-essentials, required for running the test suite, are are installed. Also GNU parallel is installed, which will be used to run the tests in multiple containers concurrently. I’ll come back to this later. To get the tests running on a headless WebKit, PhantomJS is installed. I separated adding the Gemfile and installing the gems with bundler from adding the whole project. Therefore changes on the tests are separately cached and the gems are used form cache if the Gemfile didn’t change like described in Brian Morearty’s blog.

Using the Dockerfile we can build the image:

$ docker build -t parallel-docker-uats .

When the build process has finished we can check if the image has been created:

docker images

Looks good, now lets run the tests in a container.

$ docker run -ti parallel-docker-uats rspec --color

This command uses the image, creates a new container and runs the rspec command which executes the test(s) with the color option inside the container.

run single test inside the container

Now we use GNU parallel, which I mentioned before, to run the test concurrently. Wait, there is actually only a single test right now… to save some time I’ve just copied the test example 7 times.

ls spec/features/\*_spec.rb | parallel -k docker run -t parallel-docker-uats rspec --color

parallel docker uats

Doesn’t look to different compared to a normal rspec run, except from the output format:

simple rspec run

But have a look at the time rspec took. The test ran almost 3 times faster, not too bad.
I also added simple rake tasks for building the image and running the tests, which makes running the “dockerized” test suite much more comfortable.


desc "Builds the docker image for running the tests"
task :build_docker_image do
  exec "docker build -t #{@image_tag} ."

desc "Executes the test suite in parallel each test isolated in a docker container"
task :parallel_docker_tests do |t|
  exec "ls spec/\*\*/\*_spec.rb | parallel -k docker run -t #{@image_tag} rspec --color"

Now the image can be build easily by running

$ rake build_docker_image

and the whole test suite is executed using

$ rake parallel_docker_tests

Running the test in containers makes the test suite much more shippable. Combined with boot2docker a virtual machine with a nice cli management tool for Windows and OS X everyone should be able to run the suite. More than that it shouldn’t be difficult executing your test suite on a CI-Server using the docker image. When executing UATs in different environments you will notice that the environment could have strong impact on the behavior of your test. Running your test suite everywhere within containers will prevent such effects.

The big advantage of using docker containers by using virtual machines besides the shipability is ease and speed. But there are some points where running UATs in virtual machines shines. First of all, although boot2docker is a really neat solution, an environment with nested isolation technologies like running docker in a virtual machine doesn’t always bring the desired simplicity as I see it. For example using samba for sharing files between the boot2docker-vm didn’t feel too naturally for me.

Another point is that containers are only running on Linux, but maybe you want to test the browser compatibility using your UATs. The driver used for controlling the Internet Explorer obviously only runs on Windows and thus can not be tested within the docker containers.

Of course we have not been the first ones running tests in docker containers . Nick Gauthier describes a quite similar setup for running tests in containers. There is a comment for this post mentioning CircleCI offering “dockerized” testing as a service. Also I found dsgrid, an interesting Github project that runs Selenium Grid with docker. I created a public repository where you can find the source code and play around by yourself.