For our suite of Cucumber-JVM based frontend end-2-end tests we use Jenkins as continuous integration system for parallel test execution and monitoring of test results.
The manual creation and configuration of Jenkins jobs can be a tedious task especially if there are a lot of feature files and the configuration has to be changed frequently. The design decision to have one job per feature file is based on providing an easy way to run and repeat single features, keeping a separated history of stability and performance metrics and of course to be able to execute all features in parallel.
We use Gradle to automate the build and testing process because it combines the advantages of several Java build tools e.g. dependency management and a DSL which increases the readablity of build scripts (a comparison of build tools favoring Gradle too can be found here). The use of Fluentlenium supports the testers in writing Selenium tests by providing a convenient interface to the Selenium Web Driver API.
The evaluation of alternatives for ways to improve and facilitate the execution process by automating the creation of Jenkins jobs and views lead us to the decision to give the Gradle-Jenkins-Plugin a try. The plugin allows it to maintain jenkins configurations in source control and initiate the update process on the Jenkins server via Gradle tasks. The parallel test execution is made possible by using the Multijob Plugin. It allows to aggregate jobs to so called upstream jobs. The build of a certain upstream job triggers the parallel execution of the associated downstream jobs. The executors can be distributed in multiple jenkins slave nodes or locally increased in the global Jenkins settings.
A Github repository containing our setup for an example project can be found here.
After having prepared jenkins as described in the readme of the repo and executing
$ ./gradlew updateJenkinsItems -Pjenkins_user=jenkins -Pjenkins_pass=password
we are notified that the jobs and views have been generated.
Executing external task 'updateJenkinsItems'...
Creating new item .cucumber.smoke.prod.multijob on http://localhost:8080/
// single feature jobs
Creating new item AnotherSearch.feature.smoke.prod on http://localhost:8080/
Creating new item AnotherSearch.feature.smoke.uat on http://localhost:8080/
Creating new item Search.feature.smoke.prod on http://localhost:8080/
Creating new item Search.feature.smoke.uat on http://localhost:8080/
Creating new item smoke.prod on http://localhost:8080/
Creating new item smoke.uat on http://localhost:8080/
Total time: 0.635 secs
16:09:19: External task execution finished 'updateJenkinsItems'.
The outcome of a couple of builds looks like the following in the Jenkins user interface:
Here we can see the build results of a multijob with two child jobs. When looking at the Cucumber reports we have a nice visual representation of the aggregated test results where each row corresponds to the test results of the scenarios in one feature file.
An example of the results of one child job containing one scenario looks like this:
So how did we make it work?
The usage of the Gradle Jenkins Plugin is nicely demonstrated with a lot of examples in the plugin’s Wiki. The basic concept is to provide a wrapper to the Jenkins API and to manipulate the job and view xml templates to accomplish the desired outcome. We created custom templates for job, view and multijob creation. They can be found here.
To manually run one feature file from the command line you would enter
$ ./gradlew -Penv=prod -Pfeature=AnotherSearch.feature -PadditionalArgs="--tags @smoke" cucumber
To automate the process we loop through all the feature files to collect the jobs to create. For each job definition we assemble the desired gradle command line switches. Then we gather the jobs for each environment and tag combination in views and multijobs. By means of the Jenkins Copy Artifact Plugin we make the test results of the child jobs accessible for the respective multijob.
The integration of all kinds of testing tasks like rerunning failing tests automatically and for example doing so just for scenarios tagged as monitor test has been made a cakewalk.
Automating the creation of our Cucumber integration tests on Jenkins has made it so easy to adapt the test suite setup to changing needs. Moreover, it has made the tests run faster. What is your preferred continuous integration setup for automated testing?