The frontend of any modern web-application uses static resources such as images, CSS, and JavaScripts to render the pages for the end user. Such resources are collectively known as web-assets or just assets. Caching the assets in the browser is the most obvious and simple way to improve the frontend performance. Only, it’s not always that obvious, neither that simple.
In this article, we share some experience and lessons learned during the redesign of a more efficient asset delivery system at our client, idealo.
Where we started: original design and its limitations
Idealo is the leading price comparison website in Europe and one of the largest e-commerce markets in Germany. At the heart of the website, the ‘offers of product’ page aggregates the offers from over 50.000 shops online, allowing quick comparisons and purchases.
The backend supporting these functionalities is a Spring Boot-based application. In the original architecture, the web-assets were packed into a deployable artifact, deployed on each application server, and then distributed via CDN using the application servers as the origin. The continuous improvements to the ‘offers of product’ page led to daily deployments and frequent updates of the assets. To tell the CDN which images, scripts, and stylesheets were required for a given application release, each asset was served under a URL specific to that release. The automatic generation of such URLs occurred at compile time, by including the version identifier of the application into the URL path (i.e., the short Git version hash). For example, if the internal location of an asset was ‘img/logo.png’ and the application version was ‘a123’, then the resulting URL looked like ‘cdn.idealo.com/offerpage/a123/img/logo.png’. Obviously, the same asset had a different URL in the next application release, even if the asset did not change.
As you might have already noticed, this asset delivery strategy is all about ensuring version consistency between assets and releases, but it’s not caching friendly. In fact, the browser must download all assets after each deployment, because the release dependent URLs prevent browsers from using their cached copies. That’s not very efficient, especially for mobile devices or clients with limited bandwidth.
The existing deployment pattern contributed to another issue with the original asset delivery strategy, causing continuous 404 Errors after each new deployment. In more detail, the client requests reached a load balancer running in front of several application servers, with each server compiling the asset URLs based on its release version. Thus, if not all the application servers were running the same version, the assets of the outdated versions were not reachable on the latest release, and vice versa (see diagram below). Sadly, that’s what happens at every new deployment, when - for a limited period - some application servers are running the old release because there were not updated yet [1].
Besides the direct impact on the application availability, this issue was also a major impediment toward continuous deployment. In fact, the more we deploy, the more 404s we will get.
But so much for the problems. Let’s talk about solutions.
Fingerprinting and Cloud Storage to the rescue
To overcome the limitations described above, we designed the new asset delivery solution around two technologies: file fingerprinting and cloud storage.
File fingerprinting is a versioning technique that binds the name of a file to its actual content. In other words, a fingerprinted file will never change its name as long as its content doesn’t change, even across different application releases. Usually, this is done by adding the file hash to the name (e.g., ‘logo-bc456de78f.png’). As a result, the browser can cache the web-assets for a virtually unlimited time, because only the updated assets need to be downloaded again from their new URL. This practice is also known as ‘cache busting’.
With fingerprinting, the occurrence of 404 Errors during deployments decreased significantly, together with the number of assets that change URL across releases. However, one updated asset is enough to raise the same issue again. In fact, if the previous version of an asset was never requested and therefore never cached by the CDN, a new request for that version could not be resolved by any server running the latest release. To solve this problem once and for all, we store the recent versions of each asset on a cloud storage service, which is used as the origin server for the CDN instead of the application servers. This ensures that previous versions of an asset will be available long enough to prevent 404 Errors during deployments.
Eager for more information? Let’s dive into the technical details.
To Spring or not to Spring: choosing the right Tool for Fingerprinting
Spring Framework supports web asset versioning out-of-the-box. It provides a set of managed components to create and resolve fingerprinted versions of any asset in an automated and easy-to-configure fashion. Spring calculates the version of an asset at runtime at every first request. Then, it caches in-memory the mapping between original and versioned names for the next requests. Unfortunately, serving the assets through the application will still cause 404 Errors during deployments, making the Spring way unsuitable for our needs.
In our solution, fingerprinting and resolution of web-assets are performed by separate components. For the former task, we rely on the popular build tool and module bundler Webpack. The output is a fingerprinted copy of each asset, along with a manifest file (a JSON file) that maps the original file name to the renamed file. An example of the manifest file is below.
{
‘css/style.css’: ‘css/style-13qet57adg9.css’,
‘img/logo.png’: ‘img/logo-bc456de78f.png’,
‘js/script.js’: ‘js/script-09zyx87wvu6.js’
}
We decided to use Webpack for different reasons:
- Relevance: the tool is designed to deal with web-assets.
- Reusability: another module of the application already required Webpack.
- Efficacy: the tool comes with plugins that greatly simplify fingerprinting (e.g., MD5 hash calculation, manifest generation, preprocessing of versioned JS/CSS files).
The fingerprinted files are packed into the deployable artifact together with the manifest file. To resolve asset requests at runtime, the application has to (i) query the manifest, (ii) generate the versioned URL, and (iii) render that URL into the page DOM. We implemented these functionalities by extending existing Spring components as well as developing new ones.
Moving to the Cloud
The redesigned web-asset delivery system avoids 404 Errors during deployments by using cloud storage as the origin server for the CDN. The designated cloud storage is Akamai NetStorage, due to its perfect integration with the CDN of the same vendor in use at idealo.
The assets are moved to NetStorage during the deployment pipeline run, right before deploying the application. The process includes the steps listed below. Note that if the process fails, then the deployment will fail also.
- Extract manifest and fingerprinted assets from the latest application artifact.
- Upload to NetStorage the assets listed in the manifest and not already in the cloud.
- Remove the obsolete assets from NetStorage. The clean-up policy is to keep a configurable number of latest versions for every given asset and remove the others.
- Check the availability of each asset in the manifest via CDN. As a positive side effect, these checks will issue a request for every asset, forcing the CDN to cache them.
We implemented this process as a Python program. One of the main reasons for choosing this language is that the Python NetStorage HTTP API open-sourced by Akamai is the most maintained and popular on GitHub when compared with the APIs released for other languages. The deployment pipeline is managed by the Jenkins build server, which coordinates the different stages of the application deployment, including the execution of the Python program described above. We recommend executing the program in an isolated environment, to keep the program installation clean as well as to avoid permission issues on the Jenkins server (for example, when installing Python dependencies via PiP). Virtualenv and Docker are both valid tools for serving this purpose [2].
Conclusions
In this article, we shared some of our experiences at idealo improving the web-asset delivery system for the Spring Boot application that powers the ‘offers of product’ page. Our goal was to increase the caching efficiency and reliability of the assets distribution. To this end, we based the new design on two key ideas: (i) assets fingerprinting - to enable cache busting, and (ii) storing the assets on Akamai NetStorage - to improve their availability.
Thanks to the new system, we had no more 404 Errors during deployments and we greatly reduced the number of assets to download at every application release. To give some numbers, we now upload only 2% of all the assets, which translates into more than 95% of traffic saving on the end user’s side per deployment (average over the last 30 deployments).
We expect these results to continue and be solid enough to withstand the upcoming shift of the ‘offers of product’ team toward continuous deployment.
Let’s check our expectations in a few months and report on that in a follow-up post, shall we?
image source: Chinabrands
[1] One might think that configuring sticky sessions for the load balancer would solve this problem. However, when an application server eventually restarts to run the last release, all its clients will lose their session, and the issue will appear again - unless a costly failover strategy is in place.
[2] Here is a small lesson learned while working with Virtualenv in Jenkins. If you plan to create a virtual environment by using the sh (shell) step in your Jenkinsfile, make sure to include all commands within the same shell, and not to specify each command in a different sh step. This is because Jenkins creates a new session for every shell, preventing a virtual environment activated in one sh step from being used in any other step.