Be honest: who doesn’t love a bargain? Cyber Mondays, Black Fridays and countless other savings events trigger our hunter instinct. But is that bargain really as cheap as retailer A claims? Or is retailer B not just a few euros cheaper? If you want to know, check out idealo, the leading platform for price comparisons. Since it was launched in 2000, idealo pages have listed all possible and impossible products, neatly sorted by price: cheap at the top, expensive at the bottom. They make bargain hunting child’s play.
As simple as the price comparison is for users, the technology behind it is complex: 530 million offers from 50,000 online retailers are currently listed in the database. Every day, idealo processes billions of data points and assigns them to the correct product.
Consumers can find the product they are looking for in a matter of seconds. idealo has 76 million visits in Germany alone. This makes idealo number 1 in Germany. This achievement is also set to be reached in the other countries in which the company is active, namely France, the UK, Italy, Austria and Spain. Further countries are expected to be added.
Technology to scale
These ambitious goals, however, were in danger of not being reached because of the on-premises infrastructure. Over the last 10 years, idealo had massively modernized large parts of the price comparison platform. However, “Metal”, i.e., the hardware for more computing power, had to be ordered well in advance and installed and set up in the rented data center. This had unpleasant consequences. For example, retailers send changes such as voucher promotions to idealo via an API. However, during peak periods like Black Friday or Cyber Monday, these vouchers were sometimes only visible online when the vouchers were already invalid. The idealo management was faced with a choice: either deal with the outdated on-premises IT and update it manually, with all the restrictions for growth and cyber security. Or migrate to the cloud with the latest technology and greater flexibility and security. The management opted for the second option: to make a clean cut and migrate to the AWS Cloud.
A partner was quickly found: kreuzwerker. A decision that Andreas Hankel, CTO at idealo, has not regretted for a second: “kreuzwerker has helped our teams to scale their systems flexibly with growing data volumes. This helps us achieve our ambitious growth targets and expand into new markets.”
Done on time
The migration project started at the beginning of 2022 and was scheduled to be completed by the end of 2023. The kreuzwerker experts and their colleagues at idealo met this challenging deadline on time: the essential work was already completed by Black Friday 2023, so that the rush of bargain hunters was easily handled on 24 November. Mission accomplished, you could say. For Honey Feelisch, migration program manager at idealo, it is also extraordinary that a major migration project not only remains within budget, but is completed even faster than planned. The secret to success according to Feelisch: “We were better prepared for the cloud than we wanted to admit. The migration was something like an imaginative giant that became smaller the more concretely we dealt with it.” Feelisch’s achievement was taking away the teams’ fear of actually tackling this imaginative giant. Instead of chewing over all the if’s and but’s countless times, reinventing the wheel and sapping the team’s energy, the program manager encouraged the teams to simply get started and use tried-and-tested solutions.
Even for the migration-experienced kreuzwerkers, idealo was a “huge project”, says Kristine Jetzke, CTO at kreuzwerker. On average, three developers worked full-time for the customer for one and a half years to move the data infrastructure and the 30 on-premises Kafka clusters, including websites and around 100 apps such as price alerts or wish lists from the idealo data center to the AWS cloud. This requires in-depth technical expertise - and tact and sensitivity. The real challenge was the special organizational structure. At idealo, 36 teams work autonomously, divided into product groups. Each product team consists of product owners, software engineers, team leaders and, in some cases, analysts and designers. Each team also has its own tech stacks. The motto is: “You build it, you run it”.
Small teams, big freedom
The open and self-determined organization reminds Kristine Jetzke of the way kreuzwerker works, she says, but in the case of idealo, it also brings obstacles. When a centrally managed technology is suddenly introduced to an organization consisting of small teams and a lot of freedom, this almost automatically generates resistance. The concerns were dispelled fairly quickly though. For one thing, the chemistry in the kreuzwerker and idealo teams was right from the start. Secondly, the kreuzwerker experts were often on site and worked hand in hand. “Sometimes we also had to mediate between the teams at idealo and build bridges,” says Kristine Jetzke. Honey Feelisch is full of praise: “Through the collaboration, our teams also gained knowledge about AWS and solutions that will help them to be happy on the new platform long after the migration.”
In the beginning…
The migration to the AWS cloud is not the first project that kreuzwerker has carried out for idealo. The collaboration began back in 2017 with the restructuring of the website, and in 2018 idealo took its first steps into the AWS cloud with a data lake. And it won’t be the last joint project either. “The migration is just the beginning,” says Andreas Hankel. A cloud-native company is the goal. The aim is to reap the rewards of cost optimization. IT will also become AI-ready. For example, the product teams can use large language models to automatically create product descriptions from retailers’ offers.
The hassle of invalid vouchers on Black Fridays or Cyber Mondays is now also a thing of the past thanks to the migration to the AWS cloud, says Andreas Hankel. “We can now process millions of such changes within 15 minutes.”