About the Client
Though the client had a robust analytical system in place, they recognized it would not scale to meet future needs. They were looking for a future-focused, innovative solution able to keep pace with a continuously growing volume of data, analyze it faster, and enable comprehensive advertising channel analysis.
Project Overview
For the new analytical system, Deltacubes selected Apache Hadoop for data storage, Apache Hive for data aggregation, query, and analysis, and Apache Spark for data processing, with AWS and Microsoft Azure as the cloud platforms. During migration, the old and new systems ran in parallel at the client's request.
The system ingested raw data from multiple sources — TV views, mobile browsing history, website visits, and surveys — across more than 1,000 different data types (archives, XLS, TXT, and more). Data preparation was coded in Python across transformation, parsing, merging, and loading stages.
Challenges
The existing analytical system could not keep up with the growing amount of data the client needed to process, limiting their ability to run timely, comprehensive advertising channel analysis across markets.
Results
At project close, the new system processed queries up to 100 times faster than the outdated solution. Analysis of nearly 30,000 attributes gave the client the insight needed to run comprehensive advertising channel analysis across different markets.
Technologies & Tools
Conclusion
A modern, cloud-backed big-data pipeline replaced a system that could no longer keep pace — giving the client dramatically faster queries and deeper cross-market advertising insight.