A system-to-be was to cope with the continuously

Data Analytics System Enabling Cross Analysis of 30,000 Attributes and 100x Faster Reporting

Technologies : Hadoop, Python, Scala, Spark, AWS, Big data, Cloud, Azure, .NET, C#, WPF, XAML, MVVM

Customer : The Customer is a leading market research company.

Challenge : Though having a robust analytical system, the Customer believed that it would not be able to satisfy the company’s future needs. Acknowledging this situation, the Customer was keeping their eyes open for a future-focused innovative solution. A system-to-be was to cope with the continuously growing amount of data, to analyse big data faster and enable comprehensive advertising channel analysis.

Solution : During the project, the Customer’s business intelligence architects were cooperating closely with Delta cubes Team designed an idea, and the latter was responsible for its implementation.

For the new analytical system, the Customer’s architects selected the following frameworks:

• Apache Hadoop – for data storage
• Apache Hive – for data aggregation, query and analysis
• Apache spark – for data processing

Amazon Web Services and Microsoft Azure were selected as cloud computing platforms.

Upon the Customer’s request, during the migration, the old system and the new one were operating in parallel.

The system has been supplied with raw data taken from multiple sources, such as TV views, mobile devices browsing history, website visits data and surveys. To enable the system to process more than 1,000 different types of raw data (archives, XLS, TXT, etc.), data preparation included the following stages coded in Python:

• Data transformation
• Data parsing
• Data merging
• Data loading into the system

IMPACT : At the project closing stage, the new system was able to process several queries up to 100 times faster than the outdated solution. With the valuable insights that the analysis of almost 30,000 attributes brought, the Customer was able to carry out comprehensive advertising channel analysis for different markets.

Technologies and Tools : Apache Hadoop, Apache Hive, Apache Spark, Python( ETL) Scala (Spark, ETL), SQL (ETL), Amazon Web Services (Cloud storage), Microsoft Azure (Cloud storage), .NET, WPF, C#.