Case Study
Centralized, managed data lake helped global retailer accelerate analytical insights
OUR CLIENT
This American multinational retailer operates more than 10,000 stores in various formats and has a robust footprint of e-commerce websites around the world. Focusing on value and economic opportunities, the company employs millions of associates.
THE CHALLENGE
Streamlining data ingestion and processing
The retailer wanted to deploy a managed data lake with an extract, load, transform (ELT) framework to simplify data ingestion processes and also:
- Consolidate data from multiple relational database management systems (RDBMS)—into a Hadoop distributed file system (HDFS)
- Improve analytics capabilities overall—with more efficient data processing
- Boost developer productivity—by reducing complexities in its data analytics environment
- Improve customer service and strategic business decisions—with a unified view of customer and sales data
THE TRANSFORMATION
Delivering a centralized data lake for data-driven decision-making
After assessing the retailer’s data analytics needs, UST created a big data architecture design that consolidated data from disparate sources in a Hadoop storage system for easy processing and curation. During the engagement, UST:
- Implemented a batch processing framework—to ingest data from different sources into a managed data lake
- Deployed an ELT process—to abstract the complexity of data processing on the Hadoop platform and ensure data is efficiently transformed and loaded into the target system
- Orchestrated data processing workflows—using Automic to schedule and monitor automated data ingestion processes
- Created data transfer processes—from source systems to the HDFS, using a Sqoop connector for RDBMS files and a Teradata connector for Teradata files
- Enabled efficient, scalable data processing in the HDFS—using Hive and Tez
THE IMPACT
Harnessing data for informed decision-making
The new data lake accelerated analytical insights, eliminated data errors and increased developer productivity. Meanwhile, business users gained a holistic view of customers and operational processes to improve staffing, product selection, service and support.
Resources
https://www.ust.com/en/what-we-do/experience-design