Overview
This project involved designing and implementing a robust data architecture to create a Single Customer View (SCV) by integrating data from multiple heterogeneous sources. The solution ensured high-quality, real-time, and batch data processing, enabling reliable insights across platforms.
Data is passed to Databricks for cleansing and transformation:
Null value elimination
Date format normalization
Duplicate removal
Standardization across datasets
Cleaned data from all sources is then integrated to build a Single Customer View (SCV), enabling a 360° profile of each customer.
The transformed and unified data is stored back into ADLS, with two key downstream flows:
Batch loading into Azure Synapse Analytics and SQL databases for further analysis and BI reporting.
Streaming data pipelines powered by Azure Stream Analytics (ASA) and Azure Data Explorer (ADX) for near-real-time insights.
Extraction
Transformation
Loading
Data is sourced from:
Salesforce (event-driven)
SAP (REST API)
Mobile/Web via Firebase
Finacle Core Banking (SOAP API)
Data from each source is first landed in Azure Data Lake Storage (ADLS). Azure Data Factory (ADF) is used, particularly for the Finacle integration, to orchestrate and temporarily store the extracted data securely.
End-to-End Data Integration Architecture for Unified Customer View Using Azure Ecosystem
Impact
This architecture enabled unified, real-time, and self-service analytics capabilities across business units, enhanced decision-making, and improved customer engagement through personalized insights.
