Senior Cloud Data Engineer

Lead data engineer on a digital transformation program aimed at consolidating the client's project
operational data into a unified cloud platform, providing management with a comprehensive 360-
degree view of projects. The solution, built on the Azure cloud platform, involved creating a
bespoke modular framework to handle large volumes of data using Azure Data Lake, Azure Data
Factory, Azure Synapse Analytics, and Azure Databricks.

Responsibilities
* Collaborated with the technical lead to provide clear progress updates to senior leadership
and acted as a point of escalation when needed.
* Assisted in creating and monitoring the project timeline and plan.
* Engaged with source system owners and downstream consumers to analyze source data
and define requirements.
* Defined the solution architecture on the Azure cloud platform. The ETL solution defined was
built to be a generic, fully parametrized framework which would enable the client to easily
extend the solution to include additional data-sets solely through configuration updates. The
solution incorporated:
o Azure Data Lake Gen2 (ADLS): For storing data in Delta and Parquet formats.
o Azure Data Factory: For orchestrating data ingestion.
o Azure Synapse Analytics: For data transformation.
o Azure Purview: For data governance.
o Azure Key Vault: For securing sensitive information such as keys and credentials.
o Azure Virtual Networks and Virtual Machines: For establishing private networks.
* Led a team of data engineers and modelers in building the solution, defining target data
models (both relational and dimensional), and producing essential documentation including
Low and High-Level Design documents, Source-to-Target mapping documents, and
handover training materials.
* Developed a proof of concept in Databricks using Python and PySpark, designed to be
generic and modular, facilitating easy extension to additional datasets if the client opted for
Databricks.




Telecommunication & Multimedia Provider

Consolidated client's disparate data into a single on-premise data warehouse to provide a
streamlined view of project data from lead generation, down to planning and delivery. Served as
a data modeler and business analysts, tasked with defining data requirements and translating
them into a data model, and subsequently carrying out extensive testing on the solution.

Responsibilities
* Carried out workshops with client teams to understand business processes and source data
models
* Defined target data models (a dimensional model and a relational model) which
consolidated source data for consumption by the downstream client reporting teams
* Support client reporting teams in dashboard building activities by defining KPI logic and
definition and mapping them to fields within target data model
* Owned and produced key deliverables: the source-to-target mapping document, and data
dictionary
* Carried out extensive testing on the solution based on business logic to ensure the solution
works as intended

Served as the Lead Data Engineer on a major cloud migration project, guiding the client through
the migration from legacy on-premise data silos to a secure Azure cloud-based solution. Ensured
careful handling of PII and other sensitive information. The solution made use of Azure Data
Lake for storage, Azure Synapse Analytics for data transformation, and Azure Data Factory for
orchestration.

Responsibilities
* Facilitated discovery and design workshops with source system owners and end-users to
define requirements and use cases. This included the analysis and comprehension of
complex data models from source systems.
* Designed the solution architecture on the Azure cloud platform, adhering to Microsoft's
reference architecture guidelines, and implemented best practices for development
lifecycle, source control, and testing methodologies. Key components of the solution
included:
o Azure Data Lake Gen2 (ADLS): For storing data in Delta or Parquet formats based
on its usage.
o Azure Data Factory: For orchestrating data ingestion and performing simple
transformations.
o Azure Synapse Analytics: For executing complex transformations and building a
logical data model for downstream use.
o Azure Purview: For data governance.
o Azure Key Vault: For managing and securing sensitive information such as keys and
credentials.
* Managed a team of junior data engineers and data modelers to develop the target data
model and build the solution, successfully meeting client deadlines
* Delivered key documentation, including High and Low-Level Design Documents and
Source-to-Target mapping documents
* Supported business analysts and product owners in defining and sizing user stories
* Supported the testing team by developing high-level test scripts and providing insights into
the solution design and development