11/03/2026 updated
KV
100 % available
Senior Azure Data Engineer | ADF | Databricks | PySpark | Delta Lake
Bengaluru, India
Worldwide
Bachelor of Engineering in Computer SciencedbtApache AirflowPython (Programming Language)SQL DatabasesSnowflakePySpark
Azure Data Engineering & Pipeline Development
Over 7 years of experience building scalable batch and streaming pipelines using Azure Data Factory, Azure Databricks, PySpark, Azure Event Hubs and ADLS Gen2, including real-time event-driven processing and Delta Lake/Medallion architectures.
Data Modeling & Query Optimization
Design and optimization of Azure data models using partitioning, clustering and Delta Lake techniques, resulting in significantly faster query performance and cost savings across analytics workloads.
ETL/ELT Pipeline Design & Data Quality
Development of secure ETL/ELT pipelines with data cleansing, deduplication, transformation and validation frameworks to ensure accuracy and reliability of enterprise datasets, including automated pipeline orchestration using ADF triggers.
Apache Spark & PySpark
Proficiency in Apache Spark and PySpark for large-scale data processing, transformation and loading of curated datasets into Azure Synapse Analytics.
Azure Synapse Analytics
Experience with Azure Synapse Analytics for building star schemas, partitioned tables and optimizing SQL/Synapse queries to support downstream analytics and reporting workloads.
Database Management (SQL Server, MySQL, Oracle)
Design and optimization of SQL and MySQL queries, indexes and partitioning strategies, as well as monitoring of blocking sessions, deadlocks and resource bottlenecks in production databases.
Cloud Security & Governance (RBAC, Key Vault)
Implementation of secure, governed data solutions using Azure RBAC, Key Vault and ADLS Gen2 to support financial and retail data compliance requirements.
Python & SQL Programming
Programming in Python, SQL and PySpark for data pipeline development, automation and data quality checks across various cloud and on-premise environments.
Azure Monitor & Alerting
Automated performance health checks and alert monitoring for database and data platform environments, including RCA documentation and best practices for query tuning.
Over 7 years of experience building scalable batch and streaming pipelines using Azure Data Factory, Azure Databricks, PySpark, Azure Event Hubs and ADLS Gen2, including real-time event-driven processing and Delta Lake/Medallion architectures.
Data Modeling & Query Optimization
Design and optimization of Azure data models using partitioning, clustering and Delta Lake techniques, resulting in significantly faster query performance and cost savings across analytics workloads.
ETL/ELT Pipeline Design & Data Quality
Development of secure ETL/ELT pipelines with data cleansing, deduplication, transformation and validation frameworks to ensure accuracy and reliability of enterprise datasets, including automated pipeline orchestration using ADF triggers.
Apache Spark & PySpark
Proficiency in Apache Spark and PySpark for large-scale data processing, transformation and loading of curated datasets into Azure Synapse Analytics.
Azure Synapse Analytics
Experience with Azure Synapse Analytics for building star schemas, partitioned tables and optimizing SQL/Synapse queries to support downstream analytics and reporting workloads.
Database Management (SQL Server, MySQL, Oracle)
Design and optimization of SQL and MySQL queries, indexes and partitioning strategies, as well as monitoring of blocking sessions, deadlocks and resource bottlenecks in production databases.
Cloud Security & Governance (RBAC, Key Vault)
Implementation of secure, governed data solutions using Azure RBAC, Key Vault and ADLS Gen2 to support financial and retail data compliance requirements.
Python & SQL Programming
Programming in Python, SQL and PySpark for data pipeline development, automation and data quality checks across various cloud and on-premise environments.
Azure Monitor & Alerting
Automated performance health checks and alert monitoring for database and data platform environments, including RCA documentation and best practices for query tuning.
Experience
Professional experience5 - 10 years
Freelancer since2024
Languages
EnglishFluent
Project history
Architected and developed large-scale Azure batch and streaming pipelines using ADF, Azure Databricks, PySpark and ADLS Gen2, improving data ingestion throughput by 40%. Designed and optimized Azure data models using partitioning, clustering and Delta Lake techniques, achieving 35% faster query performance and 25% cost savings. Built real-time Azure Event Hubs and Databricks Structured Streaming pipelines for financial data streams. Collaborated with fraud and risk teams to deliver secure, governed financial datasets using Azure RBAC, Key Vault and ADLS Gen2.
Designed and developed Azure data pipelines for ingesting retail and customer transactions using ADF, ADLS Gen2, Azure Databricks and PySpark. Built star schemas and partitioned tables in Azure analytical platforms. Implemented data cleansing, deduplication and transformation using PySpark and Databricks before loading into Azure Synapse Analytics. Automated end-to-end pipeline orchestration using ADF pipelines and triggers. Optimized Azure SQL/Synapse queries leading to 30-40% cost savings. Delivered analytics-ready datasets supporting sales and customer churn dashboards.
Designed and optimized Azure SQL and MySQL queries, indexes and partitioning strategies, improving database performance by up to 40%. Developed and maintained Azure data ingestion and ETL pipelines using Azure Data Factory, ADLS Gen2 and Azure Databricks to integrate relational data sources. Implemented data quality checks, validation and reconciliation frameworks for enterprise datasets. Monitored blocking sessions, deadlocks and resource bottlenecks in production databases. Automated performance health checks and alert monitoring. Documented RCA and provided best practices for query tuning and data model design.
Certificates
Databricks certified Data Engineer professional
Databricks2025