09/18/2026 updated
100 % available
Senior Data Engineer | Databricks & Big Data Specialist | 10+ Years Experience
Paonta Sahib, India
Only remote
B.Tech, National Institute of Technology (NIT), Hamirpur, 2014Big DataData ArchitectureData WarehousingDatabase TuningAzure Data LakeData StreamingApache SparkPySparkApache KafkaSpark StreamingData PipelineDatabricks
Databricks & Apache Spark
Expertise in designing and optimizing end-to-end data pipelines using Databricks and Apache Spark, including Delta Lake, Delta Live Tables (DLT), and Structured Streaming. Proven track record of reducing Spark job run times by up to 67% through code redesign, query optimization, and efficient partitioning strategies.
Cloud Platforms (Azure & AWS)
Deep knowledge of cloud data engineering on Microsoft Azure (ADF, Azure Cosmos DB, AKS, Azure Blob Storage) and AWS (EMR). Hands-on experience deploying containerized, cloud-native data workloads and integrating cloud services for scalable data movement and processing.
Real-Time Streaming & Data Pipeline Development
Strong background in building real-time streaming pipelines using Apache Kafka, PySpark Structured Streaming, and Delta Live Tables. Experience ingesting, transforming, and delivering high-volume transactional and market data for financial reporting, healthcare analytics, and e-commerce use cases.
Data Warehousing & Lakehouse Architecture
Proficiency in data warehousing technologies including Snowflake, Delta Lake, Cassandra, Oracle DB, and SQL Server. Experience leading Data Lakehouse migration engagements from legacy data warehouses to modern Databricks Delta Lake architectures.
ETL Pipeline Design & Big Data Ecosystem
Extensive experience designing reusable, scalable ETL frameworks using Apache Spark, Scala, PySpark, Hadoop, Hive, Impala, and HDFS. Skilled in data cleansing, change data capture (CDC), historical data purging, and analytical data loading for healthcare and pharma domains.
Orchestration & DevOps
Hands-on experience with workflow orchestration using Apache Airflow (DAG design and scheduling) and containerization with Docker and Kubernetes. Familiar with CI/CD pipelines, Git, GitHub, and Jenkins for automated deployment and version control.
Python & Scala Programming
Solid programming skills in Python and Scala for building data engineering solutions, including rule-based scoring modules, Spark jobs, and automated reporting pipelines across healthcare, finance, and retail domains.
Performance Tuning & Cost Optimization
Demonstrated ability to optimize Spark job performance through deep-dive profiling, query plan analysis, and resource tuning, achieving up to 50% improvement in job run times. Experience implementing Databricks Instance Pools to reduce cluster spin-up time and lower infrastructure costs.
Data Security & Compliance
Implementation of role-based access control (RBAC) to enforce data security and regulatory compliance in financial and healthcare data environments. Experience with automated alerting systems for pipeline monitoring and SLA adherence.
Expertise in designing and optimizing end-to-end data pipelines using Databricks and Apache Spark, including Delta Lake, Delta Live Tables (DLT), and Structured Streaming. Proven track record of reducing Spark job run times by up to 67% through code redesign, query optimization, and efficient partitioning strategies.
Cloud Platforms (Azure & AWS)
Deep knowledge of cloud data engineering on Microsoft Azure (ADF, Azure Cosmos DB, AKS, Azure Blob Storage) and AWS (EMR). Hands-on experience deploying containerized, cloud-native data workloads and integrating cloud services for scalable data movement and processing.
Real-Time Streaming & Data Pipeline Development
Strong background in building real-time streaming pipelines using Apache Kafka, PySpark Structured Streaming, and Delta Live Tables. Experience ingesting, transforming, and delivering high-volume transactional and market data for financial reporting, healthcare analytics, and e-commerce use cases.
Data Warehousing & Lakehouse Architecture
Proficiency in data warehousing technologies including Snowflake, Delta Lake, Cassandra, Oracle DB, and SQL Server. Experience leading Data Lakehouse migration engagements from legacy data warehouses to modern Databricks Delta Lake architectures.
ETL Pipeline Design & Big Data Ecosystem
Extensive experience designing reusable, scalable ETL frameworks using Apache Spark, Scala, PySpark, Hadoop, Hive, Impala, and HDFS. Skilled in data cleansing, change data capture (CDC), historical data purging, and analytical data loading for healthcare and pharma domains.
Orchestration & DevOps
Hands-on experience with workflow orchestration using Apache Airflow (DAG design and scheduling) and containerization with Docker and Kubernetes. Familiar with CI/CD pipelines, Git, GitHub, and Jenkins for automated deployment and version control.
Python & Scala Programming
Solid programming skills in Python and Scala for building data engineering solutions, including rule-based scoring modules, Spark jobs, and automated reporting pipelines across healthcare, finance, and retail domains.
Performance Tuning & Cost Optimization
Demonstrated ability to optimize Spark job performance through deep-dive profiling, query plan analysis, and resource tuning, achieving up to 50% improvement in job run times. Experience implementing Databricks Instance Pools to reduce cluster spin-up time and lower infrastructure costs.
Data Security & Compliance
Implementation of role-based access control (RBAC) to enforce data security and regulatory compliance in financial and healthcare data environments. Experience with automated alerting systems for pipeline monitoring and SLA adherence.
Experience
Professional experience10+ years
Freelancer since2025
Languages
EnglishFluent
Project history
Building a robust, real-time financial data pipeline to ingest, transform, and deliver financial data to end-users and reporting dashboards. Processing transactional and market data from multiple sources in near real-time using PySpark, Apache Kafka, Structured Streaming (Delta Live Tables), Snowflake, MySQL, Databricks, and Delta Lake. Key achievements include optimizing Spark job run times by 67%, implementing Databricks Instance Pools, and enforcing RBAC for data security and compliance.
Worked as a Senior Technical Solutions Engineer with 10+ enterprise customers (JPMC, Swiggy, Meesho, and others) across Finance, Retail, and E-commerce domains. Led Data Lakehouse migration engagements, real-time streaming pipeline development, and performance tuning using Apache Spark, Delta Lake, DLT, Structured Streaming, AWS EMR, PySpark, Scala, Python, and SQL. Delivered up to 50% improvement in Spark job run times and handled data volumes up to 12 TB.
Developed the Commercial Cradle healthcare claims intelligence platform to analyze, prioritize, and flag insurance claims using a rule-based recommendation engine. Built end-to-end Spark modules to ingest claims data from Oracle DB into Cassandra, implemented Azure Cosmos DB for claims metadata, and containerized workloads using Docker on AKS. Tech stack included Apache Spark, Python, Apache Cassandra, Apache Airflow, Azure Data Factory, Azure Blob Storage, AKS, Azure Cosmos DB, Docker, Kubernetes, and Oracle DB.