11/15/2026 updated


100 % available
Lead Data Engineer & Platform Architect | Azure | Databricks | GenAI
Bhopal, India
Only remote
M.Sc. Computer ScienceAbout me
Lead AI Data Engineer and data scientist with 12+ years of experience designing, scaling, Proven expertise orchestrating high-performance Lakehouse architectures (Medallion schema and CICD) utilizing Azure Databricks, PySpark, Python, Sql, Docker, Azure Data Generation (RAG), and LangChain
Microsoft AzureCloud EngineeringData ArchitectureData MartEnterprise Architecture FrameworkData IntelligencePython (Programming Language)Azure SQLMicrosoft Business IntelligenceNumPyOracle Business Intelligence Enterprise EditionDataOpsAzure Machine LearningAzure Data LakeSQL Databases
Cloud & Data Engineering (Azure)
Expertise in Azure Data Factory, Azure Databricks, Azure Synapse Analytics, ADLS Gen2, Delta Lake, Azure SQL, SQL Server, Lakehouse and Medallion Architecture, Databricks Genie, and Microsoft Fabric for building enterprise-grade cloud-native data platforms.
Generative AI & NLP
Hands-on experience with Azure OpenAI (GPT-4o), LangChain, RAG Chatbot Development, ChromaDB, Azure AI Search, Vector Embeddings, Semantic Chunking, Azure Document Intelligence, OCR, and PyMuPDF for productionizing enterprise AI solutions.
ETL/ELT Framework Design & Architecture
Deep knowledge in enterprise platform design, reusable ETL/ELT framework engineering, schema evolution engines, cost and performance optimization, and automated data validation frameworks including schema drift detection and SLA monitoring.
Programming & DevOps
Proficiency in Python, PySpark, Spark SQL, SQL, FastAPI, Docker, Docker Compose, Azure DevOps, Git, Bitbucket, Jenkins, SonarQube, and Release Management for end-to-end pipeline development and CI/CD governance.
BI & Reporting
Experience with Power BI, Tableau, Looker, BusinessObjects, and DAX/SQL logic for business intelligence, data quality automation, schema drift detection, RCA, and production support.
NoSQL & Event Streaming
Integration of Azure Cosmos DB for low-latency NoSQL ingestion including partition key design, document-index models, and RU strategies, as well as Azure Event Hubs for event-driven data architectures.
Team Leadership & Delivery
Managing teams of up to 10 engineers with task allocation, delivery oversight, technical mentoring, BI and UI team collaboration, cross-functional coordination, sprint planning, estimation, and stakeholder management.
Data Warehouse & SQL Analytics
Design and maintenance of SQL Server data warehouse and data mart solutions including stored procedures, views, CTEs, window functions, and transactions for KPI and operational reporting.
Expertise in Azure Data Factory, Azure Databricks, Azure Synapse Analytics, ADLS Gen2, Delta Lake, Azure SQL, SQL Server, Lakehouse and Medallion Architecture, Databricks Genie, and Microsoft Fabric for building enterprise-grade cloud-native data platforms.
Generative AI & NLP
Hands-on experience with Azure OpenAI (GPT-4o), LangChain, RAG Chatbot Development, ChromaDB, Azure AI Search, Vector Embeddings, Semantic Chunking, Azure Document Intelligence, OCR, and PyMuPDF for productionizing enterprise AI solutions.
ETL/ELT Framework Design & Architecture
Deep knowledge in enterprise platform design, reusable ETL/ELT framework engineering, schema evolution engines, cost and performance optimization, and automated data validation frameworks including schema drift detection and SLA monitoring.
Programming & DevOps
Proficiency in Python, PySpark, Spark SQL, SQL, FastAPI, Docker, Docker Compose, Azure DevOps, Git, Bitbucket, Jenkins, SonarQube, and Release Management for end-to-end pipeline development and CI/CD governance.
BI & Reporting
Experience with Power BI, Tableau, Looker, BusinessObjects, and DAX/SQL logic for business intelligence, data quality automation, schema drift detection, RCA, and production support.
NoSQL & Event Streaming
Integration of Azure Cosmos DB for low-latency NoSQL ingestion including partition key design, document-index models, and RU strategies, as well as Azure Event Hubs for event-driven data architectures.
Team Leadership & Delivery
Managing teams of up to 10 engineers with task allocation, delivery oversight, technical mentoring, BI and UI team collaboration, cross-functional coordination, sprint planning, estimation, and stakeholder management.
Data Warehouse & SQL Analytics
Design and maintenance of SQL Server data warehouse and data mart solutions including stored procedures, views, CTEs, window functions, and transactions for KPI and operational reporting.
Experience
Professional experience5 - 10 years
Freelancer since2021
Languages
EnglishFluentHindiFluent
Project history
Directly managing a team of 7 data engineers, overseeing sprint delivery, code reviews, and quality. Architected and delivered a production RAG-based chatbot for HCL official documents using Azure OpenAI (GPT-4o), LangChain, ChromaDB, and Azure AI Search. Engineered a reusable Dynamic Join Framework (PySpark) eliminating repetitive logic across 40+ source integrations. Reduced Databricks compute costs by 80% through right-sized cluster configurations and partitioning strategies. Designed Automated Data Validation Framework with schema drift detection and SLA monitoring. Led architecture and code reviews, managed environment promotions via Azure DevOps and Bitbucket.
Led and managed a team of 10 engineers across Dev, QA, and Production environments. Architected Medallion Architecture (Bronze, Silver, Gold) on Azure Databricks and ADF serving 9 source systems, processing 2 TB/day of global healthcare data. Integrated Azure Cosmos DB for low-latency NoSQL ingestion. Drove CI/CD governance using Azure DevOps, Jenkins, SonarQube, and Bitbucket. Supported BI and BusinessObjects reporting teams and performed production RCA and support.
Supported data modeling and platform migration from PostgreSQL to Cloud SQL, including system testing and QA validation across all environments.
Certificates
data bricks professional
Databricks2024
azure-203
microsoft2024