11/15/2026 updated

**** ******** ****
100 % available

Lead Data Engineer & Platform Architect | Azure | Databricks | GenAI

Bhopal, India
Only remote
M.Sc. Computer Science
Bhopal, India
Only remote
M.Sc. Computer Science

Profile attachments

gayatri-gautam-resume (1).pdf

About me

Lead AI Data Engineer and data scientist with 12+ years of experience designing, scaling, Proven expertise orchestrating high-performance Lakehouse architectures (Medallion schema and CICD) utilizing Azure Databricks, PySpark, Python, Sql, Docker, Azure Data Generation (RAG), and LangChain

Microsoft AzureCloud EngineeringData ArchitectureData MartEnterprise Architecture FrameworkData IntelligencePython (Programming Language)Azure SQLMicrosoft Business IntelligenceNumPyOracle Business Intelligence Enterprise EditionDataOpsAzure Machine LearningAzure Data LakeSQL DatabasesDatabase EnginesAzure AutomationBusiness Intelligence Development StudioCloud Platform SystemAzure Data FactoryGoogle Data StudioApache SparkSQL Server Business IntelligenceGenerative AIpandasMicrosoft FabricPySparkAWS Data AnalyticsSpark StreamingAzure Data AnalyticsAzure Synapse AnalyticsDatabricks
Cloud & Data Engineering (Azure)
Expertise in Azure Data Factory, Azure Databricks, Azure Synapse Analytics, ADLS Gen2, Delta Lake, Azure SQL, SQL Server, Lakehouse and Medallion Architecture, Databricks Genie, and Microsoft Fabric for building enterprise-grade cloud-native data platforms.

Generative AI & NLP
Hands-on experience with Azure OpenAI (GPT-4o), LangChain, RAG Chatbot Development, ChromaDB, Azure AI Search, Vector Embeddings, Semantic Chunking, Azure Document Intelligence, OCR, and PyMuPDF for productionizing enterprise AI solutions.

ETL/ELT Framework Design & Architecture
Deep knowledge in enterprise platform design, reusable ETL/ELT framework engineering, schema evolution engines, cost and performance optimization, and automated data validation frameworks including schema drift detection and SLA monitoring.

Programming & DevOps
Proficiency in Python, PySpark, Spark SQL, SQL, FastAPI, Docker, Docker Compose, Azure DevOps, Git, Bitbucket, Jenkins, SonarQube, and Release Management for end-to-end pipeline development and CI/CD governance.

BI & Reporting
Experience with Power BI, Tableau, Looker, BusinessObjects, and DAX/SQL logic for business intelligence, data quality automation, schema drift detection, RCA, and production support.

NoSQL & Event Streaming
Integration of Azure Cosmos DB for low-latency NoSQL ingestion including partition key design, document-index models, and RU strategies, as well as Azure Event Hubs for event-driven data architectures.

Team Leadership & Delivery
Managing teams of up to 10 engineers with task allocation, delivery oversight, technical mentoring, BI and UI team collaboration, cross-functional coordination, sprint planning, estimation, and stakeholder management.

Data Warehouse & SQL Analytics
Design and maintenance of SQL Server data warehouse and data mart solutions including stored procedures, views, CTEs, window functions, and transactions for KPI and operational reporting.

Experience

Professional experience5 - 10 years
Freelancer since2021

Languages

EnglishFluentHindiFluent

Project history

Lead AI Data Engineer

Shubh Data Security & Services (MP Government Project)
Directly managing a team of 7 data engineers, overseeing sprint delivery, code reviews, and quality. Architected and delivered a production RAG-based chatbot for HCL official documents using Azure OpenAI (GPT-4o), LangChain, ChromaDB, and Azure AI Search. Engineered a reusable Dynamic Join Framework (PySpark) eliminating repetitive logic across 40+ source integrations. Reduced Databricks compute costs by 80% through right-sized cluster configurations and partitioning strategies. Designed Automated Data Validation Framework with schema drift detection and SLA monitoring. Led architecture and code reviews, managed environment promotions via Azure DevOps and Bitbucket.

Senior Data Engineer

Syren Cloud (Johnson & Johnson - Healthcare Analytics)
Led and managed a team of 10 engineers across Dev, QA, and Production environments. Architected Medallion Architecture (Bronze, Silver, Gold) on Azure Databricks and ADF serving 9 source systems, processing 2 TB/day of global healthcare data. Integrated Azure Cosmos DB for low-latency NoSQL ingestion. Drove CI/CD governance using Azure DevOps, Jenkins, SonarQube, and Bitbucket. Supported BI and BusinessObjects reporting teams and performed production RCA and support.

Senior Analyst

Eagle Constructions (Contract - Data & Cloud Migration)
Supported data modeling and platform migration from PostgreSQL to Cloud SQL, including system testing and QA validation across all environments.

Data Engineer / Senior Data Analyst

Tata Communications Ltd (Lobo Staffing - Telecom Analytics)
Designed and maintained SQL Server data warehouse and data mart solutions for telecom KPI and operational reporting. Pulled data from multiple sources (SQL Server, SAP HANA, Excel), applied ETL transformation logic, and automated repetitive metric logic through reusable SQL templates. Built Power BI and Tableau executive dashboards and delivered predictive analytics using time series analysis, Decision Tree, and Random Forest models. Led technical Root Cause Analysis to investigate database performance bottlenecks and pipeline failures.

Senior Process Analyst

Infosys
Process documentation and transactional quality assurance.

MIS Analyst

Excellent Enterprise
MIS Analyst role at Excellent Enterprise, Gwalior, India.

Certificates

data bricks professional

Databricks

2024

azure-203

microsoft

2024


Contact form

Log in to get in touch

You need to be logged in to use the contact form.

Sign upLog in