10/06/2026 updated
100 % available
Azure DataEngineer and Databricks Engineer
Kanpur, India
Only remote
BtechAbout me
Lead Data Engineer and founding member of an early-stage startup building a Databricks-native platform for entity resolution, MDM, and data quality. Deep experience shipping scalable Lakehouse and cloud data solutions — improving processing efficiency by 35%, productivity by 20% budget.
Microsoft AzureAzure Data LakeAzure Data FactoryPySparkAzure Synapse AnalyticsDatabricks
I'm an experienced Data Engineer with a strong background in implementing scalable data solutions using Microsoft Azure. You’ve led and contributed to projects with significant impact, optimizing data pipelines, enabling real-time data processing, and driving automation. You are results-oriented, with a track record of improving operational efficiency and enhancing data accessibility.
Key Skills
- Cloud Platforms: Microsoft Azure, including Azure Data Factory (ADF), Azure Databricks, Azure Synapse Analytics, Azure Functions, Azure Data Lake Storage (Gen2), Microsoft Fabric
- Programming & Scripting: Python, PySpark, PowerShell
- Data Tools & Platforms: Kafka, SQL Server, Snowflake, Power BI
- DevOps & Automation: Azure DevOps, GitHub Actions, Logic Apps
- Data Engineering Techniques: ETL/ELT pipelines, real-time data ingestion, data validation and cleansing, query optimization, partitioning strategies
- Reporting & Visualization: Power BI dashboards, dataset automation, emailing reports using custom frameworks
Experience
Professional experience5 - 10 years
Freelancer since2020
Languages
EnglishNative speaker
Project history
• Founding member building a Databricks-native platform for entity resolution, MDM/RDM, data contracts and
data quality enforcement, distributed via the Databricks Marketplace.
• Architected end-to-end GenAI & LLM workflows on the Lakehouse — RAG pipelines, agentic systems, Vector
Search–based intelligent matching, and LLM-driven extraction from unstructured enterprise data.
• Built production pipelines using Delta Live Tables (DLT), Unity Catalog, Mosaic AI and MLflow; shipped
in-platform applications via Databricks Apps and automation through the Databricks SDK.
• Designed three-stage entity resolution (Exact → Fuzzy → AI/RAG), a configurable Delta MERGE strategy
framework, thread-safe Vector Search UDFs, and FastAPI conflict-resolution services.
• Closed seed and seed-extension funding within 7 months.