10/28/2025 updated
100 % available
Data Pipeline Engineer | Web Scraping & Automation | LLM Context Pack Developer
Surat, India
Only remote
Bachelor of Technology in Computer Science and EngineeringAbout me
I’m a Data Pipeline Engineer skilled in automating data extraction and processing using APIs and web scraping tools like Firecrawl and BeautifulSoup. Experienced in Python, CSV/JSONL structuring, and building LLM-ready datasets for RAG. I create documented, efficient workflows in Jupyter Notebooks.
JavaScriptApplication Programming Interfaces (APIs)Amazon Web ServicesAutomated Storage and Retrieval SystemsAutomationCloud ComputingWeb ScrapingExtract Transform Load (ETL)Data MiningData RetentionData RetrievalDevOpsGitHubJSONPython (Programming Language)
Data Pipeline Development
Experienced in designing end-to-end automated data pipelines for large-scale data extraction, transformation, and loading (ETL) using Python and APIs.
Web Scraping & Data Extraction
Proficient with Firecrawl API, BeautifulSoup, Playwright, and Requests for structured and semi-structured data collection from websites.
API Integration & Automation
Skilled in integrating multiple APIs, managing key rotation, and automating data ingestion workflows for high-volume processing.
LLM Data Preparation & RAG Systems
Hands-on experience creating LLM-ready datasets, context packs, and retrieval-augmented generation (RAG) pipelines.
Data Processing & Formatting
Strong command over Pandas and JSON handling for clean, consistent CSV and JSONL data formats optimized for machine learning.
Jupyter Notebook Development
Experienced in building well-documented, reproducible data workflows and technical notebooks for automation and analysis.
Cloud & DevOps Tools
Familiar with AWS, Docker, and GitHub for deploying and managing scalable data pipelines.
Programming Languages & Tools
Python, JavaScript, TypeScript, SQL, Git, Postman, VS Code.
Experience
Professional experience0 - 5 years
Freelancer since2024
Languages
EnglishFluentHindiNative speaker
Project history
Developed a custom web scraping solution to collect WeChat articles containing specific target keywords.
Automated data extraction, filtering, and delivery using Python and Firecrawl API.
Processed large volumes of text data and exported structured results in CSV and JSON formats.
Successfully delivered high-quality, verified datasets to the client within the project timeline.
Introduced role-based architecture, designed cloud functions for real-time data monitoring, and built a web application with location-based notifications.
Managed AI-Powered Medical ABDM FHIR Document Generation Project, engineered an AI integration pipeline, and deployed a full-stack application using Docker.