09/15/2026 updated
partially available
info: Available evenings and weekends
Data Engineer – Distributed Processing & Continuous Ingestion
Mesnil-Mauger, France
Only remote
Advanced Master in AI, Expert Data & MLOps – Telecom ParisAutomatic Data Processing EquipmentApache AirflowContinuous DeliveryContinuous IntegrationData IntegrationData Processing SystemsFlow Control (Data)Python (Programming Language)DataOpsSpinnakerDatabase EnginesData StreamingSystems IntegrationRust (Programming Language)Automated Data Processing (ADP)
Distributed & Streaming Processing
Expertise in building and operating distributed data pipelines using Spark (PySpark) and Kafka, with hands-on experience refactoring historisation pipelines for reliability and scalability on production workloads handling 500K+ daily transactional records.
Cloud Infrastructure & Containerisation
Proficiency in provisioning and managing cloud infrastructure on AWS and Azure, combined with containerised deployments using Docker, Kubernetes, OpenShift, and Terraform, including full CI/CD pipeline management.
Data Ingestion & Pipeline Engineering
Design and implementation of continuously running ingestion systems collecting from heterogeneous external sources, including retry, backoff, and rotation logic, as well as storage and modelling layers with validation and reconciliation between sources.
Programming Languages
Strong command of Python, SQL, Rust, and JavaScript for data engineering, automation scripting, and pipeline development across multiple production environments.
Storage & Databases
Experience working with Snowflake, PostgreSQL, and MongoDB for structured and unstructured data storage in production data platforms.
Observability & Monitoring
Use of Grafana and MLflow for pipeline monitoring, performance profiling, and ensuring failed runs are visible and actionable within short timeframes.
AI & LLM Integration
Application of LLMs, Agentic AI, RAG, and LangChain in data enrichment pipelines, including building LLM-based extraction stages and classification layers on Azure.
Workflow Orchestration
Orchestration of data workflows using Apache Airflow to schedule, monitor, and manage complex pipeline dependencies in production environments.
Quality Engineering & Test Automation
Development of 180+ JavaScript automation scripts for regression testing, migration of scripts using the Page Object pattern, and management of the complete incident lifecycle in collaboration with development teams.
Expertise in building and operating distributed data pipelines using Spark (PySpark) and Kafka, with hands-on experience refactoring historisation pipelines for reliability and scalability on production workloads handling 500K+ daily transactional records.
Cloud Infrastructure & Containerisation
Proficiency in provisioning and managing cloud infrastructure on AWS and Azure, combined with containerised deployments using Docker, Kubernetes, OpenShift, and Terraform, including full CI/CD pipeline management.
Data Ingestion & Pipeline Engineering
Design and implementation of continuously running ingestion systems collecting from heterogeneous external sources, including retry, backoff, and rotation logic, as well as storage and modelling layers with validation and reconciliation between sources.
Programming Languages
Strong command of Python, SQL, Rust, and JavaScript for data engineering, automation scripting, and pipeline development across multiple production environments.
Storage & Databases
Experience working with Snowflake, PostgreSQL, and MongoDB for structured and unstructured data storage in production data platforms.
Observability & Monitoring
Use of Grafana and MLflow for pipeline monitoring, performance profiling, and ensuring failed runs are visible and actionable within short timeframes.
AI & LLM Integration
Application of LLMs, Agentic AI, RAG, and LangChain in data enrichment pipelines, including building LLM-based extraction stages and classification layers on Azure.
Workflow Orchestration
Orchestration of data workflows using Apache Airflow to schedule, monitor, and manage complex pipeline dependencies in production environments.
Quality Engineering & Test Automation
Development of 180+ JavaScript automation scripts for regression testing, migration of scripts using the Page Object pattern, and management of the complete incident lifecycle in collaboration with development teams.
Experience
Professional experience0 - 5 years
Freelancer since2025
Languages
EnglishFluentFrenchNative speaker
Project history
Owned the full data platform for a multi-country operation: provisioned the cloud infrastructure, built the ingestion layer, and ran it in production alone. Built ingestion pipelines handling 500K+ daily transactional records from an ERP and heterogeneous external sources. Handled monitoring, on-call, and incident response. Designed the storage and modelling layers with validation and reconciliation between sources.
Refactored a PySpark historisation pipeline for reliability and scalability on a distributed workload running against production data. Profiled and resolved performance bottlenecks, reducing processing time and eliminating recurring failures. Operated pipelines and dashboards consumed daily by downstream teams. Built ML models on operational data extracted from SAP systems.
Built a continuously running ingestion system collecting from external platforms, decoupling capture from processing. Designed retry, backoff, and rotation logic to keep a never-ending feed alive against rate limiting, IP bans, and authentication changes. Built the Azure processing and enrichment path including an LLM-based extraction stage and a classification layer.