09/12/2024 updated
BB
100 % available
Data Engineer, Data Engineer, Data Engineer
Bertrange, Luxembourg
Worldwide
AirflowAmazon Web ServicesData Analysis
Airflow, Amazon Redshift, AWS, AWS services, Ansible, HBase, Hadoop, Hive, Kafka, Apache Spark, Spark, Azure Data Factory, Bash, cloud technologies, CLOUD ARCHITECTURE, cloud migration, Cloudera, Impala, PROGRAMMING, continuous delivery, CI/CD, data analysis, data architecture, Data Architect, data cleansing, data ingestion, Data Integration, data integrity, data extraction, Data Pipeline, data pipelines, Data Processing, data security, secure data, data flows, data systems, data system, data solutions, data transfer, data validation, data warehouse, Databricks, DEVOPS, Docker, ElasticSearch, ETL, GitHub, Google Analytics, Groovy, HDFS, DATA ENGINEERING, Computer Science, Jenkins, Kubernetes, Microsoft Azure, Azure, Azure Cloud, Azure services, Azure DevOps, Migration Strategy, NoSql, performance optimization, Python, real-time data, RDBMS, system reliability, Azure SQL Database, Azure SQL Server, SCRIPTING, Snowflake, Teradata, Terraform
Languages
EnglishGood
Project history
Project Objective:
Strategic Data Solutions : Spearheaded the growth of the data team, leading the
design of sustainable solutions for the data warehouse and the development of
robust data ingestion and scheduling applications. Enhanced security measures and
orchestrated a strategic migration from on-premise Airflow to a Kubernetes-based
Airflow on AWS, coupled with a transition from on-prem Exasol to Amazon Redshift.
Market and Customer Energy Data Integration : Pioneered the creation of a
comprehensive data product to provide a consolidated view of energy trading,
correlating energy purchased on the market with the distribution to customers.
Key Contributions:
Designed and executed scalable data architectures for the AWS cloud, ensuring a
seamless transition of existing on-premise systems to AWS services.
Conducted comprehensive data analysis and business requirement assessments to
inform the migration plan, emphasizing data integrity, volume, and security.
Developed and optimized data workflows on AWS, laying the groundwork for
efficient AWS migration.
Led cross-functional collaboration to facilitate the data transition, ensuring
stakeholder alignment and post-migration data validation.
Strategic Data Solutions : Spearheaded the growth of the data team, leading the
design of sustainable solutions for the data warehouse and the development of
robust data ingestion and scheduling applications. Enhanced security measures and
orchestrated a strategic migration from on-premise Airflow to a Kubernetes-based
Airflow on AWS, coupled with a transition from on-prem Exasol to Amazon Redshift.
Market and Customer Energy Data Integration : Pioneered the creation of a
comprehensive data product to provide a consolidated view of energy trading,
correlating energy purchased on the market with the distribution to customers.
Key Contributions:
Designed and executed scalable data architectures for the AWS cloud, ensuring a
seamless transition of existing on-premise systems to AWS services.
Conducted comprehensive data analysis and business requirement assessments to
inform the migration plan, emphasizing data integrity, volume, and security.
Developed and optimized data workflows on AWS, laying the groundwork for
efficient AWS migration.
Led cross-functional collaboration to facilitate the data transition, ensuring
stakeholder alignment and post-migration data validation.
Project Objective:
Azure Cloud Data Engineering : Devised a sophisticated solution for data ingestion
into Azure SQL Server using Azure Databricks and Azure Data Factory. Implemented
monitoring, security, and scheduling enhancements using Azure Data Factory.
Key Contributions:
Transitioned key data processes to the Azure platform.
Oversaw ETL processes and integrated Azure Data Factory for efficient data
extraction and management.
Utilized PySpark and Databricks within Azure, preparing for the Azure-based data
transformations.
Enabled advanced marketing analytics and performance forecasting through
integration with Google Analytics.
Azure Cloud Data Engineering : Devised a sophisticated solution for data ingestion
into Azure SQL Server using Azure Databricks and Azure Data Factory. Implemented
monitoring, security, and scheduling enhancements using Azure Data Factory.
Key Contributions:
Transitioned key data processes to the Azure platform.
Oversaw ETL processes and integrated Azure Data Factory for efficient data
extraction and management.
Utilized PySpark and Databricks within Azure, preparing for the Azure-based data
transformations.
Enabled advanced marketing analytics and performance forecasting through
integration with Google Analytics.
Project Objective:
Platform Enhancement and Migration Leadership : Led the upgrade and
maintenance of the Cloudera Platform and data warehouse, ensuring optimal data
querying via HDFS and Impala.
Design and Solution Architecture : Designed the architecture for the migration to
GCP, defended the proposed solution to the steering committee, and was
instrumental in the decision-making process for adopting technologies like BigQuery
and Terraform within the project scope.
Key Contributions:
Migration Strategy and Execution : Orchestrated the migration project, applying
technical expertise and strategic vision to ensure alignment with business
objectives and a smooth transition to GCP.
Data Process Optimization : Developed real-time data ingestion processes on GCP,
enhancing data workflow efficiency and setting the foundation for scalable data
solutions.
CI/CD and Monitoring Architect : Architected and implemented CI/CD pipelines,
incorporating DBT for data build tooling, and established monitoring protocols that
guarantee system reliability and performance optimization.
Security and Compliance: Championed security and compliance measures, ensuring
the data architecture adhered to industry standards and provided a secure data
environment.
Platform Enhancement and Migration Leadership : Led the upgrade and
maintenance of the Cloudera Platform and data warehouse, ensuring optimal data
querying via HDFS and Impala.
Design and Solution Architecture : Designed the architecture for the migration to
GCP, defended the proposed solution to the steering committee, and was
instrumental in the decision-making process for adopting technologies like BigQuery
and Terraform within the project scope.
Key Contributions:
Migration Strategy and Execution : Orchestrated the migration project, applying
technical expertise and strategic vision to ensure alignment with business
objectives and a smooth transition to GCP.
Data Process Optimization : Developed real-time data ingestion processes on GCP,
enhancing data workflow efficiency and setting the foundation for scalable data
solutions.
CI/CD and Monitoring Architect : Architected and implemented CI/CD pipelines,
incorporating DBT for data build tooling, and established monitoring protocols that
guarantee system reliability and performance optimization.
Security and Compliance: Championed security and compliance measures, ensuring
the data architecture adhered to industry standards and provided a secure data
environment.