11/30/2022 updated
MH
100 % available
Data engineer
le pre saint gervais, France
Worldwide
Java (Programming Language)Agile MethodologyApache AirflowAtlassian ConfluenceAtlassian JiraMicrosoft AzureBig DataBigTable ImplementationsGoogle BigQueryInformation EngineeringExtract Transform Load (ETL)Apache HadoopMapReduceApache HBaseApache Hive
Méthode agile, Agile, Airflow, Flink, Hbase, Hadoop, Hive, Spark, Backend, Big Data, Technologies Big Data, Bigquery, Bigtable, Confluence, Dataflow, transformation de données, entrepôt de données, Databricks, ETL, prévention de la fraude, Gitlab-ci , Google Cloud Platform, Grafana, Data Engineer, Jira, java, Jenkins, Machine learning, Map-reduce, Azure, SQL Server, Pandas, PowerBI, Python, SQL, Scala, Génie logiciel, Terraform, Vertica
Experience
Professional experience0 - 5 years
Languages
EnglishFrenchNative speaker
Project history
The projects concern the management of the Datapipeline of reporting and analysis / ML training:
+ Design and implementation of ETL (data transformation pipelines) to aggregate advertising data received from client
websites, and store them in our Clickhouse data warehouse as well as on Hbase and maintenance of existing pipelines
under Spark, java, Hbase, Hadoop, Clickhouse
+ Migration of the company's main production datapipeline from Hadoop to Google Cloud Platform (GCP),
implementation of DEV, PREPROD and PROD environments on the GCP cloud, and implementation of dashboards and
metrics to monitor the migration under Dataflow , Bigtable, Bigquery, Java, Python, Terraform
+ Implementation of security rules and access rights / roles associated with different service accounts and specific
configurations on GCP under Terraform
+ Management of the Clickhouse Datawarehouse for real-time and historical data, creation of new tables and update of
existing tables which serve as a data source for customer reports under Clickhouse, SQL, SQL Server
+ Design and development of new features and release for each Sprint
+ Maintenance and improvement of Gitlab-ci pipelines
Methodology: Agile
Technical environment:
+ Dataflow, Bigtable, Bigquery, Hadoop, Java, Terraform, Flink, Scala, Spark, Hbase, Clickhouse, Python, Confluence, Jira,
Gitlab-ci, SQL, Snowflake, SQL Server
+ Design and implementation of ETL (data transformation pipelines) to aggregate advertising data received from client
websites, and store them in our Clickhouse data warehouse as well as on Hbase and maintenance of existing pipelines
under Spark, java, Hbase, Hadoop, Clickhouse
+ Migration of the company's main production datapipeline from Hadoop to Google Cloud Platform (GCP),
implementation of DEV, PREPROD and PROD environments on the GCP cloud, and implementation of dashboards and
metrics to monitor the migration under Dataflow , Bigtable, Bigquery, Java, Python, Terraform
+ Implementation of security rules and access rights / roles associated with different service accounts and specific
configurations on GCP under Terraform
+ Management of the Clickhouse Datawarehouse for real-time and historical data, creation of new tables and update of
existing tables which serve as a data source for customer reports under Clickhouse, SQL, SQL Server
+ Design and development of new features and release for each Sprint
+ Maintenance and improvement of Gitlab-ci pipelines
Methodology: Agile
Technical environment:
+ Dataflow, Bigtable, Bigquery, Hadoop, Java, Terraform, Flink, Scala, Spark, Hbase, Clickhouse, Python, Confluence, Jira,
Gitlab-ci, SQL, Snowflake, SQL Server
The projects mainly revolve around the prevention and monitoring of advertising fraud:
+ Design and implement an application that assists in real-time automated bot prevention for online advertisements
using an external anti-fraud service (Whiteops, ETL, Airflow, Python, Pandas)
+ Migration of the team's datapipeline from Hadoop to Azure Databricks, implementation of DEV, PREPROD and PROD
environments, and implementation of dashboards and metrics to monitor the migration under Spark, Azure Databricks,
Scala, Terraform
+ Lead efforts across multiple teams to display important anti-fraud metrics on customer reports (Spark, Hive, Vertica)
+ Suggest, design and implement a better Data-pipeline architecture which improved execution time and overall
memory/CPU usage for many applications used for reporting and fraud prevention (Map-reduce, JAVA , Spark, Scala)
+ Miscellaneous tasks: Implementation and improvement of Dashboards (Grafana, Redash and PowerBI) for better
monitoring of fraud related to online advertising, writing of Runbooks for alert management
Methodology : Agile
Technical environment:
+ Airflow, Azure, Databricks, Python, Pandas, Hadoop, Map-reduce, JAVA, Spark, Hive, Scala, Grafana, Redash, Vertica,
Jenkins, Confluence, Jira, PowerBI
+ Design and implement an application that assists in real-time automated bot prevention for online advertisements
using an external anti-fraud service (Whiteops, ETL, Airflow, Python, Pandas)
+ Migration of the team's datapipeline from Hadoop to Azure Databricks, implementation of DEV, PREPROD and PROD
environments, and implementation of dashboards and metrics to monitor the migration under Spark, Azure Databricks,
Scala, Terraform
+ Lead efforts across multiple teams to display important anti-fraud metrics on customer reports (Spark, Hive, Vertica)
+ Suggest, design and implement a better Data-pipeline architecture which improved execution time and overall
memory/CPU usage for many applications used for reporting and fraud prevention (Map-reduce, JAVA , Spark, Scala)
+ Miscellaneous tasks: Implementation and improvement of Dashboards (Grafana, Redash and PowerBI) for better
monitoring of fraud related to online advertising, writing of Runbooks for alert management
Methodology : Agile
Technical environment:
+ Airflow, Azure, Databricks, Python, Pandas, Hadoop, Map-reduce, JAVA, Spark, Hive, Scala, Grafana, Redash, Vertica,
Jenkins, Confluence, Jira, PowerBI