Description
Data Engineer - Senior Data Engineer
We are looking for an experienced Data Engineer to help join our team for a long term project in Brussels! This role will initially start of as remote working but will move on site when the virus situation is safe to do so.
As a Data Engineer will play a key role preparing the infrastructure and data that will be used to deliver high quality products whilst also helping to design, develop and maintain data pipelines that will deliver insights.
The Role:
- Conceive and build data architectures
- Participate in the short/mid/long term vision of the overall system
- Execute ETL (extract/transform/load) processes from complex and/or large data sets
- Ensure data are easily accessible and that their exploitation is performing as requested, even in highly scalable circumstances
- Participate to the architecture and planning of the big data platform to optimize the ecosystem's performances
- Create large data warehouses fit for further reporting or advanced analytics
- Collaborate with machine learning engineers for the implementation and deployment of different solutions
- Ensure robust CI/CD processes are in place
Requirements:
- Demonstrable experience with big data platforms (Hadoop, Cloudera, EMR, Databricks, ...)
- Technical knowledge in:
- Data pipeline management
- Cluster management
- Workflow management ( Oozie, Airflow)
- Database management of SQL and noSQL databases
- Large file storage (HDFS, Data Lake, S3, Blob storage,..)
- Strong knowledge of Hadoop ecosystem: Hortonworks/Cloudera/EMR
- Strong knowledge of Java/Scala and Python
- Strong knowledge of Spark (Scala and Pyspark)
- Strong knowledge of CI/CD concepts
- Stream processing such as Kafka, Kinesis, Elisticsearch
- Good knowledge of a cloud environment
- High level understanding of data science concepts
- Knowledge of Data Visualisation framework like QlikSense is a plus
This is a fantastic opportunity to work on a long term project for a major, global organisation so please apply below!
Data Engineer - Senior Data Engineer