Big Data Engineer - Freelance/Contract - Belgium

Job type:
on-site
Start date:
ASAP
Duration:
3 months rolling
From:
Robson Bale Ltd
Location:
BE
Published at:
01/22/2021
Country:
flag_no Belgium
Project ID:
2033925

Warning
This project has been archived and is not accepting more applications.
Browse open projects on our job board.

Big Data Engineer - Freelance/Contract - Belgium

Robson Bale are looking for a Big Data Engineer to come on board for an initial 3 month contract in Belgium.

3 months, market rate

Big Data Engineer

Job Description:

Individuals within the Big Data Engineer role ensure that data pipelines are scalable, repeatable, and secure, and can serve multiple users. They help facilitate getting data from a variety of different sources, getting it in the right formats, assuring that it adheres to data quality standards, and assuring that downstream users can get that data quickly. This role usually functions as a core member of an agile team.

These professionals are responsible for the infrastructure that provides insight from raw data and handles diverse sources of data seamlessly. They enable big data and batch/Real Time analytical solutions that leverage emerging technologies. Additional responsibilities include developing prototypes and proof of concepts for the selected solutions and implementing complex big data projects with a focus on collecting, parsing, managing, analysing and visualizing large sets of data using multiple platforms. They understand how to apply technologies to solve big data problems and to develop innovative big data solutions.

The Big Data Engineer is a technical job that requires substantial expertise in a broad range of software development and programming fields. These professionals have knowledge of data analysis, end user requirements analysis, and business requirements analysis to develop a clear understanding of the business needs and to incorporate these needs into technical solutions. They have a solid understanding of physical database design principles, and the system development life cycle. These individuals must work well in a team environment.

Primary Responsibilities

  • Designing, developing, constructing, testing and maintaining the complete data management & processing systems - Data Pipelines
  • Aggregate & Transform raw data coming from a variety of data sources to fulfil the functional & non-functional business needs - Data Transformation
  • Discovering various opportunities for data acquisitions and exploring new ways of using existing data - Data Ingestion
  • Creating of data models to reduce system complexity and hence increase efficiency and reduce cost - Data Architecture & Models
  • Performance optimization & monitoring: automating processes, optimizing data delivery & re-designing the complete architecture to improve performance
  • Proposing ways to improve data quality, reliability & efficiency of the whole system - Data Quality
  • Creating a solution by integrating a variety of programming languages and tools together - Data Value
  • Competencies

    Store:

    • Datamodelling
    • Data Architecture
    • Big Data Framework/Hadoop: HDFS, Squid, Spark, Conda, Yarn and MapReduce
    • NoSQL Databases: Cassandra, Hbase, MongoDB

    Access & Transport - Connectivity:

    • ETL (Extract, Transform, Load): Informatica
    • Big Data Framework/Hadoop: Flume & Sqoop, Yarn, Zookeeper

    Enrich:

    • Real Time processing framework - Apache Spark
    • Big Data Framework/Hadoop: PIG, Hive
    • SQL and NoSQL
    • Machine learning (nice to have): Python & algorithms

    Provision:

    • Workflow
    • Programming: Java, Python (and optionally Scala)

    Development methodologies:

    • Agile: Safe or Spotify
    • DataOps

    Ancillary capabilities:

    • Very strong communication skills
    • Problem Solving
    • Teamwork
    • Innovation

    3 months, market rate

    Big Data Engineer - Freelance/Contract - Belgium