Spark Developer

North Holland, Amsterdam  ‐ Onsite
This project has been archived and is not accepting more applications.
Browse open projects on our job board.

Description

Responsibilities
• Create Scala/Spark jobs for data transformation and aggregation
• Produce unit tests for Spark transformations and helper methods
• Write Scaladoc-style documentation with all code
• Design data processing pipelines
Skills
• Scala (with a focus on the functional programming paradigm)
• Scalatest, JUnit, Mockito {{ , Embedded Cassandra }}
• Apache Spark 2.x
• {{ Apache Spark RDD API }}
• {{ Apache Spark SQL DataFrame API }}
• {{ Apache Spark MLlib API }}
• {{ Apache Spark GraphX API }}
• {{ Apache Spark Streaming API }}
• Spark query tuning and performance optimization
• SQL database integration {{ Microsoft, Oracle, Postgres, and/or MySQL }}
• Experience working with {{ HDFS, S3, Cassandra, and/or DynamoDB }}
• Deep understanding of distributed systems (e.g. CAP theorem, partitioning, replication, consistency, and consensus)
Start date
2019
From
Adroit People Ltd
Published at
17.10.2019
Contact person:
Hema S
Project ID:
1837030
Contract type
Freelance
To apply to this project you must log in.
Register