Description
Responsibilities• Become proficient with our ETL processes and help ensure that they complete in a timely and accurate manner
• Develop new ETL processes to consume JSON, XML, CSV, HTML, and other structured and unstructured data formats
• Improve the flow of information between our databases and load processes
• Proactively identify problems/issues concerning data accuracy and data integrity
• Contribute to architecture, database design and query optimization
Required Qualifications
• 4+ years of experience with RDBMS (MySQL)
• Minimum 5 years of experience in a professional software development environment
• Strong analytical, problem -solving and debugging skills
• Experience with back-end programming languages (PHP, Python, Bash)
• Passion for solving problems and working with a large code base
• The ideal Candidate also has
• Experience designing and scaling big data solutions
• Strong technical architecture and design skills
• Excellent verbal and written communication skills
• Strong object-oriented design skills
Spark Developer – Expert Level - 2
Responsibilities
• Create Scala/Spark jobs for data transformation and aggregation
• Produce unit tests for Spark transformations and helper methods
• Write Scaladoc-style documentation with all code
• Design data processing pipelines
Skills
• Scala (with a focus on the functional programming paradigm)
• Scalatest, JUnit, Mockito {{ , Embedded Cassandra }}
• Apache Spark 2.x
• {{ Apache Spark RDD API }}
• {{ Apache Spark SQL DataFrame API }}
• {{ Apache Spark MLlib API }}
• {{ Apache Spark GraphX API }}
• {{ Apache Spark Streaming API }}
• Spark query tuning and performance optimization
• SQL database integration {{ Microsoft, Oracle, Postgres, and/or MySQL }}
• Experience working with {{ HDFS, S3, Cassandra, and/or DynamoDB }}
• Deep understanding of distributed systems (e.g. CAP theorem, partitioning, replication, consistency, and consensus)