Description
Position: Sr. Big-data Developer/Lead
Location: Dublin, Ireland
Duration: Contract
RequiredSkills:
- Hadoop
- Scala programming
- Apache Spark
Good understanding of
- Data Formats like Parquet, Avro, Json, CSV, XML, OR, Hadoop Architecture, Distributed parallel processing concepts
Hands on technical skill set.
- Strong experience in Java, Python Sqoop, Scala, Devops, Jenkins, Hbase etc.,
The key accountabilities/responsibilities are as follows:
- General Programming (regex, functions, loops, data structures). Big Data architecture mind-set eg data can be de-normalised on Hadoop it is not such an issue because storage is cheap
- Data formats and Big Data formats (Textfile, avro, json, XML, etc.). SPARK - Scala (could be Java or python Iguess). Bash/Shell Scripting (Job scheduling) - this could be python and is OSagnostic so might be better choice.
- Hadoop File System Shell (FileMovement). Solr understanding of Javascript (host job configuration details). ETH (Jenkins/NEXUS/Maven/GIT/JIRA).
- High level Groovy Scripting (Usedprimarily for Jenkins deployments) (our pipeline is still in design will have completed in next few weeks).
- Hive/BigSQL (these are different to Teradata) (TEXTFILE, SEQUENCEFILE, RCFILE, AVRO, ORCFILE, PARQUET).
- SQL Hbase (will need in Real Time solutions). Kafka (will need in Real Time solutions) Cyber arc (security). Remedy (need to raise tickets before going live with changes) Python/Notebooks Data profiling solution currently