Description
On behalf of a very well respected partner in London, Clement May is currently seeking a highly capable developer to join a team building a calculation platform using the Hadoop/Spark stack to run forecasting calculations for a Leading Bank.
This is a long term project expecting to be 3 years+ from design to product handover, with a series of incremental released delivered throughout the timeline in a semi-agile manner. Working onsite they are currently working very closely with the customers own staff and product teams throughout the programme as their strategic delivery partner.
Responsibilities
- Interaction with Stakeholders, including business teams for risk and reporting as well as model developers. Requirements gathering and analysis.
- Assisting in the design of the platform by mapping business and technical requirements to capabilities of existing components and identifying gaps; design of individual components to fill the gaps identified.
- Development of software components to coordinate data and analytic operations based on the Hadoop stack (includes extension of existing components as well as addition of new ones).
- Integration of dev ops processes, both for delivery of the technology solution and models built on top of the technology; systems and Scripting work will be required.
- Assuming a level of ownership for the components you have designed and/or bui ie ensuring fitness for purpose, usability, robustness, maintainability and performance.
- Testing, deployment planning and handover.
Requirements
This position will suit a capable developer familiar with a number of different technologies.
- Software Development:
Solid foundation of Java coding skills. Experience using Scala in a real-world context is a distinct advantage. Python, R and SQL-like languages will also be useful, additional languages is a plus though not directly required. Event driven programming using promises and futures or the reactive pattern.
- Component design:
Understanding of OOP, functional, declarative and actor paradigms, their strengths, weakness and where to use each pattern to achieve simplicity, flexibility, robustness and performance. Ability to design and implement interfaces and high-level software components. Confidence discussing and challenging architecture (guidance will be provided on platform architecture and design of distributed components).
- Hadoop experience:
Real world experience using Hadoop successfully to solve a business problem is essential. They are particularly interested in Spark, HBase, Phoenix, HDFS and YARN (choosing the right components for the task at hand will count for more than having used that specific set of technologies).
- Performance Optimisation:
Understanding of key performance drivers for distributed systems. Data locality, layout and marshalling, caching, disk and memory access patterns. Empirical performance testing, profiling and tuning. Guidance in performance tuning distributed systems will be provided so expert-level skills in this area are not required; however, familiarity and understanding of the key concepts will be needed.
- Functional programming:
Use of functional paradigms to create stateless components for concurrent execution. Use of a DAG and lazy evaluation. A purist approach to functional programming is not required, this is more about understanding how functional concepts simplify design and eliminate execution bottlenecks.
- Dev ops:
Configuration of complex CI/CD pipelines in eg Jenkins, Travis etc.
- Unix:
Familiarity with the Unix command line and common utilities is distinct advantage