Description
Our client in the public sector is looking for a Senior Data Engineer to join them on this potentially long-term contract. Working as part of a multidisciplinary agile team, you will identify, build, modify and manage data flows between complex systems. These data flows must make use of open source technologies, be maintainable and be delivered in a timely fashion. Strong collaboration across disciplines is a must.
This will be a two year contract with an initial Statement of Works for up to 2 months.
At this time the role has been deemed as outside of IR35 however
The role
- You will lead the design and implementation of a data streaming service, API's to expose data to our users and their systems.
- Use your Big Data knowledge to optimise the current platform to ensure it is working efficiently as possible.
- Manage code versioning using Git.
- Work in an agile environment following a prioritised backlog
- You will work with Network Architects, Technical Architects, Data Engineers, DevOps Engineers and Business Analysts within the team
Skills and experience
- Designs, write and iterate code from prototype to production-ready. Understands security, accessibility and version control. Can use a range of coding tools and languages.
- Has led the introduction of a data stream service based on Java, Scala or python.
- Optimises the code to ensure processes perform optimally
- Has knowledge of specific technologies Hadoop, Flume, Sqoop, Map Reduce, Hive/Impala, Hbase, Kafka, Spark Streaming, experience with ETL tools incorporating Big Data (like Informatica, Talend etc...), Shell Scripting, Python
- Has a demonstrable understanding of how to expose data from systems (for example, through APIs), link data from multiple systems and deliver streaming services.
- Data Manipulation (SQL/No-SQL)
- DevOps (GitHub, Ansible, Continuous Integration, Test Driven Deployment)
- Produces data models and understands where to use different types of data models. Able to reverse engineer a data model from a live system.
Working arrangements
5 days a week with minimum 3 days on-site in Leeds working as part of a multi-disciplinary agile delivery team with work being specified by a work management system (JIRA)
Candidates must have SC level clearance or the ability to be cleared to SC level.
Location: Leeds
Duration: Two year contract with an initial Statement of Works for up to 2 months
Rate: between £500-£530 per Day (based on experience)