Description
SC Data Engineer
9 month contract
Must hold SC and be lives pat 2022
Inside IR35
Key Responsibilities
Collaborating with key members of the Data Engineering team to develop automated coding solutions for a range of ETL, data cleaning, structuring and validation processes.
Working with large semi-structured datasets to construct linked datasets derived from multiple underlying internal and external sources as well as supporting the wider team in delivering a range of data profiles across key strategic administrative data flows.
Working with area leads across the broader Data Architecture Division providing ad-hoc coding support on a range of projects underway in Data Architecture utilising cross-government data;
Assisting in a range of ETL and warehousing design projects in migrating data from a number of Legacy environments;
Proving training and coaching to new members of staff across the Data Engineering team.
Person Specification
Good inter-personal and communication skills;
Self-starter eg problem solving and taking initiative to sort out issues;
A quick learner, able to assimilate the complexities of unfamiliar data sources and business data requirements;
Flexibility in being able to work on several projects at the same time.
Willingness to work to the requirements of the service/deliverables and not conditioned hours.
Skills and Experience
Essential
Extensive proven experience of data engineering and architectural techniques, including data wrangling, data profiling, data preparation, metadata development, and data upload/download;
Proven experience of 'big data' environments, including the Hadoop Stack (Cloudera), including data ingestion, processing and storage using HDFS, Spark, Hive and Impala;
Extensive hands-on experience of developing ETL functionality in a cloud or on-premise environment;
Experience of using tools such as python and SQL (in Spark) to profile, query and structure large-volume data;
Proven experience of using Cloud Services particularly in the context of Hadoop;
Experience of developing/utilising programming and query languages eg SQL (Hive Impala specifically), Python (through Spark), Scala.
SC-level clearance valid for at least 1 year on commencement of the contract. PLEASE NOTE APPLICATIONS NOT MEETING THIS CRITERIA AT THE APPLICATION STAGE WILL NOT BE CONSIDERED;
Understanding of data bases and applying data models in relational database formats.
Desirable
Experience of coaching and training others in programming and ETL techniques;
Experience of UK Government Administrative Data;
Certes Computing (and all of its subsidiary companies) is committed to promoting equality and diversity in its business operations.