Description
My client are a prestigious government organisation. They are seeking Data Engineer on a contract basis, initially until the end of March. The organisation have three UK locations that candidates can work from - London, Hampshire and South Wales. However offices are currently closed due to covid, so the role will start remotely.
A live SC Clearance is needed for this position.
This is an opportunity to work in the Data Architecture (DA) Division, and more specifically within the Products Branch.
The role will involve:
1. Development of ETL methods for a range of internal ONS and external data sources converting these from unstructured formats to dynamic tables and views in Hive and Impala;
2. Supporting the development and transformation of RDMF v0.2 and the associated indexes. 4. Assisting in the development of analytical layers of data from raw HDFS files for use in producing a range of RDMF outputs.
Skills and Experience
Essential
- Extensive proven experience of data engineering and architectural techniques, including data wrangling, data profiling, data preparation, metadata development, and data upload/download;
- Proven experience of 'big data' environments, including the Hadoop Stack (Cloudera), including data ingestion, processing and storage using HDFS, Spark, Hive and Impala;
- Extensive hands-on experience of developing ETL functionality in a cloud or onpremise environment;
- Experience of using tools such as python and SQL (Spark) to profile, query and structure large-volume data;
- Proven experience of using Cloud Services particularly in the context of Hadoop;
- Experience of developing/utilising programming and query languages eg SQL (Hive Impala specifically), Python (through Spark), Scala.
- Understanding of data bases and applying data models in relational database formats.
Desirable
- Experience of UK Government Administrative Data;