Description
SC Cleared Data Engineer (Big Data, Hadoop Stack) - 3 monthsMULTIPLE ROLES BASE LOCATIONS in Hampshire/South Wales
Data Engineer | SC Cleared | Hadoop | Big Data | Python | Scala | SQL | Spark | ETL | Hive | Impala
Our government client are developing a new and exciting framework for the management of reference data which will transform the business. They are looking for multiple Data Engineers to support the development of the framework and associated processing. This will involve the development of ETL methods for a range of internal and external data sources converting these from unstructured formats to dynamic tables and views in Hive and Impala; supporting the development and transformation of the framework and the associated indexes and assisting with the development of analytical layers of data from raw HDFS files for use in producing a range of outputs.
ESSENTIAL SKILLS EXPERIENCE
Extensive proven experience of data engineering and architectural techniques, including data wrangling, data profiling, data preparation, metadata development, and data upload/download;
Proven experience of big data environments, including the Hadoop Stack (Cloudera), including data ingestion, processing and storage using HDFS, Spark, Hive and Impala;
Extensive hands-on experience of developing ETL functionality in a cloud or on-premise environment;
Experience of using tools such as python and SQL (Spark) to profile, query and structure large-volume data;
Proven experience of using Cloud Services particularly in the context of Hadoop;
Experience of developing/utilising programming and query languages eg SQL (Hive Impala specifically), Python (through Spark), Scala.
Understanding of data bases and applying data models in relational database formats.
PLEASE NOTE - Active SC Clearance is required due to the urgent nature of these roles.
IR35: Off-payroll working rules do not apply.
SmartSourcing Contact: Kelly Kan, Senior Recruitment Consultant
- SmartSourcing provides services as an Employment Agency and welcomes applications from all suitably qualified people regardless of age, race, religion, disability, age, gender or sexual orientation.