Description
SmartSourcing are looking for a SECURITY CLEARED Data Engineer to work on a contract based in London although some work will be remote, please only apply for this role if you hold valid SC clearance all other applications will be rejected.Data Engineer | 6 months | London | Inside IR35 |Security cleared
Technical Skills required
- Translate business requirements to ensure they are clear and robust, and can be transformed into reusable production ready code and/or effective data models;
- Define version control mechanisms and strategies for the source code of products;
- Support Quality Assurance colleagues and Data Managers responsible for the day to day testing and live running of data products and services;
- Shape solutions in a fit for purpose way; following the agreed principles and contribute to the overall Data Engineer Development life cycle;
- Perform data profiling and quality measurements;
- Designing and developing reusable metadata libraries;
- Owning the stability of new products designed, including the on-going robustness, resilience and stability of these products;
- Identifying, managing and resolving issues preventing the delivery or continuous development of productsSpecific to Big Data Developer role:
- Responsible for delivering data processing pipelines in a cloud or on premise environment using distribute file store technology.
- Responsible for creating and maintaining data pipelines using data streaming or batch transfer technology
- Designing ELT processes, data loading strategy, data archiving, data reconciliation, data manipulation, data integration, error handling and error logging mechanisms.
Knowledge and Experience required:
- Proven experience of big data engineering techniques and concepts using the Hadoop Stack (Cloudera/EMR), including data ingestion, processing and storage using HDFS, Spark, Hive and Impala.
- Extensive, hands-on experience of large complex Data Engineering projects designing and developing ETL pipelines in a cloud or on premise environment
- Experience of design and implementation of data storage, including HDFS, S3, relational and NoSQL
- Experience of developing/utilising programming and query languages eg SQL, Java, Scala
- Monitoring performance and advising on any required infrastructure or changes.
- A good understanding of data management, governance and quality frameworks, and how these integrate with big data solutions
- SmartSourcing provides services as an Employment Agency and welcomes applications from all suitably qualified people regardless of age, race, religion, disability, age, gender or sexual orientation.