Description
Job title: Big Data Engineer
Job Type: Contract
Job Location: London, UK
Working Model: Willing to work from office at least twice a week.
Job Description:
3+ years of working experience
Essential domain expertise/skill set:
Experience of Big Data Technologies - eg Hadoop, Hive, etc
Experience of MPP (Massive Parallel Processing) databases - eg Teradata, Netezza
Challenges involved in Big Data - large table sizes (eg depth/width), even distribution of data
Experience of programming- SQL, Python, Pyspark
Data Pipelining skills - Data blending, etc
Experience working with ETL Integration Tools - SSIS, Informatica, etc
Data Management experience - eg Data Quality, Security, etc
Experience of working in a cloud environment (less relevant)
Development/Delivery methodologies - Agile, SDLC
Experience working in a geographically disparate team
Desirable:
Are an avid learner, initiative-taker, and team player
Have knowledge of current technology trends, especially in areas like AWS,AZUR, AI etc
Context and Responsibilities of Role
Context/Background:
Work as part of an Agile software delivery team; typically delivering within an Agile Scrum framework.
Will work on the full range of software engineering and solutions development activities covering requirements gathering and analysis, solutions design, software coding and development, testing, implementation and operational support.
Analyse data sources, data structure and combine data from multiple sources into a functioning dataset and can publish to visualisation tools
Review technical/functional specifications for each use case "pipeline" with technical teams for new data ingestion from source systems
Develop and maintain data catalogue and information models following the project standards
Develop solutions based on best practices by promoting reuse, leveraging design patterns, etc
Develop tools to automate and simplify manual processes such as data cleanse, etc
Collaborate with the Product Owner, Business Analyst, and visualisation developers to meet the acceptance criteria
Help lead and conduct design and code reviews
Implement Master Data Management solutions, processes and governance to support Data Management team
Troubleshoots data issues within the business and across the business and presents solutions to these issues
Analyse complex data elements and systems, data flow, dependencies, and relationships in order to contribute to & define conceptual physical and logical data models
Project Deliverables
Experience in coordinating and managing the work of a group of relatively junior data engineers
Comfortable in ensuring best practices are implemented and adhered too. For example, code reviews, testing and refactoring where appropriate
Potentially some experience in finance integration projects
Support the implementation of data engineering use cases
Map, analyse and document business requirements
Work closely with the Product Owner to completely understand client and stakeholder needs, and develop epics and user stories
Participate in Agile events, including daily scrums, sprint planning, backlog refinement and retrospectives
Drive client sprint demos alongside the scrum team
Design, implementation and maintenance of reliable and scalable data and analytics infrastructure, including design and development of industrial scale data and ML pipelines on Azure and AWS data platforms and services, building data ingestion and publishing pipelines, and development and provisioning of data sets and ML models for wide scale access
Experience and Knowledge Required Including Specific Functional, Product or Domain Expertise