Description
Data Engineer
Remote/City
Insurance
AWS/Redshift/Talend/Hadoop/Big Data/Data pipelines/SQL/NoSQL databases/Apache Spark/Jira/Python
Key aspect of the role:
The role will be expected to build an excellent knowledge of my client's products with a deep understanding of how data is managed, transformed and used within the company. The data engineer will work within a product team to guide and advise on best practices around data storage, transformation and cleanliness.
The role will involve understanding my client's current data and data pipeline architecture and looking at how they can optimise this for cross-functional teams.
The data engineer will work closely with, and support, data scientists, software developers and data analysts to ensure that data is prepared and shipped in the best possible manner for their needs.
Key responsibilities:
- Work with the PO to understand how data should be managed within a product
- Build out complex data transformation pipelines and processes
- Advise on best practices around data lakes and related technologies in the cloud
- Maintain and support creation of data warehouses in the cloud
- Assemble large, complex data sets to meet functional/non-functional requirements
- Support in the quality assurance of data to ensure it is fit for purpose
- Define data access policies and help police them
- Create data tools for analytics and data science teams
Essential Technical Experience
- Excellent SQL knowledge and experience working with relational databases
- Experience in manipulating, processing and extracting data from dispirit data sources
- Experience building data pipelines, ideally in the cloud
- Strong analytical experience
- Knowledge of queues and big data stores such as Hadoop
- Experience with NoSQL databases
- Experience with AWS services such as RDS, EC2 and Redshift
- Experience with stream processing services such as Apache Spark etc.
- Ability to elicit requirements from stakeholders across the business
Desirable Technical Experience
- Experience using JIRA
- Insurance industry experience
- Experience with programming languages such as Python or Java
- Talend
Data Engineer
£600 day INSIDE IR35
Remote/City
Insurance
AWS/Redshift/Talend/Hadoop/Big Data/Data pipelines/SQL/NoSQL databases/Apache Spark