Description
Data Engineer. Urgently looking for x2 Data Engineers who have Data Ingestion and Transformation skills.
Technical experience looked for inc:
Kafka, Spark, Hadoop, Hive, Scala, Java, R, Python, Teradata, Sqoop, PostgreSQL, ElasticSearch, Git and Jira.
You need to be skilled at:
Manipulating data with a variety of tools and programming languages, eg SQL, Java, Python. Building streaming & batch applications with technologies including: JMS, Kafka, Kinesis and Spark Streaming.
Working with data formats including: JSON, XML, Flatfiles, Sequence, Avro, Parquet, etc. Cloud computing and related challenges around security, bandwidth, etc.
Understanding statistics and data science to work closely with business analysts.
Using and potentially building Service oriented architecture and APIsUnit testing including creating test data. Version control. Relational and non-relational databases and when to use them
My client are looking for a Data Engineer to help provision valuable business data to their Data Platform. You will work closely with Data Platform Engineers, Data Scientists, and InfraDev Engineers across multiple platforms and technologies to create a highly scalable, secure and performant data platform.
As a Data Engineer, you will:
Build capabilities to efficiently integrate new data sources into a common platform.
Build data ingestion/integration pipelines for large scale Real Time (streaming) and batch applications.
Develop necessary transformation logic/event logic for data processing across pipeline, taking into account the business needs.
Care about your craft, be aware of new technologies and be keen to use them when appropriate (eg Continuous Integration, Continuous Deployment). Looking for Engineers who feel comfortable in feeding insights back to the product team - to continuously improve the platform, play an active contributing part in an agile development team.
Must be diligent and positive about tracking (JIRA), system monitoring, security, auditing and other ways of ensuring the quality of the platform and the processes around it.