Description
Big Data Hadoop Engineer, 6 months, £675/day (Inside IR35), ASAP Start
Data engineer is responsible for designing data pipelines and infrastructure around them. Data Pipelines might include but are not limited to adjusting the schema, filtering out unnecessary or partial data, ensuring data integrity, feature engineering and joining multiple datasets together. Data Engineers ensures that the input data gets transformed into a desired format and is fed into a system of interest - AI/ML model, report, etc - on a regular schedule using automated orchestration jobs. Additional duties of a Data Engineer might include development of automated data validation pipelines and metadata handling.
Responsibilities
1. Creation of software in accordance with norms and specifications and/or project requirements.
2. Creating integrations and data pipelines on existing and new information systems.
3. Supporting designing of a complete system.
4. Participation in the documentation of information systems and their integrations (including installation instructions and user manuals), code written by onself corresponds to the Clean Code Principles.
5. Assisting with the diagnosis and resolution of software problems.
6. Software testing using data volumes and situations similar to the production.
7. Supervising other data engineers and team members.
8. Carrying out review of the code or solution of other developers if necessary.
Skills
Extensive professional knowledge of Big Data technologies (Hadoop, Hive, Pig, Hbase, Storm, Kafka, Ambari, Hue, Hcatalog, Mahout, Spark etc.) and data engineering
Knowledge of Machine Learning Algorithms development software (Mahout, R, Spark etc.);
Deep knowledge of SQL and NoSQL database systems.
Good communication skills.
Big Data Hadoop Engineer, 6 months, £675/day (Inside IR35), ASAP Start