Description
Data Engineer (streaming) - London - 6months - £750pd (inside)
Your expertise:
- Experience of building streaming pipelines and ensuring scalability, resilience and security.
- Hands on experience of building streaming platforms end-to-end (ie Setting up Kafka clusters, producers, brokers, consumers - connecting Spark - Transforming - Persisting data in HDFS, S3, RDBMS, putting Operational DB in place such as Hbase, Cassandra
- Hands-on experience of architecting, building event-driven, loosely coupled distributed streaming applications.
- Experience in data and cloud technologies using Microsoft Azure (Azure Stream Analytics with Event Hub and Blob Storage), Spark, Kafka, HBase, Cassandra, alternatively experience with AWS (Kinesis, SNS, SQS, S3, EMR, GLUE) or Google GCP (PubSub, Dataflow, BigQuery, DataProc)
- Experience developing containerised applications that rely on many web APIs and other protocols. (Docker, Kubernetes, Service Mesh Istio)
- Proficiency in Java (Spring boot microservices), Scala and Python.
- Experience packaging, building deploying applications in production. DevOps, CI/CD, Git, Jenkins or similar tooling, Terraform, Ansible.
- Enthusiasm and ability to pick up new technologies as needed to solve your problems.
- Experience or interest in getting involved with data infrastructure like Azure, on-premise.
- Bachelors or Masters in Computer Science, Applied Mathematics or a related technical field.
- Strong verbal and written communication skills.
Nice to have
- Knowledge of and experience in one or more product areas across FX, Rates, Credit, Equities within the financial ecosystem is a huge plus.
- Experience with Pydata stack (Pandas, Numpy, Matplotlib, scikit-learn etc.) is a plus.
- Track record of solving real-world problems using Data Science, ML (NLP, Recommenders etc.), and statistical techniques and deploying ML models.
- Experience in setting up and using data science platforms like Dataiku, Databricks or Alteryx etc.
- Experienced in containerized application deployment eg with Docker using Kubernetes, Terraform and Ansible.
- Working knowledge of UNIX or similar and bash Scripting.
- Experience working with KDB.