Description
Job Spec
Looking for a data engineer who has experience in using Spark, Scala, Azure services like Azure Data Factory, Logic apps and Databricks. The hire will be responsible for transforming and optimizing the data and data pipeline architecture, as well as optimizing data flow and collection for cross functional teams.
Responsibilities for Data Engineer:
- Transform data using Spark methods and write corresponding Scala tests for it
- Create and maintain optimal data pipeline in Azure Data Factory and Logic apps
- Assemble large, complex data sets that meet functional/non-functional business requirements.
- Analyze the bug or any new requirements using Azure databricks
- Actively participate in the daily scrum and sprint demo
- Advanced working SQL knowledge and experience working with relational databases
- Nice to have experience with stream-processing systems: Storm, Spark-Streaming