Description
Job Purpose and primary objectives: To fulfil Python Engineer role
Key responsibilities (please specify if the position is an individual one or part of a team):
Proficiency in Python, Pyspark and SQL programming for big data processing.
Solid understanding of data store and processing solutions for semi structured and relational datasets in AWS cloud platform(preferably Parquet, XML, Json and CSV).
Experience in software development practices such as version control (eg Github), CI/CD development, infrastructure as code (preferably cloudformation)
Good understanding on Datamodelling, data profiling, implementing complex stored Procedures and ETL concepts.
Experience with integration of data from multiple data sources
Key Skills/Knowledge:
Experience in engineering, optimising, orchestrating and debugging high performance data pipelines in terabyte scale data platforms(batch and streaming)
Technologies used: AWS Glue, Athena, Redshift, Lambda, Snowflake, Python, Pyspark, EC2, Kafka, AWS S3