Description
Data Engineer- Databricks/Delta Tables/ETL/Pyspark/Python- 110 days contract- Candidates can be based in London, Yorkshire or be based remotely if more convenientExciting contract paying between p/day via an umbrella company- Role falls Inscope of IR35
SmartSourcing are recruiting for a highly skilled Data Engineer with experience of the following-
- Experience in Databricks ETL processing using PySpark
- Experience in AWS S3 storage, Lambda, DynamoDB
- Experience in Databricks Delta lake on how to Ingest, transform, load on Delta tables in Bronze, Silver, and Gold zone
- Experience/knowledge in Airflow
- Experience/knowledge in Python
- Experience/knowledge in Metadata catalog - AWS Glue/Collibra will be preferred
Responsibilies-
-Data Vault Proof of Concept - Solution, Platform and Security Architecture and Data flows.
-Build, test, and promote data ingestion pipelines using Databricks
-Build, test, and promote metadata-driven data pipelines using Databricks to load into Data Vault with the defined model
-Build, test, and promote metadata-driven data pipelines using Databricks to read from Data vault and load into Data Mart with defined data aggregations/enrichment/
transformation/data quality rules/data lineage
-Orchestrate data pipelines using Airflow/AWS Lambda
-Document low-level designs as per the defined standards
Experience in an NHS/healthcare data environment would be highly advantageous.
Security Clearance: BPSS
CV Deadline: Monday 10th May at 2pm. Applications received after this time unfornuately cannot be considered.
- SmartSourcing provides services as an Employment Agency and welcomes applications from all suitably qualified people regardless of age, race, religion, disability, age, gender or sexual orientation.