06/07/2023 updated
IA
100 % available
Big data Engineer, Big data Engineer, Team leader Data warehouse
United Kingdom
Worldwide
Application Programming Interfaces (APIs)Apache AirflowIBM AIXAmazon Web ServicesApache HTTP ServerTest AutomationMicrosoft AzureBig DataUnixCloud ComputingCloud EngineeringCloud StorageApache ImpalaComputer ProgrammingDatabases
Airflow, AWS, Apache beam, Apache, Hive, Kafka, Spark, APIs, API, test framework, Azure Data Factory, Azure Data Lake, Big Data, Cassandra, celery, cloud computing, cloud, cloud architecture, cloud platform, Cloud Storage, Cloudera, Cloudera Impala, software programming, Control-M, Control M, Cucumber, data ingestion, data pipelines, data pipeline, secure data, data security, Datawarehouse, Data warehouse, databases, DataBricks, Datastax, DAGS, Docker, dynamic data, ElasticSearch, ETL, Gitlab, GCP, Google cloud, AIX, Informatica Powercenter, Kibana, Kubernetes, legacy systems, Logstash, Machine Learning, Azure, Mongodb, Mongo, MYSQL, Open sourced, Oracle, Hyperion, Pyspark, Python, Rabbit MQ, Redis, requirements documentation, SAS, Azure SQL, SQL, Business Object, scripting, software development, Tableau, Terraform, Unix, Visual Analytics
Experience
Professional experience5 - 10 years
Freelancer since2019
Languages
EnglishGood
Project history
Technologies/Tools:
* Google cloud environment (Big query + Storage )
* Airflow (Python) Contributor in the Apache project creating a operators for GCP Open sourced
* Framework ingestion from Hortonworks to the new Datawarehouse in the cloud GCP, creating
dinamycally DAGS
* Working with Airflow in celery in GCP with Postgres+ Rabbit MQ. For Prod environments
* Azure tools for Big Data (Azure data Factory and Databricks)
Achievements and responsibilities:
* Design and building a framework for ingest all cluster of the client to the new datawarehouse in
the cloud using Airflow how orchestrator with Google cloud Storage, Data-Prep and Big-Query.
* The framework support tables partitioned, federated and materialized.
* Project working in analysis of text free of the customers using techniques of Machine learning and
API Natural Language using ElasticSearch ,Kibana, Logstash .
* Responsible for designing the config-DB, metadatastore and leading the development work and
guiding the team to use the best practices for data pipeline, data ingestion and building data
warehouse. Building of data platform is currently done on Azure platform and all the processing is
done on DataBricks using Python Spark API.
* Google cloud environment (Big query + Storage )
* Airflow (Python) Contributor in the Apache project creating a operators for GCP Open sourced
* Framework ingestion from Hortonworks to the new Datawarehouse in the cloud GCP, creating
dinamycally DAGS
* Working with Airflow in celery in GCP with Postgres+ Rabbit MQ. For Prod environments
* Azure tools for Big Data (Azure data Factory and Databricks)
Achievements and responsibilities:
* Design and building a framework for ingest all cluster of the client to the new datawarehouse in
the cloud using Airflow how orchestrator with Google cloud Storage, Data-Prep and Big-Query.
* The framework support tables partitioned, federated and materialized.
* Project working in analysis of text free of the customers using techniques of Machine learning and
API Natural Language using ElasticSearch ,Kibana, Logstash .
* Responsible for designing the config-DB, metadatastore and leading the development work and
guiding the team to use the best practices for data pipeline, data ingestion and building data
warehouse. Building of data platform is currently done on Azure platform and all the processing is
done on DataBricks using Python Spark API.
Technologies/Tools:
* Powercenter Oracle Unix Control M ETL
* Nuodb Memsql Datastax Cassandra Kafka AWS
* Cloudera Impala Hive
Achievements and responsibilities:
* POC during 7 months for the digital transformation of the bank about Big data integration with new
technologies: databases Nuodb, Mongodb, DSE Cassandra, MemSQL, and Kafka in the cloud
* Analysis of requirements documentation for software development
* Responsible and own the first line of incident analysis through techno functional
review on Oracle/ PowerCenter (Informatica)/UNIX , Involved in the capacity of an Incident Analyst
for the Retail PTR Program (PCCM) project within ISBAN, UK.
* Proposal of Big Data Data warehouse, analysis and collection of specifications of the data
warehouse to adapt of the new technology Big Data.
* Working in the ingestion team in the lake with the framework for add new information in the
environment of Big data.
* Coordinate the integration team with deployment of fixes or releases by owning the Produban change
governance (raising RFCs, presenting in Pre-Prod committees, completing checklists and HBR /
Deployment forms) for them and liasing with the Batch Open Support Systems and Batch Mainframe MVS
and Transfers team to drive deployments to completion. Marking incidents false where it is raised
from an understanding gap or misalignment to requirements.
* Manage and coordination of the labs of software during build and implementation in the environment
of Pre-production and Production with Powercenter.
* Powercenter Oracle Unix Control M ETL
* Nuodb Memsql Datastax Cassandra Kafka AWS
* Cloudera Impala Hive
Achievements and responsibilities:
* POC during 7 months for the digital transformation of the bank about Big data integration with new
technologies: databases Nuodb, Mongodb, DSE Cassandra, MemSQL, and Kafka in the cloud
* Analysis of requirements documentation for software development
* Responsible and own the first line of incident analysis through techno functional
review on Oracle/ PowerCenter (Informatica)/UNIX , Involved in the capacity of an Incident Analyst
for the Retail PTR Program (PCCM) project within ISBAN, UK.
* Proposal of Big Data Data warehouse, analysis and collection of specifications of the data
warehouse to adapt of the new technology Big Data.
* Working in the ingestion team in the lake with the framework for add new information in the
environment of Big data.
* Coordinate the integration team with deployment of fixes or releases by owning the Produban change
governance (raising RFCs, presenting in Pre-Prod committees, completing checklists and HBR /
Deployment forms) for them and liasing with the Batch Open Support Systems and Batch Mainframe MVS
and Transfers team to drive deployments to completion. Marking incidents false where it is raised
from an understanding gap or misalignment to requirements.
* Manage and coordination of the labs of software during build and implementation in the environment
of Pre-production and Production with Powercenter.
Technologies/Tools:
* SAS
* UNIX AIX
* CONTROL M
* ORACLE
Achievements and responsibilities:
* Responsible of the Datawarehouse in SAS of the group Credit Andorrá
* Implementation of a new financial management tool (Hyperion) including the development of the
inputs data feeding infrastructure from source systems
* Development, implementation and BAU maintenance of the finance data warehouse for the whole Credit
Andorra Group
* Implementation of security controls on the Group's finance datamarts
* Implementation of Visual Analytics SAS capability on the Group's finance data warehouse
* Analysis of requirements documentation for software development
* Manage and coordination of software programming team (up to 4 developers) during build,
implementation and testing phases being the main IT point of contact with the project stakeholders
* Tracking and resolution of IT incidents identified in Production
* Responsible of all System informational of the Group Credit Andorra (Production, integration,
Develop)
* SAS
* UNIX AIX
* CONTROL M
* ORACLE
Achievements and responsibilities:
* Responsible of the Datawarehouse in SAS of the group Credit Andorrá
* Implementation of a new financial management tool (Hyperion) including the development of the
inputs data feeding infrastructure from source systems
* Development, implementation and BAU maintenance of the finance data warehouse for the whole Credit
Andorra Group
* Implementation of security controls on the Group's finance datamarts
* Implementation of Visual Analytics SAS capability on the Group's finance data warehouse
* Analysis of requirements documentation for software development
* Manage and coordination of software programming team (up to 4 developers) during build,
implementation and testing phases being the main IT point of contact with the project stakeholders
* Tracking and resolution of IT incidents identified in Production
* Responsible of all System informational of the Group Credit Andorra (Production, integration,
Develop)