09/08/2024 updated
SA
100 % available
Data Engineer
Hyderabad, India
Only remote
B.EAgile MethodologyAmazon S3Unit TestingMicrosoft AzureCloud ComputingDatabasesContinuous DeliveryContinuous IntegrationData TransmissionsData Definition LanguageInformation EngineeringData Manipulation LanguagesData TransformationData MiningData Warehousing
agile development methodology, Agile, AWS S3, CLOUD, cloud integration, continuous deployment, Continuous Delivery, Continuous integration, CI/CD, DDL, Data Lake, DML, data Extraction, data flow, data transformation, data transfer, data warehouse, DataOps, databases, DevOps, GIT, Data Engineer, IAAS, JSON, JINJA, Macros, Microsoft Azure, azure DevOps, Azure cloud services, Azure, Azure VM, SQL Server, Visual Studio, Windows, My SQL, Operating Systems, PAAS, raw data, RDBMS, SQL, SQL queries, SQL Servers, scripting, Scripting Languages, semi-structured data, Snowflake, debug, application development, version control, TCL, unit tests, Shell, YAML
Experience
Professional experience5 - 10 years
Freelancer since2023
Languages
EnglishFluent
Project history
Project: 1. AD PORT GROUPS
Client: Abu Dhabi
Role : Data Engineer
Environment: DBT, DataOps, SQL, Snowflake
Description:
AD Ports Group is embarking on digital transformation initiatives, this program will drive ADP
towards implementing a world class reporting & analytics capability. ADP has defined the data and
analytics capability that needs to be address as part of current gap analysis and this initiative focus on
addressing on building the data platform capability. As part of assessment below are areas of interest
which ADP has decided to address and modernize the platform to derive insights and monetize the data
* Data Lake, Governance and Real time Integration
Roles And Responsibilities:
* Created Snowflake cloud integration with Data build Tool (DBT) for continuous data transforming
for business analysis purpose.?
* Responsible for continuous integration and continuous deployment CI/CD of the data by using
DataOps pipeline for maintaining data consistency frequently.?
* Maintained Jinja Templating and Macros in DBT to avoid inconsistence in data by using SQL quires
for reusable purpose.?
* Responsible for covert client raw data (source data) into structured (organized) data to transfer
respective snowflake warehouse and schemes through DBT and DataOps by using SQL select
statements.?
* Creating merge requests and taking approvals from the client end for seamless data flow in
DataOps pipeline for the business analysis purpose.?
* Played a crucial role to debug DBT errors in day-to-day activities while doing data transfer.?
* Designed and documented the Architecture of Data Build Tool (DBT) projects.?
*
* Extensively worked in data Extraction, Transformation and Loading from source totarget system
using Integration Services.?
Client: Abu Dhabi
Role : Data Engineer
Environment: DBT, DataOps, SQL, Snowflake
Description:
AD Ports Group is embarking on digital transformation initiatives, this program will drive ADP
towards implementing a world class reporting & analytics capability. ADP has defined the data and
analytics capability that needs to be address as part of current gap analysis and this initiative focus on
addressing on building the data platform capability. As part of assessment below are areas of interest
which ADP has decided to address and modernize the platform to derive insights and monetize the data
* Data Lake, Governance and Real time Integration
Roles And Responsibilities:
* Created Snowflake cloud integration with Data build Tool (DBT) for continuous data transforming
for business analysis purpose.?
* Responsible for continuous integration and continuous deployment CI/CD of the data by using
DataOps pipeline for maintaining data consistency frequently.?
* Maintained Jinja Templating and Macros in DBT to avoid inconsistence in data by using SQL quires
for reusable purpose.?
* Responsible for covert client raw data (source data) into structured (organized) data to transfer
respective snowflake warehouse and schemes through DBT and DataOps by using SQL select
statements.?
* Creating merge requests and taking approvals from the client end for seamless data flow in
DataOps pipeline for the business analysis purpose.?
* Played a crucial role to debug DBT errors in day-to-day activities while doing data transfer.?
* Designed and documented the Architecture of Data Build Tool (DBT) projects.?
*
* Extensively worked in data Extraction, Transformation and Loading from source totarget system
using Integration Services.?
Project: 2. Insurance
Client: Millward Agency - US
Role : Data Associate
Environment: DBT, Snowflake and Azure DevOps
Description:
Millward Agency is committed to providing business owners with the insurance coverage they need
to weather any financial storms that can come along. We want to be there for you every step of the way,
from securing proper coverage, to managing claims. In addition to providing great service, we're
committed to offering clients the lowest premiums possible, by eliminating expensive frills, and passing
the savings on to our customers. The Project is mainly to develop and maintain the insurance data of the
customers and Finance of Current Year and Previous Year. They shared their data in the semi-structured
JSON format, and we did required modifications before loading into target tables by creating SQL select
statements through DBT into respective snowflake databases and maintained version control of the data
by integrating with Azure DevOps with Data build tool.
Roles and Responsibilities:
* Primarily responsible for creating SQL select statements for data transfer to snowflake.?
* Responsible for creation of warehouses, databases and schemas in snowflake as per the client
suggestions.?
* Involved in writing SQL statements for data transfer from source tables to target tables by sing
simple and complex SQL joins.?
* Create storage accounts for dev & QA teams for their application development and
testing purpose. And Creation of Azure VM's, Azure NIC, Azure Virtual networks, and
provide access, users and managing entire Azure infrastructure services.?
* Scale up and scale down various services like VMs, SQL Servers.?
* Automated Continuous Integration builds, nightly builds, deployments and unit tests across
multiple environments (DEV, QA, Production).?
* Creating repositories, new users for GIT and give the permissions to users to access repository.*
* Work with Development Teams and Testing Team to establish a build schedule, execute the builds
and troubleshoot build failures.*
* Good exposure of agile development methodology in designing and implementation of Continuous
Integration and Delivery (CI/CD) methodology.*
* Build and release definition configuration and enabling triggers for CI and CD.*
* High level of understanding and experience of Continuous Integration, Continuous Delivery (CI/CD),
Agile and other DevOps related methodologies.*
Client: Millward Agency - US
Role : Data Associate
Environment: DBT, Snowflake and Azure DevOps
Description:
Millward Agency is committed to providing business owners with the insurance coverage they need
to weather any financial storms that can come along. We want to be there for you every step of the way,
from securing proper coverage, to managing claims. In addition to providing great service, we're
committed to offering clients the lowest premiums possible, by eliminating expensive frills, and passing
the savings on to our customers. The Project is mainly to develop and maintain the insurance data of the
customers and Finance of Current Year and Previous Year. They shared their data in the semi-structured
JSON format, and we did required modifications before loading into target tables by creating SQL select
statements through DBT into respective snowflake databases and maintained version control of the data
by integrating with Azure DevOps with Data build tool.
Roles and Responsibilities:
* Primarily responsible for creating SQL select statements for data transfer to snowflake.?
* Responsible for creation of warehouses, databases and schemas in snowflake as per the client
suggestions.?
* Involved in writing SQL statements for data transfer from source tables to target tables by sing
simple and complex SQL joins.?
* Create storage accounts for dev & QA teams for their application development and
testing purpose. And Creation of Azure VM's, Azure NIC, Azure Virtual networks, and
provide access, users and managing entire Azure infrastructure services.?
* Scale up and scale down various services like VMs, SQL Servers.?
* Automated Continuous Integration builds, nightly builds, deployments and unit tests across
multiple environments (DEV, QA, Production).?
* Creating repositories, new users for GIT and give the permissions to users to access repository.*
* Work with Development Teams and Testing Team to establish a build schedule, execute the builds
and troubleshoot build failures.*
* Good exposure of agile development methodology in designing and implementation of Continuous
Integration and Delivery (CI/CD) methodology.*
* Build and release definition configuration and enabling triggers for CI and CD.*
* High level of understanding and experience of Continuous Integration, Continuous Delivery (CI/CD),
Agile and other DevOps related methodologies.*