03/26/2023 updated
AP
100 % available
Data Engineer, Data Engineer, Data Engineer
Dhanbad, India
Only remote
BCA (HON'S)ASP.NETADO.NETApplication Programming Interfaces (APIs)Artificial IntelligenceAmazon Web ServicesAmazon Elastic Compute CloudAmazon S3Data AnalysisUnit TestingCA Workload Automation AeBackup DevicesBig DataCloud ComputingDatabasesData Transmissions
Ado.net, Asp.net, AWS Glue, ABAP, DynamoDB, EC2, AWS EMR, Redshift, AWS S3, S3, AWS, Spark, Apache Spark, API, artificial intelligence, backend, data back, big data, big data Technologies, Autosys, Cloud, Cloud Services, Cloud Formation, analytics, Data Lake, logging, DML, data pipelines, data streaming, data transfer, Data warehouses, database designing, database testing, databases, Databricks, xml, ETL, Google Cloud, Data Engineer, Computer Science, Informatics, Integration testing, excel, Sql server, Visual Studio Code, mysql, MySql Server, OOPS, oracle, oracle 11g, Pandas, performance tuning, Programming Language, Pytest, Python, REST web services, HANA, SAP Technical, Sales Distribution, SQL, stored procedures, Shell Scripting, tableau, Unit testing
Experience
Professional experience5 - 10 years
Freelancer since2017
Languages
EnglishGood
Project history
Amit Paul
Data Engineer
Computing Framework : Spark API- RDD & SparkSql
AWS Cloud Services : S3, Glue, EMR, Athéna, Lambda
Tools/Ide's : Pycharm, Visual Studio Code, Microsoft Teams
Data Science : Python Libraries-Pandas
Data Engineer
Computing Framework : Spark API- RDD & SparkSql
AWS Cloud Services : S3, Glue, EMR, Athéna, Lambda
Tools/Ide's : Pycharm, Visual Studio Code, Microsoft Teams
Data Science : Python Libraries-Pandas
Project: ETL for Pharma Majorclient
In this Big Data project we built a AWS Glue transformation for Redshift Spectrum to move the data
from AWS S3 and other external destination sources ; enabling data scientists to leverage GCP
(Google Cloud Capabilities) environments for performing research , analysis and experimentation.
Built automated Glue template / lambda script in EC2 instance to provision a platform based on
batch data streaming for specific global partners and stakeholders. From here Data Scientists
requests for relevant data post GCP creation in S3 data lake. Built a datapipeline and separated
the data transfer files from AWS S3 and enabled all components of ML and BI for research and
analysis.
Activities / responsibilities:
* Analyzing the different source systems and knowing how to extract data from each source system.
* Transforming and Loading the data to S3 by the help Apache Spark on Databricks.
* Coordinated with BI team to provide them with necessary data required for reporting.
* Designed and developed complex data pipelines for customer. * Wrote production level code for
logging, querying shipping sales.
* Construct ETL to capture productivity and data quality checks Environment: Python, Spark, Glue,
S3, Lambda, Cloud Formation, DynamoDB, Code pipeline, Code Build, Pytest, Step Functions, Athena
, Snowflake , Autosys , Shell Scripting Technologies: Python, Spark, Aws Glue, Lambda, S3, Athena ,
Sagemaker , Redshift
In this Big Data project we built a AWS Glue transformation for Redshift Spectrum to move the data
from AWS S3 and other external destination sources ; enabling data scientists to leverage GCP
(Google Cloud Capabilities) environments for performing research , analysis and experimentation.
Built automated Glue template / lambda script in EC2 instance to provision a platform based on
batch data streaming for specific global partners and stakeholders. From here Data Scientists
requests for relevant data post GCP creation in S3 data lake. Built a datapipeline and separated
the data transfer files from AWS S3 and enabled all components of ML and BI for research and
analysis.
Activities / responsibilities:
* Analyzing the different source systems and knowing how to extract data from each source system.
* Transforming and Loading the data to S3 by the help Apache Spark on Databricks.
* Coordinated with BI team to provide them with necessary data required for reporting.
* Designed and developed complex data pipelines for customer. * Wrote production level code for
logging, querying shipping sales.
* Construct ETL to capture productivity and data quality checks Environment: Python, Spark, Glue,
S3, Lambda, Cloud Formation, DynamoDB, Code pipeline, Code Build, Pytest, Step Functions, Athena
, Snowflake , Autosys , Shell Scripting Technologies: Python, Spark, Aws Glue, Lambda, S3, Athena ,
Sagemaker , Redshift
Project: ETL for Retail domain client
In this Big Data project we extract data from HANA DB using Informatica
and put them into Aws S3. From here we do the transformation using AWS
Glue and write the data back to s3. The data set residing on the s3 are
being populated to the end user by using tableau.
Activities / responsibilities:
* Written spark code to read the data, transform the data and write the data in s3.
* Coordinated with BI team to provide them with necessary data required for reporting.
* Validated the data set between the source and target systems.
* Given support to the exising pipelines in the production
Technologies: Python, Spark, Aws Glue, S3, Athena
In this Big Data project we extract data from HANA DB using Informatica
and put them into Aws S3. From here we do the transformation using AWS
Glue and write the data back to s3. The data set residing on the s3 are
being populated to the end user by using tableau.
Activities / responsibilities:
* Written spark code to read the data, transform the data and write the data in s3.
* Coordinated with BI team to provide them with necessary data required for reporting.
* Validated the data set between the source and target systems.
* Given support to the exising pipelines in the production
Technologies: Python, Spark, Aws Glue, S3, Athena