03/26/2023 updated

AP
100 % available

Data Engineer, Data Engineer, Data Engineer

Dhanbad, India
Only remote
BCA (HON'S)
Dhanbad, India
Only remote
BCA (HON'S)

Profile attachments

CV - Amit Paul

ASP.NETADO.NETApplication Programming Interfaces (APIs)Artificial IntelligenceAmazon Web ServicesAmazon Elastic Compute CloudAmazon S3Data AnalysisUnit TestingCA Workload Automation AeBackup DevicesBig DataCloud ComputingDatabasesData TransmissionsInformation EngineeringExtract Transform Load (ETL)Data Manipulation LanguagesData WarehousingDatabase DesignDatabase TestingAmazon DynamoDBPython (Programming Language)Microsoft Visual StudioMicrosoft SQL ServersMySQLObject-Oriented Software DevelopmentOracle DatabasesOracle ApplicationsPerformance TuningSAP ABAPSAP HANASAP MMSAP SDShell ScriptStored ProcedureSQL DatabasesData StreamingSalesforce TableauExtensible Markup Language (XML)Data/Record LoggingGoogle CloudApache SparkBackendCloudformationpandasPytestData LakeIntegration TestsInformation TechnologyAWS GlueRestful ApiData PipelineAmazon Elastic Mapreduce (EMR)Amazon RedshiftDatabricksProgramming Languages
Ado.net, Asp.net, AWS Glue, ABAP, DynamoDB, EC2, AWS EMR, Redshift, AWS S3, S3, AWS, Spark, Apache Spark, API, artificial intelligence, backend, data back, big data, big data Technologies, Autosys, Cloud, Cloud Services, Cloud Formation, analytics, Data Lake, logging, DML, data pipelines, data streaming, data transfer, Data warehouses, database designing, database testing, databases, Databricks, xml, ETL, Google Cloud, Data Engineer, Computer Science, Informatics, Integration testing, excel, Sql server, Visual Studio Code, mysql, MySql Server, OOPS, oracle, oracle 11g, Pandas, performance tuning, Programming Language, Pytest, Python, REST web services, HANA, SAP Technical, Sales Distribution, SQL, stored procedures, Shell Scripting, tableau, Unit testing

Experience

Professional experience5 - 10 years
Freelancer since2017

Languages

EnglishGood

Project history

Data Engineer

Amit Paul
Data Engineer
Computing Framework : Spark API- RDD & SparkSql

AWS Cloud Services : S3, Glue, EMR, Athéna, Lambda

Tools/Ide's : Pycharm, Visual Studio Code, Microsoft Teams

Data Science : Python Libraries-Pandas

Data Engineer

Brillio Technologies
Project: ETL for Pharma Majorclient

In this Big Data project we built a AWS Glue transformation for Redshift Spectrum to move the data
from AWS S3 and other external destination sources ; enabling data scientists to leverage GCP
(Google Cloud Capabilities) environments for performing research , analysis and experimentation.
Built automated Glue template / lambda script in EC2 instance to provision a platform based on
batch data streaming for specific global partners and stakeholders. From here Data Scientists
requests for relevant data post GCP creation in S3 data lake. Built a datapipeline and separated
the data transfer files from AWS S3 and enabled all components of ML and BI for research and
analysis.

Activities / responsibilities:

* Analyzing the different source systems and knowing how to extract data from each source system.

* Transforming and Loading the data to S3 by the help Apache Spark on Databricks.

* Coordinated with BI team to provide them with necessary data required for reporting.

* Designed and developed complex data pipelines for customer. * Wrote production level code for
logging, querying shipping sales.

* Construct ETL to capture productivity and data quality checks Environment: Python, Spark, Glue,
S3, Lambda, Cloud Formation, DynamoDB, Code pipeline, Code Build, Pytest, Step Functions, Athena
, Snowflake , Autosys , Shell Scripting Technologies: Python, Spark, Aws Glue, Lambda, S3, Athena ,
Sagemaker , Redshift

Data Engineer

Enum Informatics Private Limited
Project: ETL for Retail domain client
In this Big Data project we extract data from HANA DB using Informatica
and put them into Aws S3. From here we do the transformation using AWS
Glue and write the data back to s3. The data set residing on the s3 are
being populated to the end user by using tableau.

Activities / responsibilities:
* Written spark code to read the data, transform the data and write the data in s3.
* Coordinated with BI team to provide them with necessary data required for reporting.
* Validated the data set between the source and target systems.
* Given support to the exising pipelines in the production

Technologies: Python, Spark, Aws Glue, S3, Athena

SAP Technical Consultant

Isquare Soft Technologies
Amit Paul
Data Engineer



HPE's new high-end storage platform to driving the next wave of the Intelligent
Edge and cloud choices. Customer validation , new creation of batch entity and
batch number of customer through HPI and HPE tenant.
Created implementation report for Customer inquiry and validation, used
interface implementation and standard method PRESET_SALES_AREA for valid
additional data under sales header area. Documented and specified it for
Implicit Implementation in BADI for Shipping Sales Requisition.
Roles & Responsibilities Undertaken
* Created the ODATA and REST web services report program to extract data in xml format using OOPS
and RFC methodologies
* Created class builder report using classic BADI and implemented it in XD01/XD02 to create and
fetch customer details based on shipping additional data screen fields and updated KNVV through
standard function methods PRESET_SALES_AREA using interface from implementation.
* Performed Unit testing and Integration testing for ALE Idocs which is generated and executed
successfully Worked on Conversion Routine to add field constraints in segment for ALE Idocs through
mandatory field LZONE with additional data and classification linking.
* Checked and verified partner details in presentation server file
* Cross verified with client to check extracted data migrated to standard field using BDC report and
merged it in custom table through field pointers concept.

Data Engineer

Enum Informatics Private Limited
Project: ETL for Retail domain client
In this Big Data project we extract data from different sources like
mysql, oracle and put them into Aws S3. From here we do the
transformation using AWS Glue and drop the data into processed layer.
The processed layer data residing on the data lake is being published to
the end user by using Athena Views.

Activities / responsibilities:
* Analyzing the different source systems and knowing how to extract data from each source system.
* Transforming and Loading the data to S3 by the help Apache Spark.
* Coordinated with BI team to provide them with necessary data required for reporting.
* Designed and developed complex data pipelines for customer
* Wrote production level code for logging, querying shipping sales
* Construct ETL to capture productivity and data quality checks

Technologies: Python, Spark, Aws Glue, S3, Athena

Testing Engineer

Infosys Pvt. Ltd.
for banking domain
Infosys Finacle solutions address the core banking liquidity management, wealth
management, analytics, artificial intelligence, and financial institutions to
drive business excellence.
Infosys Finacle universal banking solution builds on the success of Finacle to
deliver powerful benefits to global banks. The solution makes possible faster
launch of new products and services , helping banks realize a 55 percent return
on core banking transformation investments and an average improvement of 33
percent in their time to market as banks aim to reinvent their business and
navigate the current challenges in the macroeconomic environment, Finacle 11E
promises a simplified approach to banking transformation.

Roles & Responsibilities Undertaken
* Cross verified with client to check extracted data migrated to standard field using BDC report and
merged it in custom table through field pointers concept.
* Created test cases in excel sheet and written test scenarios for database testing using query on
the data on MySql Server.
* Performed DDL/DML operations on backend databases, triggered stored procedures while database
designing.

Contact form

Log in to get in touch

You need to be logged in to use the contact form.

Sign upLog in