Big Data Developer
Royal Bank of Canada
Keywords
Big Data
Apache Spark
Java (Programming Language)
Apache Hadoop
Apache HTTP Server
Apache Hive
Oracle Applications
SQL Databases
Apache Nifi
Application Programming Interfaces (APIs)
Skills
I have 5+ experience in big data technologies. I'm certified OCJP (oracle java certification) and CCA175 spark certification.
Apache Hive, Spark, Scala, HBASE, Impala, Flume, Sqoop, Java, MYSQL, python, Flink, Kylo, NIFI, Kafka, Java Rest API, Big Data, IOT, Spark framework, JSON, data ingestion, Hive, stream processing, windowing, XML, partitioning, SQL, NAS, HDFS, UDF, insert into, OCR, Cloudera CDH, IBM, Azure cluster, Hadoop, E-learning, Oracle, Intellij, Teradata, Linux, coding, Unit testing, bug fixing, Spring boot, JPA, Apache, API, SQL Server, stored procedures, BigData, RDBMS, staging table, Cassandra, real time data
Apache Hive, Spark, Scala, HBASE, Impala, Flume, Sqoop, Java, MYSQL, python, Flink, Kylo, NIFI, Kafka, Java Rest API, Big Data, IOT, Spark framework, JSON, data ingestion, Hive, stream processing, windowing, XML, partitioning, SQL, NAS, HDFS, UDF, insert into, OCR, Cloudera CDH, IBM, Azure cluster, Hadoop, E-learning, Oracle, Intellij, Teradata, Linux, coding, Unit testing, bug fixing, Spring boot, JPA, Apache, API, SQL Server, stored procedures, BigData, RDBMS, staging table, Cassandra, real time data
Project history
07/2014
-
07/2019
Royal Bank of Canada, Deutsche bank group /EcoAxis (Business Unit of A.T.E. Enterprises Private
Limited) /Idea Cellular Limited/Ellicium Solutions Private Limited
Banking/IOT
Duration (07/2014 - 07/2019)
Tasks/Achievements
* Developed Spark framework to JSON data ingestion into Hive table
* Aggregate real time stream processing data using windowing and slide function in Flink and batch
process using Spark
* Extract data from zip file in spark (zip to unzip)
* Developed Spark framework to XML data ingestion into Hive table
* Created HIVE tables to store the processed results in a tabular format. Usage of SPARK custom
partitioning
* Used Spark-SQL to run the Hive queries on top of spark engine
* Backfill Historic data in spark
* Data pipeline building using NiFi from NAS to HDFS
* Developed UDF in Spark to calculate complex transformation
* Demoralization to normalize historic data and insert into hive using Spark
* Developed FX-interpolation missing days data using spark windows function
* Processed scanned document and applying OCR to convert the text format.
* Have installed KYLO and NIFI on HDP sandbox
* Created Data ingestion using NIFI and KYLO.
* Created NIFI flow templates for extract data from website and load it into HDFS
* Delete HBase data form HBASE
* Experience in installing Cloudera CDH 5 on IBM and Azure cluster
* Data loading from MYSQL to Hadoop using Sqoop.
* Big data certifications from Big Data University (E-learning)
* Certified Oracle Java programmer
Limited) /Idea Cellular Limited/Ellicium Solutions Private Limited
Banking/IOT
Duration (07/2014 - 07/2019)
Tasks/Achievements
* Developed Spark framework to JSON data ingestion into Hive table
* Aggregate real time stream processing data using windowing and slide function in Flink and batch
process using Spark
* Extract data from zip file in spark (zip to unzip)
* Developed Spark framework to XML data ingestion into Hive table
* Created HIVE tables to store the processed results in a tabular format. Usage of SPARK custom
partitioning
* Used Spark-SQL to run the Hive queries on top of spark engine
* Backfill Historic data in spark
* Data pipeline building using NiFi from NAS to HDFS
* Developed UDF in Spark to calculate complex transformation
* Demoralization to normalize historic data and insert into hive using Spark
* Developed FX-interpolation missing days data using spark windows function
* Processed scanned document and applying OCR to convert the text format.
* Have installed KYLO and NIFI on HDP sandbox
* Created Data ingestion using NIFI and KYLO.
* Created NIFI flow templates for extract data from website and load it into HDFS
* Delete HBase data form HBASE
* Experience in installing Cloudera CDH 5 on IBM and Azure cluster
* Data loading from MYSQL to Hadoop using Sqoop.
* Big data certifications from Big Data University (E-learning)
* Certified Oracle Java programmer
10/2018
-
02/2019
Big Data Developer (Senior Technical Associate Consultant)
Royal Bank of Canada
Data Transformation Library
Royal Bank of Canada
Duration (10/2018 - 02/2019)
Banking
Pune
Intellij Idea| Position: Big Data Developer (Senior Technical Associate Consultant)
* Develop a Job control module which is used to build and create scala/spark project structure
for to do actual data transformation in data lake.
* Have done basic and advance search functionality using Spring
boot and JPA .
* Created holistic view using Apache Atlas.
* Created spring boot rest API to implement custom logic in DTL application.
Royal Bank of Canada
Duration (10/2018 - 02/2019)
Banking
Pune
Intellij Idea| Position: Big Data Developer (Senior Technical Associate Consultant)
* Develop a Job control module which is used to build and create scala/spark project structure
for to do actual data transformation in data lake.
* Have done basic and advance search functionality using Spring
boot and JPA .
* Created holistic view using Apache Atlas.
* Created spring boot rest API to implement custom logic in DTL application.
07/2018
-
10/2018
Big Data Developer (Senior Technical Associate Consultant)
Royal Bank of Canada
Retail Reporting 2.0
Royal Bank of Canada
Duration (07/2018 - 10/2018)
Banking
Pune
Intellij Idea| Spark | Hive |Sqoop Position: Big Data Developer (Senior Technical Associate
Consultant)
* Complete migration of current SQL Server process to bigdata.
* Conversion of SQL stored procedures into BigData environment using Spark SQL and DataSets.
* Data ingestion from multiple source to data lake using Sqoop.
* Data ingestion from Teradata to data lake using Sqoop.
* Scala coding for spark implementations.
* Unit testing and bug fixing.
* Data pipeline building using NiFi from NAS to HDFS
* Migrated whole process from SQL server and Teradata to EDL (data lake)
Royal Bank of Canada
Duration (07/2018 - 10/2018)
Banking
Pune
Intellij Idea| Spark | Hive |Sqoop Position: Big Data Developer (Senior Technical Associate
Consultant)
* Complete migration of current SQL Server process to bigdata.
* Conversion of SQL stored procedures into BigData environment using Spark SQL and DataSets.
* Data ingestion from multiple source to data lake using Sqoop.
* Data ingestion from Teradata to data lake using Sqoop.
* Scala coding for spark implementations.
* Unit testing and bug fixing.
* Data pipeline building using NiFi from NAS to HDFS
* Migrated whole process from SQL server and Teradata to EDL (data lake)
05/2018
-
06/2018
Big Data Developer (Senior Technical Associate Consultant)
Teradata India pvt ltd
Banking
Pune
Intellij Idea| HDP 2.6| Position: Big Data Developer (Senior Technical Associate Consultant)
* Populate data from Hive staging table to RDBMS staging table
using spark , NIFI and Kylo.
* Created Custom NIFI processor for create RDBMS table creation on the fl
fly as per the kylo feed and category.
* Populate data from RDBMS staging table to RDBMS target table using merge strategy
* Initialize the staging and target table in RDBMS with data type mapping on new feed creation
* Created custom NIFI processor to implement merge strategy on RDBMS
Table.
Real time Stock Tracker and Alerting system
Pune
Intellij Idea| HDP 2.6| Position: Big Data Developer (Senior Technical Associate Consultant)
* Populate data from Hive staging table to RDBMS staging table
using spark , NIFI and Kylo.
* Created Custom NIFI processor for create RDBMS table creation on the fl
fly as per the kylo feed and category.
* Populate data from RDBMS staging table to RDBMS target table using merge strategy
* Initialize the staging and target table in RDBMS with data type mapping on new feed creation
* Created custom NIFI processor to implement merge strategy on RDBMS
Table.
Real time Stock Tracker and Alerting system
01/2018
-
04/2018
Big Data Developer (Senior Technical Associate Consultant)
Teradata india pvt ltd
Banking
Pune
Intellij Idea| HDP 2.6| Position: Big Data Developer (Senior Technical Associate Consultant)
* Extract data from external system and put into Kafka
* Aggregate real time stream processing data using windowing and slide function in Flink and batch
process using Spark
* Insert data into hive partition table using spark custom partition
* Used Spark-SQL to Aggregate data and insert into Cassandra
Compare historic ratio with real time ratio and take automatic action
when stock price and volume fluctuations exceed a pre-defined threshold
* Alert in the form of mail and log using Flink
* Ingest real time data into Apache Ignite
Pune
Intellij Idea| HDP 2.6| Position: Big Data Developer (Senior Technical Associate Consultant)
* Extract data from external system and put into Kafka
* Aggregate real time stream processing data using windowing and slide function in Flink and batch
process using Spark
* Insert data into hive partition table using spark custom partition
* Used Spark-SQL to Aggregate data and insert into Cassandra
Compare historic ratio with real time ratio and take automatic action
when stock price and volume fluctuations exceed a pre-defined threshold
* Alert in the form of mail and log using Flink
* Ingest real time data into Apache Ignite
Local Availability
Only available in these countries:
India
open to travel availblity
Other
I have 5+ experience in Big data technologies especially on spark , apache Nifi , Hive , Scala and Java.
I have completed CCA 175 spark certification and OCJP (oracle java certified)
I have completed CCA 175 spark certification and OCJP (oracle java certified)
