Description
We are looking for an experienced Big Data Platform Engineer to work for our Pharmaceutical client to provide Cloudera/HDFS Platform Engineering/Enhancements. This is a client facing role where you will have regular interactions with various client managers.
Platform Engineering/Enhancements
- HDFS Platform upgrade
- Data and Application Migration
- Platform Performance tuning
- Cloning and cluster Setup
- Optimization of HDFS batch jobs
- Customize Scripts to enhance the functionality of the cluster
- Setting up cluster security
- Reconfiguration of the cluster
- automated script to do the migration and add disk management sections to Ansible automation
- work node by node, migrate data to new volumes, detach old one and confirm mount map
Tech Track, Skill
- Must have 5+ years of hands on Knowledge of Hadoop and HDFS Administration
- Good Understanding of Spark framework including RDDs
- Hands on experience of resource management using YARN and setup of YARN pools.
- Proficient with using Hive and Impala and demonstrated work experience in these areas.
- Knowledge of Unix Shell Scripting and Linux PERL scripts, with hands on experience of writing complex scripts.
- Python programming
- AWS Services -> should have worked on AWS, well versed with various AWS Services, RDS, AWS Lambda, etc. (Preferred)
- Should have good hands on experience with effort estimations, setting up policies management, etc.
- Experience executing in Agile team using scrum is a strong plus
- Code, test, modify, debug, document, and implement software using approved methodologies.
- Must have strong demonstrated track record in technology delivering large scale data driven applications
- Ability to communicate at all levels of management and with customers.
- Demonstrated skills both written and oral are required