Introduction:

PySpark is the Python API written in python to support Apache Spark. ... Apache Spark is written in Scala and can be integrated with Python, Scala, Java, R, SQL languages. Spark is basically a computational engine, that works with huge sets of data by processing them in parallel and batch systems.

Responsibilities:

• Design, develop, test, deploy, support, enhance data integration solutions seamlessly to connect and integrate Thermo Fisher enterprise systems in our Enterprise Data Warehouse and Data Platforms.
• Innovate for data integration in Apache Spark-based Platform to ensure the technology solutions leverage cutting edge integration capabilities.
• Facilitate requirements gathering and process mapping workshops, review business/functional requirement documents, author technical design documents, testing plans and scripts.
• Assist with implementing standard operating procedures, facilitate review sessions with functional owners and end-user representatives, and leverage technical knowledge and expertise to drive improvements.

Requirements:

• BS Degree in CS or equivalent
• Overall 7-8 years of experience in the Enterprise Data Warehouse Development environment
• 2+ years of working experience in a DevOps environment
• 2+ years working experience in data integration and pipeline development.
• 2+ years of Experience with AWS Cloud on data integration with Apache Spark, EMR, Glue, Kafka, Kinesis, and Lambda in S3, Redshift, RDS, MongoDB/DynamoDB ecosystems
• Databricks, Redshift Experience is a plus.
• 2 - 3 years of Project Management experience in Agile methodology

Mandatory Experience:

• Demonstrated skill and ability in the development of data warehouse projects/applications (Oracle & SQL Server)
• Strong real-life experience in python development especially in pySpark in AWS Cloud environment.
• Design, develop test, deploy, maintain and improve data integration pipeline.
• Experience in Python and common python libraries.
• Strong analytical experience with database in writing complex queries, query optimization, debugging, user defined functions, views, indexes etc.
• Experience with source control systems such as Git, Bitbucket, and Jenkins build and continuous integration tools.
• Knowledge of extract development against ERPs - SAP, Navision, QAD preferred
• Highly self-driven, execution-focused, with a willingness to do "what it takes to deliver results as you will be expected to rapidly cover a considerable amount of demands on data integration
• Understanding of development methodology and actual experience writing functional and technical design specifications.
• Excellent verbal and written communication skills, in person, by telephone, and with large teams.
• Strong prior technical, development background in either data Services or Engineering
• Demonstrated experience resolving complex data integration problems;
• Must be able to work cross-functionally. Above all else, must be equal parts data-driven and results-driven.

Benefits:

• Exposure to new processes and technologies.
• Competitive salary at par with the best in the industry.
• Flexible and employee friendly environment.