Job Location - Coimbatore / Permanent Work from Home

About the Role:

We are looking for a Senior Data Engineer with strong real-life experience in python development, including PySpark in an AWS Cloud environment.

The ideal engineer will have experience with AWS Cloud on data integration with Apache Spark, EMR, Glue, Kafka, Kinesis, and Lambda in S3, Redshift, RDS, and MongoDB/DynamoDB ecosystems.

Responsibilities:

• Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of sources Hadoop, Spark, AWS Lambda, etc.
• Experience with AWS Cloud on data integration with Apache Spark, EMR, Glue, Kafka, Kinesis, and Lambda in S3, Redshift, RDS, MongoDB/DynamoDB ecosystems
• Strong real-life experience in python development especially in pySpark in AWS Cloud environment.
• Design, develop, test, deploy, maintain and improve data integration pipeline.
• Develop pipeline objects using Apache Spark / Pyspark /Python or Scala
• Design and develop data pipeline architectures using Hadoop, Spark, and related AWS services.
• Load and performance test data pipelines built using the above-mentioned technologies

Required Knowledge And Level Of Experience

• At least 5 years of experience with AWS and Python programming, experience with Python frameworks (e.g. Django, Flask, Bottle)
• Expert-level knowledge of using SQL to write complex, highly-optimized queries across large volumes of data.
• Hands-on Experience using programming language python.
• Expert-level knowledge of using SQL to write complex, highly-optimized queries across large volumes of data.
• Knowledge or experience in architectural best practices in building data lakes.
• Strong problem-solving and troubleshooting skills with the ability to exercise mature judgment.