Detailed JD *(Roles and Responsibilities)

• Built Pyspark based metadata(typically MongoDB) driven framework for Datalake transformation
• Create Airflow DAG pipelines for data ingestion
• Develop docker based service using Pyspark, Redis, Fast API
• Build realtime streaming pipeline using Spark and Kafka
• Spark query tuning and performance optimization
• Experience of dealing with AWS S3 object storage from PySpark
• Performance tuning of PySpark scripts/notebook in Databricks
• Knowledge on delta lake concepts and experience working with AWS Athena

Good experience working with SQLs

Mandatory skills*

AWS, EMR, Python, Spark