The NPD Group

Big Data Engineer Lead

  • Job Type: Full Time
  • Industry Type: IT Sector
  • Industry Location: New York
  • Experience: NA
  • No. of Positions: 1
  • Primary Skills: Cloudera Data visualization Big data R Hadoop Spark Java Scala Python REST services
  • Secondary Skills: UNIX Azure Software applications Kafka Hana SQL server Data engineering ETL
  • Job Location: New York, New York
  • Posted Date: Posted today
Job Description

Position Summary:

The Big Data Engineer Lead will be responsible for big data engineering, data wrangling, data analysis and user support primarily focused on the Cloudera Hadoop platform, but in future extending to the cloud. The Big Data Engineer Lead must have strong hands-on technical skills including conventional ETL and SQL skills with programming as well as data science languages such as Python and R, using big data techniques. The role will also play a role in defining and implementing Big Data Strategy for the organization along with driving implementation of IT solutions for the business. You must be a self-starter with the ability to manage projects through all stages (requirements, design, coding, testing, implementation, and support.)

Essential Job Responsibilities:

  • Analyze the business needs, profile large data sets and build custom data models and applications to drive business decision making and customers experience
  • Build workflows that empower analysts to efficiently use data
  • Develop and extend design patterns, processes, standards, frameworks, and reusable components for various data engineering functions areas.
  • Requirements analysis, planning and forecasting for Hadoop data engineering/ingestion projects
  • Design optimized Hadoop and big data solutions for data ingestion, data processing, data wrangling, and data delivery
  • Design, develop tune data products, streaming applications, and integrations on large-scale data platforms (Hadoop, Kafka Streaming, Hana, SQL server, Data warehousing, big data, etc) with an emphasis on performance, reliability and scalability, and most of all quality.
  • Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
  • Build the infrastructure required for efficient extraction, transformation, and loading of data from a wide variety of data sources
  • Build data tools for analytics and data scientist team members that assist them in building and optimizing our product into an innovative industry leader.
  • Develop custom data models and algorithms
  • Identify opportunities for data acquisition
  • Peer Review of the code developed by team members

Additional Job Responsibilities:

  • May be a project or function lead and has an impact on the results produced.
  • All other duties as assigned.
  • Share your passion for staying on top of tech trends, experimenting with and learning new technologies, participating in internal & external technology communities, and mentoring other members of the engineering community

Minimum Qualifications:

  • Bachelor’s degree in a technology filed or equivalent experience
  • 10+ years’ experience in developing software applications
  • 5+ years’ experience working on big data technologies like Hadoop and Spark -Proficiency in at-least one of the following languages Java, Scala, Python, R -Experience with building and supporting distributed systems -Experience with REST services preferred -Fluency in Unix command line tools and bash is preferred -Experience with data visualization

Knowledge/Skills/Abilities Required:

  • Strong problem-solving skills with an ability to isolate, deconstruct and resolve complex data engineering challenges
  • Ability to quickly ramp up on new tools/software

Technical Skills:

  • Demonstrated experience in architecture, engineering, and implementation of enterprise-grade production big data use case
  • Strong SQL, ETL, scripting, and or programming skills with a preference towards Python, Java, Scala, shell scripting
  • Demonstrated ability to clearly form and communicate ideas to both technical and non-technical audiences.
  • Willingness to explore new alternatives or options to solve data engineering and data mining issues, and utilize a combination of industry best practices, innovations and experience to get the job done
  • Experience performing root cause analysis on internal and external data and processes to answer specific business questions and find opportunities for improvement
  • Strong Microsoft Office skills, particularly Excel and analytical platforms


  • Technical skills – data and visual analytics, design, creativity
  • Problem solving – set priorities and process management
  • Process/planning – perceptive and perseverance


Relevant Job Openings
Oracle Developer
Mobile or API tester
.Net Core Developers
Tableau developer
Senior DevOps engineer