YipitData is the market-leading data and analytics firm, analyzing billions of data points daily to deliver insights across industries like consumer brands, technology, software, and healthcare. The company uses proprietary technology to process data for global investment funds and corporations, backed by The Carlyle Group and Norwest Venture Partners. Recognized as one of Inc’s Best Workplaces, YipitData seeks a hands-on Data Engineering Manager to lead a team, shape data architecture, and drive best practices in pipeline performance, reliability, and cost-efficiency. This hybrid role (currently remote, transitioning to hybrid in India) requires collaboration with US and LatAm teams during adjusted IST hours (2:30–10:30pm IST, 2-3 days/week).
Responsibilities
- Lead, mentor, and grow a team of data engineers working on large-scale distributed data systems.
- Architect and oversee the development of end-to-end data solutions using AWS Data Services and Databricks.
- Hire, onboard, and develop a high-performing team—1-on-1s, growth plans, and performance reviews.
- Collaborate with cross-functional teams including data science, analytics, product, and business stakeholders to understand requirements and deliver impactful data products.
- Drive best practices in data engineering, coding standards, version control, CI/CD, and monitoring.
- Ensure high data quality, governance, and compliance with internal and external policies.
- Optimize performance and cost efficiency of data infrastructure in the cloud.
- Architect and evolve our data platform (batch & streaming) for scale, cost, and reliability.
- Own the end-to-end vision and strategic roadmap for various projects.
- Create documentation, architecture diagrams, and other training materials.
- Translate product and analytics needs into a clear data engineering roadmap and OKRs.
- Stay current with industry trends, emerging technologies, and apply them to improve system architecture and team capabilities.
Requirements
- Bachelor’s or Master’s degree in Computer Science, STEM, or a related technical discipline.
- 8+ years in data engineering (or adjacent), including 2-3+ years formally managing 1-3 engineers.
- Proven hands-on experience with Big Data ecosystems (Spark, Hive, Hadoop).
- Proven hands-on experience with Databricks (including Delta Lake, MLFlow, Unity Catalog).
- Robust programming experience in Python and PySpark.
- Deep understanding of data modeling, ETL/ELT processes using Streaming, and performance tuning.
- Experience managing Agile teams and delivering complex projects on time.
- Excellent problem-solving, leadership, and communication skills.
- Experience designing and implementing Agentic Models with Data Pipelines (Data Cleaning and creative Feature Engineering).
- Practical LLM/RAG experience for search quality such as query understanding, semantic retrieval, reranker design.
- Self-starter who enjoys working with both internal and external stakeholders.
- 3+ years of managing data engineers.
- 5+ years of experience working with PySpark and Python.
Nice to have
- Familiarity with ML/AI workflows and collaboration with data science teams.
- Experience with Airflow, Docker, or equivalent.
Additional details
- This is a hybrid opportunity based in India. (Currently, the role is remote, but once we establish an office space in India, the role will become hybrid. The hybrid work schedule is flexible, meaning the team in collaboration with the manager will determine when they come into the office.)
- Standard IST working hours are permitted with the exception of 2-3 days per week, when you will join meetings with the US and LatAm team. On these days, work hours will be between 2:30 - 10:30pm IST.
- We allow for flexibility on the following days to make up for the previous day’s late work schedule!
- We are committed to equal employment opportunity regardless of race, color, ancestry, religion, sex, national origin, sexual orientation, age, marital status, disability, gender, gender identity or expression, or veteran status. We are proud to be an equal-opportunity employer.
- Job Applicant Privacy Notice
- Originally posted on Himalayas
Zobhira
