Lead Data Engineer
HackerRank
Hybrid in Bangalore, IndiahybridPosted 1 month ago
Skill Required
EngineeringData EngineerDatabricksRedshiftSparkDesign PatternsRAGAWSGenerative AI
Key highlights
- 6+ years of data engineering experience required, including 2+ years in a senior/lead role.
- Deep expertise with OLAP databases such as StarRocks, ClickHouse, or Druid.
- Role focuses on building an AI‑native data layer for natural language querying.
- Equal employment opportunity and affirmative action employer.
- Hybrid AWS + open‑source self‑managed environment.
Role overview
HackerRank's data platform is at an inflection point after completing a multi-year modernisation, migrating from Redshift to StarRocks and Apache Hudi, and reducing export latencies from 25 seconds to under 5 seconds. As a Lead Data Engineer, you will be a senior individual contributor shaping the next‑gen AI‑native data layer that powers revenue‑generating features like natural language querying for HackerRank for Work customers, while owning complex platform decisions and collaborating cross‑functionally with AI, product, and go‑to‑market teams.
Responsibilities
- Own and evolve the data platform – StarRocks (OLAP), Apache Hudi (Data Lake), Trino, Spark, and Apache Ranger – ensuring performance, reliability, and security at scale.
- Build the next‑gen AI‑optimised data layer: clean, structured datasets that power natural language querying and AI add‑on features for HackerRank for Work customers.
- Own in‑product data features – exports, insights dashboards, interview analytics, and the self‑serve Custom Reports interface.
- Enable self‑service pipelines for internal teams (AI platform, analytics, go‑to‑market), reducing ad‑hoc data requests and scaling data access across the org.
- Enforce robust data security – access controls, Apache Ranger policies, and confidence‑scoring guardrails for AI‑generated outputs.
- Lead technical design reviews and define engineering standards for the data team.
- Partner with PMs and business stakeholders to proactively identify and scope AI‑enabled data use cases.
Requirements
- 6+ years of data engineering experience, with at least 2 years in a senior or lead capacity.
- Deep hands‑on expertise with OLAP databases – StarRocks, ClickHouse, Druid, or similar.
- Strong experience with data lake technologies – Apache Hudi, Iceberg, or Delta Lake.
- Proficient with distributed query engines (Trino / Presto) and batch/streaming compute with Apache Spark.
- Solid understanding of data security, RBAC, and access control tools like Apache Ranger.
- Comfortable working in a hybrid AWS + open‑source self‑managed environment.
- Strong communicator who can translate technical decisions for non‑technical stakeholders and drive cross‑functional projects independently.
Nice to have
- Hands‑on experience with AI/LLM‑adjacent data work – confidence scoring, agentic pipelines, RAG architectures, or vector stores.
- Prior exposure to agentic workflows and understanding how to operationalise emerging AI concepts at production scale.
- Experience scaling data infrastructure at a SaaS or B2B product company.
- Familiarity with natural language querying interfaces or building data products for end‑customer consumption.
Benefits
- Equal employment opportunity and affirmative action employer.
- All applicant information kept confidential according to EEO guidelines.
- No payment or credit‑check information ever requested during application, interview, or employment.
Additional details
- HackerRank helps companies like NVIDIA, Amazon, and Microsoft hire and upskill the next generation of developers based on skills, not pedigree.
- Our platform is trusted by over 2,500 of the world’s most innovative companies to build strong engineering teams ready for what’s next.
- Software has entered an era where humans and AI build side by side, changing the definition of strong technical talent.
- People at HackerRank care deeply about the impact of their work and sweat the small details so our customers can be wildly successful with products they genuinely love to use.
- We move with urgency and believe great outcomes come from high standards.
- Want to learn more about HackerRank? Check out HackerRank.com to explore our products, solutions and resources, and dive into our story and mission here.
- Linkedin | X | Blog | Instagram | Life@HackerRank
- Notice to prospective HackerRank job applicants: Our Recruiters use @hackerrank.com email addresses.
- We never ask for payment or credit check information to apply, interview, or work here.