Senior Data Engineer - Databricks

Srijan Technologies

IndiaremotePosted 1 month ago

Skill Required

Data-EngineerLead-Data-EngineerData-EngineeringDatabricks-EngineerSpark-DeveloperSenior-Databricks-EngineerDatabricks-Data-EngineerSenior-Databricks-DeveloperSenior-Databricks-ArchitectSenior-Data-Engineer-JobsSenior-Data-EngineeringSenior-Data-Engineer-PositionsData-Engineering-(Databricks)Data EngineerDatabricksQuery OptimizationCD pipelinesEngineeringR ProgramminganalyticaldesigningdebuggingsecuritybuildingRAGSparkPythondesignDevOpsGitCI/CDGenerative AIAPIsETLSQLCICDAIMachine LearningFulltime

Key highlights

Senior/Lead Data Engineer role
Expertise in Databricks, PySpark & Applied AI/LLM Engineering
5+ years of hands-on Data Engineering experience required
Retail domain experience highly preferred

Role overview

We turn customer challenges into growth opportunities. Material is a global strategy partner to the world’s most recognizable brands and innovative companies. Our people around the globe thrive by helping organizations design and deliver rewarding customer experiences. We use deep human insights, design innovation and data to create experiences powered by modern technology. Our approaches speed engagement and growth for the companies we work with and transform relationships between businesses and the people they serve. Srijan, a Material company, is a renowned global digital engineering firm with a reputation for solving complex technology problems using their deep technology expertise and leveraging strategic partnerships with top-tier technology partners.

Responsibilities

Design and build scalable data platforms using Databricks, Spark, Delta Lake, and PySpark.
Develop and optimize large-scale data pipelines and distributed processing applications.
Perform Spark performance tuning, query optimization, and troubleshooting.
Build and manage CI/CD pipelines using GitLab and automation tools.
Work on REST APIs, Databricks automation, and deployment workflows.
Support AI/ML workflows including LLMOps, RAG pipelines, vector search, and model evaluation.
Implement data governance, security, metadata management, and data quality standards.
Conduct code reviews, mentor junior engineers, and drive engineering best practices.
Collaborate with DevOps, ML, and Business teams to deliver scalable AI and analytics platforms.

Requirements

5+ years of hands-on experience in Data Engineering with strong enterprise delivery experience.
Proven experience as a Senior Data Engineer / Lead Data Engineer in large-scale environments.
Expert-level programming experience in Python and PySpark.
Strong expertise in Apache Spark internals and Spark performance optimization.
Hands-on experience with Databricks.
Hands-on experience with Delta Lake.
Hands-on experience with Spark SQL.
Advanced SQL optimization.
Extensive experience building and maintaining enterprise CI/CD pipelines.
Strong expertise in GitLab pipelines and automated deployment workflows.
Hands-on experience with Databricks REST APIs.
Strong understanding of distributed data processing and scalable architecture design.
Experience with LLM evaluation frameworks and ML model evaluation pipelines.
Experience supporting or implementing LLM fine-tuning workflows.
Experience with vector-space architectures, embeddings, semantic search, or vector databases.
Strong experience in scalable data integration and distributed processing frameworks.
Experience with data governance, metadata management, lineage, and data quality frameworks.
Strong analytical, debugging, and problem-solving capabilities.
Excellent stakeholder communication and technical leadership skills.

Nice to have

Experience working on Python IDE-based enterprise development environments.
Experience in ML model evaluation and AI platform engineering.
Experience supporting GenAI, RAG, or LLMOps initiatives.
Experience with vector databases such as Pinecone, FAISS, ChromaDB, or similar platforms.
Experience designing reusable framework-based engineering solutions rather than project-specific implementations.
Retail domain experience.