Lead Data Engineer - ETL
Srijan Technologies
IndiaremotePosted 14 days ago
Skill Required
Lead-Data-EngineerData-Engineering-LeadETL-EngineerData-Pipeline-EngineerAWS-Data-EngineerETL-Data-EngineerSenior-ETL-EngineerETL-LeadSenior-Lead-Data-EngineeringLead-Data-Engineer-JobsLead-Data-EngineeringTech-Lead-Data-EngineeringData EngineerETLQuery Optimizationdata engineeringDesign PatternsMicroservicesCD pipelinesEngineeringServerlessdesigningsimilar)analyticsMongoDBRedshiftDynamoDBbuildingSparkdesignDevOpsGitCI/CDAgileAPIsAWSandCIFulltime
Key highlights
- AWS data services expertise required (Glue, S3, Redshift, EMR)
- 5+ years data engineering experience with 2+ years in lead role
- Retail/MMS/POS data experience highly preferred
- Couchbase or similar NoSQL for ODS implementation
Role overview
Material is a global strategy partner helping brands design and deliver rewarding customer experiences through human insights, design innovation, and data. Srijan, a Material company, is a digital engineering firm solving complex technology problems. This role leads data engineering for a retail-focused platform, building scalable data pipelines to ingest and transform merchandise management and point-of-sale data into a centralized Operational Data Store supporting downstream analytics and applications.
Responsibilities
- Design and implement scalable ETL/ELT pipelines to ingest data from MMS/POS or third-party systems into AWS-based data platforms.
- Build and maintain a centralized Operational Data Store (ODS) using Couchbase or similar NoSQL technologies, optimized for low-latency application access.
- Develop and optimize data processing workflows using Apache Spark/PySpark and AWS services such as AWS Glue and Amazon EMR.
- Create denormalized, API-ready data models aligned with downstream microservices and application consumption patterns.
- Implement idempotent processing, CDC merge (upsert) strategies, and data reconciliation mechanisms to ensure consistency across batch and streaming pipelines.
- Leverage Amazon S3, Amazon Redshift, or Amazon Aurora for efficient storage and querying of structured and semi-structured data.
- Implement data ingestion patterns (batch and streaming) using tools like Amazon Kinesis or AWS Lambda where applicable.
- Apply performance tuning and optimization techniques to improve pipeline efficiency, scalability, and cost-effectiveness.
- Define and enforce data governance, data quality, and metadata management standards across the data platform.
- Collaborate with DevOps teams to design and maintain CI/CD pipelines using tools like AWS CodePipeline, GitLab, or similar.
- Conduct peer reviews and provide technical leadership and mentorship to the data engineering team.
- Collaborate with microservices and application teams to ensure seamless integration with the ODS via APIs and event streams.
Requirements
- 5+ years of experience in data engineering, with at least 2+ years in a lead role.
- Strong hands-on experience with AWS data services such as AWS Glue, Amazon S3, Amazon Redshift, and Amazon EMR.
- Proficiency in PySpark/Spark for large-scale data processing and optimization.
- Experience designing and implementing ODS layers using NoSQL databases, preferably Couchbase, or similar (DynamoDB, MongoDB).
- Strong expertise in ETL/ELT design patterns, data ingestion, and transformation pipelines.
- Experience working with MMS, POS, or retail transaction data is highly preferred.
- Hands-on experience with CI/CD pipelines (GitLab, AWS CodePipeline, or similar).
- Good understanding of data governance, data quality, and metadata frameworks.
- Experience supporting microservices architectures with data platforms.
- Strong problem-solving skills and ability to optimize complex data workflows.
- Excellent communication and stakeholder management skills.
- Ability to work in a fast-paced, agile environment and manage multiple priorities.
Additional details
- About US: We turn customer challenges into growth opportunities. Material is a global strategy partner to the world's most recognizable brands and innovative companies. Our people around the globe thrive by helping organizations design and deliver rewarding customer experiences. We use deep human insights, design innovation and data to create experiences powered by modern technology. Our approaches speed engagement and growth for the companies we work with and transform relationships between businesses and the people they serve. Srijan, a Material company, is a renowned global digital engineering firm with a reputation for solving complex technology problems using their deep technology expertise and leveraging strategic partnerships with top-tier technology partners. Be a part of an Awesome Tribe.