Data Engineering - Technical Lead
Paytm
Noida, Uttar Pradesh, IndiaPosted 8 months ago
Skill Required
TechnologyData EngineerTechnical LeadEngineeringTest AutomationData Structuresrelated fieldETLSpring BootR ProgrammingPostgreSQLKubernetesCassandrasecuritybuildingAirflowMongoDBPythonOracleDockerdesignScalaSparkKafkaFlinkDesign PatternsJavagRPC
Key highlights
- 6+ years of required experience
- Strong SQL and ability to write complex queries
- Opportunity to work with 500 million+ registered users
- Potential to create personal wealth through the platform
- Experience with AWS/GCP and streaming technologies
- Rotational on‑call responsibilities for live service operations
Role overview
Build systems for collection & transformation of complex data sets for use in production systems, collaborate with engineers on building & maintaining back‑end services, implement data schema and data management improvements for scale and performance, provide insights into key performance indicators for the product and customer usage, serve as the team’s authority on data infrastructure, privacy controls and data security, and support continuous improvement, metrics and test automation while maintaining live service operations on a rotational on‑call basis.
Responsibilities
- Build systems for collection & transformation of complex data sets for use in production systems
- Collaborate with engineers on building & maintaining back‑end services
- Implement data schema and data management improvements for scale and performance
- Provide insights into key performance indicators for the product and customer usage
- Serve as team's authority on data infrastructure, privacy controls and data security
- Collaborate with appropriate stakeholders to understand user requirements
- Support efforts for continuous improvement, metrics and test automation
- Maintain operations of live service as issues arise on a rotational, on‑call basis
- Verify whether data architecture meets security and compliance requirements and expectations
- Design, develop, and maintain scalable data pipelines to support data ingestion, processing, and storage
- Implement data integration solutions to consolidate data from multiple sources into a centralized data warehouse or data lake
- Collaborate with data scientists and analysts to understand data requirements and translate them into technical specifications
- Ensure data quality and integrity by implementing robust data validation and cleansing processes
- Optimize data pipelines for performance, scalability, and reliability
- Develop and maintain ETL (Extract, Transform, Load) processes using tools such as Apache Spark, Apache NiFi, or similar technologies
- Monitor and troubleshoot data pipeline issues, ensuring timely resolution and minimal downtime
- Implement best practices for data management, security, and compliance
- Document data engineering processes, workflows, and technical specifications
- Stay up‑to‑date with industry trends and emerging technologies in data engineering and big data
Requirements
- Bachelor's degree in computer science, computer engineering or a related field, or equivalent
- 6+ years of progressive experience demonstrating strong architecture, programming and engineering skills
- Firm grasp of data structures, algorithms with fluency in programming languages like Java, Python, Scala
- Strong SQL language skills and ability to write complex queries
- Experience with Airflow or similar orchestration tools
- Demonstrated ability to lead, partner, and collaborate cross‑functionally across many engineering organizations
- Experience with streaming technologies such as Apache Spark, Kafka, Flink
- Backend experience including Apache Cassandra, MongoDB and relational databases such as Oracle, PostgreSQL
- Experience with AWS/GCP
- Solid hands‑on experience with 4+ years of experience
- Strong communication and soft skills
- Knowledge and/or experience with containerized environments, Kubernetes, Docker
- Experience in implementing and maintaining highly scalable micro services in Rest, Spring Boot, GRPC
Nice to have
- Appetite for trying new things and building rapid POCs
- Ability to fast learn and quickly adapt at rapid pace
- Java/Scala, SQL proficiency (emphasized as a plus)
- Familiarity with emerging data engineering and big data technologies
Benefits
- Opportunity to create wealth through involvement with 500 million+ registered users and 25 million+ merchants
- Chance to be part of India's largest digital lending story
- Contribution to democratizing credit for deserving consumers and merchants
- Exposure to cutting‑edge data infrastructure, privacy controls, and security at scale
- Work on high‑impact, large‑scale financial services platform
Additional details
- Paytm is India's leading mobile payments and financial services distribution company.
- Pioneer of the mobile QR payments revolution in India, Paytm builds technologies that help small businesses with payments and commerce.
- Paytm’s mission is to serve half a billion Indians and bring them to the mainstream economy with the help of technology.