Data Pipeline Engineer

Location: India Employment Type: Full-Time Experience: Senior (7+ Years) Roles & Responsibilities Design, build, and maintain scalable batch and real-time data ingestion pipelines. Develop and manage Bronze, Silver, and Gold (Medallion) data layers. Implement CDC, watermarking, checkpointing, and batch-to-stream data processing. Build robust data quality, validation, reconciliation, and monitoring frameworks. Develop identity resolution, deduplication, and Golden Record (MDM) solutions. Create and maintain source-to-canonical data mappings and crosswalks. Ensure schema validation, versioning, and data contract enforcement. Collaborate with cross-functional teams to onboard new data sources and optimize data pipelines. Mandatory Skills 7+ years of experience in Data Engineering / Data Pipeline development. Strong experience with Apache Kafka (Producers, Consumers, Replay, DLQ, Exactly-once/Idempotent processing). Strong SQL and ETL/ELT fundamentals. Hands-on experience in Java and/or Python. Experience with CDC, Batch & Streaming pipelines, and Medallion/Lakehouse architecture. Experience implementing Data Quality, Validation, and Reconciliation frameworks. Knowledge of Master Data Management (MDM), Identity Resolution, Deduplication, and Golden Record concepts. Experience with source-to-target mapping, canonical data models, YAML/JSON configuration, and Git. Good to Have Experience with probabilistic record matching and record linkage. Schema Registry (Avro/Protobuf). Experience extracting data from legacy/Mainframe systems. Financial reconciliation experience. Healthcare or Benefits Administration domain experience. Originally posted on Himalayas