Posted today · be early
Data Processing Engineer - I/O
DataPelago
India, United StatesremotePosted 1 day ago
Skill Required
Data-Processing-EngineerData-EngineerSystems-EngineerSoftware-EngineerFile-Processing-EngineerData-Processing-EngineeringData-Ingestion-EngineerData-Processing-SpecialistData-Management-EngineerData EngineerSystem DesignQuery OptimizationMachine Learningrelated fieldEngineeringR ProgramminganalyticaldevelopinganalyticssecuritybuildingTestNGdesignC++SparkFlinkLinuxCI/CDCloudGenerative AIjQueryRustandCICDFulltime
Key highlights
- Compensation: Competitive compensation and stock options
- Experience: 5-7+ years depending on degree
- Locations: Mountain View, CA, Hyderabad, IN, or Remote
- Requirement: Exceptional C/C++ skills
- Requirement: 3+ years deep technical experience in data read/write interfaces
Role overview
DataPelago is building an innovative data processing engine transforming how Apache Spark, Apache Flink, Ray and others operate on diverse, large-scale data. As a Data Processing Engineer - I/O, you will be a key individual contributor advancing the data read and write capabilities of the engine, enhancing functional breadth, performance, scale, and reliability across various data types, sources, and sinks.
Responsibilities
- Architect: Influence the architecture of how our data processing engine interfaces with datasources and sinks, catalogs, data formats.
- Design: Lead design of functional and performance enhancements to adapters/connectors, data representations, data filtering, caching and more in our data processing engine.
- Core Development: Individually design, implement, test, optimize, and maintain components of the data processing engine.
- Innovation and Differentiation: Analyze technology roadmap of existing and emerging dataformats and libraries, open table formats, catalog services, and more (e.g., Apache Arrow, Apache Parquet, Apache Iceberg) and identify opportunities for our engine to enhance technology and product leadership.
- Collaboration: Partner effectively with engineering and product management in defining and realizing the data I/O roadmap of our product.
- Continuous Improvement: Foster best practices in design and code reviews, testing, CI/CD, and issue resolution to maintain highest product quality, security, efficiency, & productivity.
Requirements
- Bachelor's degree in Computer Science or a related field with 7+ years of relevant experience OR a Master's degree in Computer Science or a related field with 5+ years of relevant experience.
- 3+ years of deep technical experience in developing and optimizing data read and write interfaces for large-scale data processing, particularly related to Apache Parquet, Apache ORC, Apache Iceberg, Apache Spark, and similar technologies.
- Demonstrated experience in instrumenting, analyzing, and optimizing the performance of data processing engine components on benchmark and customer workloads.
- Demonstrated experience in the design, development, and successful release of high-performance data processing engine features for large production deployments.
- Good knowledge of the architecture of one or more of Apache Spark, Apache Flink, Presto/Trino.
- Exceptional programming skills in C, C++.
- Extensive development experience in Linux environments.
- Strong analytical and problem-solving skills with a passion for performance optimization.
Nice to have
- Rust experience preferred.
Benefits
- Competitive compensation
- stock options
- comprehensive benefits package
- leadership development opportunities
- Technology Leadership: Shape the architecture and development of how our core engine works with advanced data store platforms.
- Cutting-Edge Innovation: Work on challenging problems at the forefront of accelerated computing and data processing.
- Significant Impact: Your contributions will directly impact the performance and scalability of our mission-critical platform.
- Growth: Expand your technical expertise and scope of responsibilities working with other talented engineers and with a growing product.
Additional details
- Mountain View, CA / Hyderabad, IN / Remote
- We value face-to-face collaboration, but recognize that talent can be found anywhere. Our engineering team works at our headquarters in Mountain View, CA, at our India office in Hyderabad, and at remote locations.
- Originally posted on Himalayas
