Data Engineer (Spark)
Addepto
Skill Required
Key highlights
- Required experience: At least 4 years of commercial Big Data experience
- Notable requirement: Fluent English (C1 level)
- Notable requirement: Consulting experience with direct client interaction
- Key benefit: Flexible work arrangements (remote or modern offices)
- Key benefit: Sponsored training and certifications via Databricks and Anthropic
Role overview
Addepto is a leading AI consulting and data engineering company, part of the US-based global technology group KMS Technology. Recognized by Forbes as one of the top 10 AI consulting companies worldwide, Addepto builds scalable, ROI-focused AI solutions for large enterprises and startups, while also developing its own product, ContextClue, and contributing to open-source AI solutions. As a Data Engineer, you will work with a team of technology experts on challenging projects across various industries, including the development of a large automotive data processing platform, a universal data platform for global aerospace companies using Azure and Databricks, and a centralized reporting platform for a US telecommunications company using BigQuery and Looker.
Responsibilities
- Develop and maintain a high-performance data processing platform for automotive data, ensuring scalability and reliability.
- Design and implement data pipelines that process large volumes of data in both streaming and batch modes.
- Optimize data workflows to ensure efficient data ingestion, processing, and storage using technologies such as Spark, Cloudera, and Airflow.
- Work with data lake technologies (e.g., Iceberg) to manage structured and unstructured data efficiently.
- Collaborate with cross-functional teams to understand data requirements and ensure seamless integration of data sources.
- Monitor and troubleshoot the platform, ensuring high availability, performance, and accuracy of data processing.
- Leverage cloud services (AWS) for infrastructure management and scaling of processing workloads.
- Write and maintain high-quality Python (or Java/Scala) code for data processing tasks and automation.
Requirements
- At least 4 years of commercial experience implementing, developing, or maintaining Big Data systems, data governance and data management processes.
- Strong programming skills in Python (or Java/Scala): writing a clean code, OOP design.
- Hands-on with Big Data technologies like Spark, Cloudera, Kafka, Data Platform, Airflow, NiFi, Docker, and Iceberg.
- Excellent understanding of dimensional data and data modeling techniques.
- Experience implementing and deploying solutions in cloud environments.
- Consulting experience with excellent communication and client management skills, including prior experience directly interacting with clients as a consultant.
- Ability to work independently and take ownership of project deliverables.
- Fluent English (at least C1 level).
- Bachelor’s degree in technical or mathematical studies.
Nice to have
- Experience with an MLOps framework such as Kubeflow or MLFlow.
- Familiarity with Databricks and/or dbt.
Benefits
- Work in a supportive team of passionate enthusiasts of AI & Big Data.
- Engage with top-tier global enterprises and cutting-edge startups on international projects.
- Enjoy flexible work arrangements, allowing you to work remotely or from modern offices and coworking spaces.
- Accelerate your professional growth through career development paths, knowledge-sharing initiatives, language classes, and sponsored training and conferences.
- Benefit from partnerships with Databricks and Anthropic, which provide access to industry-leading training materials and certification programs.
- Participate in team-building events and utilize the integration budget.
- Celebrate work anniversaries, birthdays, and milestones.
- Access medical and sports packages, eye care, and well-being support services, including psychotherapy and coaching.
- Get full work equipment for optimal productivity, including a laptop and other necessary devices.
- With our backing, you can boost your personal brand by speaking at conferences, writing for our blog, or participating in meetups.
- Experience a smooth onboarding with a dedicated buddy, and start your journey in our friendly, supportive, and autonomous culture.
Additional details
- Originally posted on Himalayas