Data Engineer (Databricks)
Addepto
WorldwideremotePosted 1 month ago
Skill Required
Data-EngineeringBig-DataData-Platform-EngineeringCloud-Data-EngineeringDatabricksData-Pipeline-EngineeringDatabricks-Data-EngineerDatabricks-EngineerData-Engineering-(Databricks)Data-EngineerDatabricks-EngineeringGCP-Databricks-EngineerData EngineerMachine Learningdata engineeringAirflowrelated fieldEngineeringR ProgrammingdevelopingObservabilityPower BIdesigningbuildingSparkAzurePythondesignCI/CDCloudRustGenerative AIdbtandDesign PatternsCICDFulltime
Key highlights
- Required experience: 3+ years of commercial Big Data experience
- Key benefit: Flexible work arrangements (remote or modern offices)
- Notable requirement: Bachelor’s or Master's degree in Computer Science, Data Science, Mathematics, Physics, or related field
- Key benefit: Sponsored training, conferences, and partnerships with Databricks and Anthropic
Role overview
Addepto is a leading AI consulting and data engineering company that builds scalable, ROI-focused AI solutions for some of the world's largest enterprises and pioneering startups, including Rolls Royce, Continental, Porsche, ABB, and WGU. As part of KMS Technology, a US-based global technology group, Addepto combines deep AI specialization with enterprise-scale delivery capabilities—enabling the partnership to move clients from AI experimentation to production impact, securely and at scale. As a Data Engineer, you will have the exciting opportunity to work with a team of technology experts on challenging projects across various industries, leveraging cutting-edge technologies.
Responsibilities
- Design scalable data processing pipelines for streaming and batch processing using Big Data technologies like Databricks, Airflow and/or Dagster.
- Contribute to the development of CI/CD and MLOps processes.
- Develop applications to aggregate, process, and analyze data from diverse sources.
- Collaborate with the Data Science team on Machine Learning projects, including text/image analysis and predictive model building.
- Develop and organize data transformations using Databricks/DBT and Apache Airflow.
- Translate business requirements into technical solutions and ensure optimal performance and quality.
Requirements
- At least 3+ years of commercial experience implementing, developing, or maintaining Big Data systems.
- Strong programming skills in Python: writing a clean code, OOP design.
- Experience in designing and implementing data governance and data management processes.
- Familiarity with Big Data technologies like Airflow or Dagster, Databricks, Spark and DBT.
- Experience implementing and deploying solutions in cloud environments (with a preference for Azure).
- Knowledge of how to build and deploy Power BI reports and dashboards for data visualization.
- Excellent understanding of dimensional data and data modeling techniques.
- Excellent communication skills and consulting experience with direct interaction with clients.
- Ability to work independently and take ownership of project deliverables.
- Bachelor’s or Master's degree in Computer Science, Data Science, Mathematics, Physics, or a related field.
Benefits
- Work in a supportive team of passionate enthusiasts of AI & Big Data.
- Engage with top-tier global enterprises and cutting-edge startups on international projects.
- Enjoy flexible work arrangements, allowing you to work remotely or from modern offices and coworking spaces.
- Accelerate your professional growth through career development paths, knowledge-sharing initiatives, language classes, and sponsored training and conferences.
- Benefit from partnerships with Databricks and Anthropic, which provide access to industry-leading training materials and certification programs.
- Participate in team-building events and utilize the integration budget.
- Celebrate work anniversaries, birthdays, and milestones.
- Access medical and sports packages, eye care, and well-being support services, including psychotherapy and coaching.
- Get full work equipment for optimal productivity, including a laptop and other necessary devices.
- With our backing, you can boost your personal brand by speaking at conferences, writing for our blog, or participating in meetups.
- Experience a smooth onboarding with a dedicated buddy, and start your journey in our friendly, supportive, and autonomous culture.
Additional details
- Addepto has developed its own product - ContextClue - and actively contributes open-source solutions to the AI community.
- Addepto has been recognized by Forbes as one of the top 10 AI consulting companies worldwide.
- Project Example: Design and development of a universal data platform for global aerospace companies using Azure and Databricks, combining diverse enterprise and public data sources.
- Project Example: Data Platform Transformation for energy management association body implementing a comprehensive data catalog leveraging Databricks and Apache Spark/PySpark, and utilizing Delta Live Table tests.
- Project Example: Design of the data transformation and following data ops pipelines for global car manufacturer for real-time streaming and batch data, including exploring LLMs for chatbots and data analysis.
- Originally posted on Himalayas