Data Engineer w/ Databricks
Codest Ltd. Company No. 12590542, VAT number: GB363431020
WorldwideremotePosted 10 days ago
Skill Required
Data-EngineerBig-Data-EngineerData-Platform-EngineerAnalytics-EngineerDatabricks-Data-EngineerDatabricks-EngineerData-Engineering-(Databricks)Senior-Databricks-EngineerSenior-Databricks-DeveloperData-Engineer-JobsData EngineerDatabricksdata engineeringDockerKubernetesEngineeringanalyticaldevelopingObservabilitydesigninganalyticsMLflowRedshiftbuildingSparkTestNGPythondesignDevOpsCI/CDUnityCloudDesign PatternsAPIsETLSQLFulltime
Key highlights
- The Codest is an International Tech Software Company with tech hubs in Poland delivering global IT solutions
- They are seeking an experienced Databricks Engineer to design, build, and manage scalable data solutions and pipelines using Databricks
- The position requires 6+ years of experience in data engineering with a proven track record of production-grade data pipelines and ETL/ELT workflows
- Mandatory proficiency in Python, PySpark, and SQL is required
- Extensive hands-on experience with AWS services such as Glue, Lambda, Redshift, and S3 is essential
- A Databricks Certified Professional Data Engineer (or equivalent certification) is a notable preference
Role overview
The Codest is an International Tech Software Company with tech hubs in Poland delivering global IT solutions. They specialize in web development, cloud engineering, DevOps, and quality, having previously developed Yieldbird which was honored as a laureate of the Top25 Deloitte awards. Their current mission is to help tech companies build impactful products and scale their IT teams through improving IT delivery performance. They are seeking a Data Engineer with Databricks expertise to contribute to a banking app for a leading financial group in Japan.
Responsibilities
- Experience with Medallion Architecture (Bronze, Silver, Gold) for lakehouses using optimized performance patterns
- Build strong data quality frameworks with automated testing and monitoring
- Implement Delta Lake features such as time travel, vacuum operations, and Z-ordering
- Develop and maintain complex ETL/ELT pipelines processing large-scale datasets daily
- Design and implement CI/CD workflows for data pipelines using Databricks Asset Bundles or equivalent tools
- Create real-time and batch data processing solutions with Structured Streaming and Delta Live Tables
- Optimize Spark jobs for cost efficiency and performance, leveraging cluster auto-scaling and resource management
- Develop custom integrations with Databricks APIs and external systems
- Design scalable data architectures using Unity Catalog, Delta Lake, and Apache Spark
- Establish data mesh architectures with governance and lineage tracking
Requirements
- 6+ years of experience in data engineering, with a strong track record of designing and deploying production-grade data pipelines and ETL/ELT workflows
- Proficiency in Python, PySpark, and SQL
- Extensive hands-on experience with AWS services such as Glue, Lambda, Redshift, and S3
- Solid background in data governance and management tools, including platforms like Unity Catalog and AWS Sagemaker Unified Studio
- Proven experience in data migration initiatives and modernizing legacy systems
- Communicative English language skills (the project is in an international team)
Nice to have
- Databricks Certified Professional Data Engineer (or equivalent certification)
- Excellent communication and teamwork skills
- Strong analytical thinking and problem-solving ability
- Experience with containerization and orchestration tools (e.g., Docker, Kubernetes) is considered an advantage
Additional details
- The Codest operates with core values centered on "Customers and People First"
- Expertise areas include web development, cloud engineering, DevOps, and quality
- Past success includes developing Yieldbird, which was honored as a laureate of the prestigious Top25 Deloitte awards
- The role involves contributing to a banking app for one of the leading financial groups in Japan
- The platform is equipped with bank modules and data management features and is customer-facing
- Recruitment process consists of a 30-minute screening call online with the recruiter, a 45-60 minute technical call with an engineer, and a 1-hour call with the team leader
- Contact the recruiting team via email for questions or insights regarding the offer