Data Platform Engineer (102-08SENG-01)

OpsBrasil Serviços Cloud LTDA

WorldwideremotePosted 16 days ago

Skill Required

Data-Platform-EngineeringData-EngineeringCloud-Data-EngineeringDatabricks-EngineerData-MigrationData-Platform-EngineerSenior-Data-Management-Platform-EngineerAI-Data-Platform-EngineerCloud-Data-Platform-EngineerData-Platforms-EngineeringLead-Data-and-AI-Platform-EngineerData EngineerPlatform EngineerDatabricksServerlessTerraformanalyticsbuildingAirflowwrittenSparkPythonKafkaAzureUnityDesign PatternsdbtSQLAWSandContract

Key highlights

Databricks
PySpark
Python
Azure Data Factory
AWS (S3, Glue, Athena, Lambda, Step Functions)
SQL

Role overview

This role converts a large Azure Data Factory estate into Databricks workflows on AWS. The scope for year one includes 2,094 ADF pipelines to migrate as reusable templates, 9,859 pipeline activities to translate (some mapping directly, others requiring rewriting as Lambda or Step Functions), 471 Spark dataflows to move onto Databricks on AWS, and 4 Databricks workspaces (Dev, QA, Pre-prod, Prod) to rehost, including notebook paths and Unity Catalog rewiring. This is a regulated environment, so reconciling migrated data against source systems is part of the definition of done, not an afterthought.

Responsibilities

Convert Azure Data Factory pipelines into Databricks workflows on AWS, building reusable templates rather than migrating one at a time.
Rehost Databricks workspaces onto AWS and migrate ADLS Gen2 storage to S3.
Rewrite ADF Web Activities as Lambda functions or Step Functions tasks, and replace ADF-specific scaling with native Databricks mechanisms.
Build and tune PySpark transformations for production data volumes.
Replace Azure Synapse Serverless with Databricks SQL Warehouse.
Reconcile migrated data against source systems as part of the definition of done.

Requirements

Production experience with Databricks: workspaces, jobs and workflows. The central skill for this role.
Strong Spark and PySpark experience for real data volumes, including tuning.
Production-grade Python.
Experience building or migrating Azure Data Factory pipelines, with a solid understanding of the ADF activity model.
AWS data services: S3, Glue, Athena, Lambda and Step Functions.
Advanced SQL, including reading and reasoning about stored procedures.
Professional written and spoken English.