Seeking an experienced Data Engineer to design, build, and optimize scalable data pipelines and data platforms using PySpark, Databricks, SQL, and Cloud technologies. The ideal candidate will have expertise in big data processing, ETL development, and modern data lakehouse architectures.
Responsibilities
- Design and develop scalable ETL/ELT pipelines using PySpark and Databricks.
- Build and maintain data ingestion frameworks from multiple data sources.
- Develop and optimize Spark jobs for large-scale data processing.
- Design and manage data lake and lakehouse solutions.
- Implement Delta Lake and medallion architecture best practices.
- Ensure data quality, governance, and security compliance.
- Monitor, troubleshoot, and optimize data pipeline performance.
- Collaborate with Data Scientists, Analysts, and Business teams.
- Support real-time and batch data processing requirements.
- Participate in code reviews, technical design, and Agile delivery.
Requirements
- 5–8 years of experience in Data Engineering.
- Strong hands-on experience with PySpark and Databricks.
- Proficiency in Python and Advanced SQL.
- Experience with Spark, Delta Lake, and Lakehouse architecture.
- Hands-on experience with Azure Databricks, ADF, ADLS, or equivalent cloud services.
- Strong understanding of ETL/ELT frameworks and data warehousing concepts.
- Experience with data modeling and large-scale data processing.
- Knowledge of Kafka, Spark Streaming, or real-time data processing tools.
- Familiarity with Git, CI/CD pipelines, and Agile methodologies.
- Bachelor's degree in Computer Science, Engineering, or a related field.
Nice to have
- Databricks/Azure certifications preferred.
Zobhira
