Data Engineer

We are seeking a skilled Data Engineer responsible for designing, developing, and maintaining scalable data pipelines and data platforms. The role focuses on building reliable data ingestion and transformation processes using modern data engineering tools and frameworks. The ideal candidate will have strong experience with Python-based data engineering, SQL, and the Databricks platform, along with a solid understanding of data architecture and data modeling. This role will collaborate closely with analytics, platform, and business teams to ensure high-quality, well-structured data is available for reporting, analytics, and advanced data use cases. Key Responsibilities Design, develop, and maintain scalable data pipelines for ingesting, transforming, and processing large volumes of data. Build and manage ELT/ETL pipelines using Python and modern data engineering frameworks. Utilize the Databricks platform to process and manage distributed data workloads. Develop and optimize Python-based data transformation logic using libraries such as Pandas, Polars, and Delta Live Tables (DLT). Implement and maintain data transformations using dbt and SQL. Design and manage data schemas and data models to support analytical and operational workloads. Build and maintain data ingestion pipelines integrating with various ELT tools such as cData, Hevo, MuleSoft, or Fivetran. Monitor, troubleshoot, and resolve issues in data pipelines and workflows to ensure reliability and performance. Apply data architecture principles, including data lake structures and medallion architecture (Bronze, Silver, Gold layers). Optimize data processing and transformation logic for performance, scalability, and maintainability. Document data pipelines, transformations, and data models to ensure transparency and knowledge sharing. Collaborate with cross-functional teams to understand data requirements and deliver reliable data solutions. Use version control systems (Git) and follow engineering best practices for collaborative development. Participate in Agile/Scrum ceremonies and contribute to continuous improvement of engineering processes. Essential Qualifications Strong experience with the Databricks platform for building and managing data workloads. Proficiency in Python for data engineering, including libraries such as Pandas, Polars, and DLT. Strong understanding of ELT/ETL concepts and experience building end-to-end data ingestion pipelines. Hands-on experience building Python-based ELT pipelines. Strong knowledge of data schema design and data modeling concepts. Experience writing efficient and scalable data transformation logic. Ability to debug and troubleshoot data pipelines effectively. Good understanding of distributed computing and storage concepts. Experience using dbt for data transformations. Knowledge of data lake architecture and medallion architecture. Experience with pipeline orchestration, monitoring, and troubleshooting. Familiarity with Git-based version control systems. Experience working in Agile development environments. Data Handling: Large dataset management, manipulation, cleansing, pattern identification, organization Excel & Analysis: Advanced formulas, pivot tables, data summarization, visualization, handling large Excel files Data Understanding: Data structures, types, relationships, structured vs unstructured data, dependencies Analytical Thinking: Problem solving, real-world business context thinking, breaking down complex issues, logical reasoning Reporting: Preparing reports, communicating insights, explaining data issues, supporting decision-makin Nice to Have Skills Experience with ELT tools such as cData, Hevo, MuleSoft, Fivetran, or similar tools. Exposure to Salesforce or NetSuite data schemas. Knowledge of vector databases or vector storage platforms such as Weaviate. Understanding of embeddings and vector-based data concepts. Experience with LLM tooling such as Ollama or OpenAI APIs. Familiarity with data validation frameworks such as Great Expectations. Agile or Scrum Master certification. Soft Skills Strong analytical and problem-solving skills. Ability to work collaboratively with cross-functional teams. Good communication and documentation skills. Strong attention to detail and focus on data quality. Ability to manage multiple priorities in a fast-paced environment. Proactive mindset with a focus on continuous improvement and learning. Benefits Comprehensive benefits package including health insurance, paid time off, and professional development opportunities. About Potentiam Potentiam is a global provider of highly qualified professionals to European SMEs from our offices in Romania, South Africa and India. Potentiam works with clients in finance, energy, leisure, marketing, business services and technology industries, providing technical, professional multi- lingual highly motivated staff, most of whom have had experience of working for international companies. Staff cover a wide range of roles from accounting, marketing, data management, HR, sales/account management, engineering, technology, and operations. Potentiam manages our staff’s career development and personal development training, all infrastructure, HR and payroll with our clients directly managing day-to-day staff responsibilities and role training and development. If interested please apply here, if you have any questions regarding the role, please feel free to write to Indiajobs@potentiam.co.uk Data Privacy Notice The personal information you provide during the application and recruitment process will be used solely for recruitment purposes, in accordance with our data protection policies. For any questions regarding data processing related to HR activities, please contact at Indiahr@potentiam.co.uk All data shared with third parties complies with applicable confidentiality and retention requirements.