Databricks Architect - India
Cogniify
IndiaremotePosted 29 days ago
Skill Required
Databricks-ArchitectData-ArchitectData-EngineeringAnalytics-EngineeringLakehouse-ArchitectureSenior-Databricks-ArchitectDatabricks-Solution-ArchitectDatabricks-Solutions-ArchitectDatabricks-ConsultantDatabricksQuery Optimizationdata engineeringrelated fieldObservabilityEngineeringStatisticsanalyticssecurityAirflowSparkPythonMLflowdesignKafkaFlinkAzureUnityCloudDesign PatternsISO 27001RAGdbtSQLAWSFulltime
Key highlights
- Senior/Specialist level role
- 6–9 years of professional experience required
- Competitive salary, equity options, and continuous learning opportunities
- Flexible, remote-first culture
- Join a team of industry veterans from Google, Meta, and top-tier tech companies
- Work on impactful, high-scale data and analytics projects with leading global clients
Role overview
We're seeking a Databricks Architect (Senior/Specialist level) to serve as a senior technical authority for data architecture and analytics engineering on the Databricks Lakehouse platform across Cognify Analytics and our client engagements. In this role, you will design and own Databricks-based data platform architecture, lead complex data engineering initiatives, and ensure our data capabilities are enterprise-grade, governed, and future-ready. You will bridge data engineering, analytics, and AI/ML — combining deep hands-on expertise in Databricks, Python, and SQL with strong architectural judgment and stakeholder communication.
Responsibilities
- Own the architecture and design of Databricks-based data platforms, including lakehouse design (Delta Lake), medallion architecture, and unified analytics layers.
- Serve as the senior technical authority on Databricks platform design, providing guidance on architecture, tooling, modeling, and engineering standards across teams and projects.
- Design and build ingestion, transformation, and orchestration pipelines using Python, SQL, PySpark, and Databricks Workflows.
- Architect data mesh and data product strategies, defining domain ownership, data contracts, and self-service consumption patterns on Databricks.
- Establish and evangelize best practices across the data lifecycle: ingestion, transformation, modeling, quality, observability, governance, and consumption.
- Drive the integration of Databricks with AI/ML capabilities, including feature engineering pipelines, vector data infrastructure for RAG, and MLflow-based model workflows.
- Lead complex data migration, platform modernization, and consolidation initiatives for enterprise clients across industries.
- Evaluate emerging data technologies (Apache Iceberg, Unity Catalog, Delta Live Tables, Mosaic AI) to inform architectural decisions.
- Solve complex, ambiguous, high-impact data architecture problems that span multiple teams, platforms, or organizational boundaries.
- Drive cross-functional alignment between data engineering, analytics, AI/ML, platform engineering, security, and product teams.
- Mentor senior data engineers and analysts, fostering a culture of technical excellence.
- Represent Cognify Analytics in discussions with client stakeholders and leadership on data capabilities, strategy, and technical roadmaps.
- Define and enforce data governance, compliance (GDPR, HIPAA, SOC2), and responsible data management standards across all Databricks environments.
- Drive FinOps maturity for the Databricks platform, including compute optimization, cluster policies, storage lifecycle management, and cost forecasting.
Requirements
- Databricks — hands-on architecture and engineering experience on the Databricks Lakehouse Platform (production-grade, at scale)
- Python — strong professional proficiency for data engineering and pipeline development
- SQL — advanced proficiency for data modeling, transformation, and performance tuning
- Bachelor's, Master's, or equivalent professional experience in Computer Science, Data Science, Statistics, or a related field.
- 6–9 years of professional experience in data engineering, analytics engineering, or data architecture, with demonstrated technical leadership on at least a few large-scale engagements.
- Strong working knowledge of the modern data stack: dbt, Airflow/Dagster, Fivetran/Airbyte, Spark, Kafka, and cloud-native data services.
- Solid grasp of data modeling methodologies (Kimball, Data Vault, Activity Schema, OBT) and the judgment to apply them across contexts.
- Experience with cloud data infrastructure on AWS, Azure, or GCP (any combination is acceptable, alongside Databricks).
- Proven ability to influence technical direction and align technical and business stakeholders on complex architecture topics.
- Strong understanding of data governance, data quality, cataloging, lineage, and regulatory compliance frameworks.
- Experience mentoring engineers and contributing to a high-performing data team.
- Strong understanding of how data platforms serve AI/ML workloads, including feature engineering, vector data, and model input/output pipelines.
Nice to have
- Experience architecting Databricks-based platforms that directly serve LLM-based systems, RAG pipelines, and agentic AI architectures at enterprise scale.
- Deep familiarity with lakehouse table formats: Delta Lake, Apache Iceberg, and Apache Hudi.
- Track record of implementing data mesh, data product, or federated data governance patterns.
- Experience with real-time analytics and streaming architectures: Kafka, Flink, Spark Structured Streaming.
- Experience with advanced Databricks features: Unity Catalog, Delta Live Tables, Databricks Workflows, MLflow, and Mosaic AI.
- Contributions to open-source data projects, data architecture publications, or industry standards bodies.
- Background in financial services, healthcare, SaaS, or enterprise consulting requiring high data compliance, security, and operational rigor.
- Experience leading data engineering across geographically distributed teams and multi-client engagements.
Benefits
- Unlimited PTO
- Generous parental leave, above industry standards
- Open communication with management and company leadership
- Small, dynamic teams = massive impact
- Medical, Dental, and Vision coverage
- Access to Disability & Life insurance
- Mental health and wellbeing support
- Annual bonus program
- Employer Stock Purchase Program (ESPP)
- Yearly team building experiences
- Mentorship and sponsorship opportunities
- Manager resources and support
Additional details
- We are an equal opportunity employer. We celebrate diversity and are committed to creating an inclusive environment for all employees. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, veteran status, or any other protected characteristic.
- Originally posted on Himalayas