Posted today · be early
Senior Data Platform Engineer
Mirantis
IndiaremotePosted 1 day ago
Skill Required
Data-Platform-EngineerPlatform-EngineerEvent-Streaming-EngineeringKafka-EngineerInfrastructure-EngineerSenior-Data-Management-Platform-EngineerSenior-Cloud-Data-Platform-EngineerSenior-Data-Infrastructure-EngineerSenior-Data-EngineeringSenior-Data-Management-EngineerDataOps-EngineerData EngineerPlatform EngineerBackup and RecoveryMachine LearningGitHub Actionstroubleshootingrelated fieldElasticsearchObservabilityKafkaEngineeringR ProgrammingPostgreSQLKubernetesPrometheusISO 27001networkinganalyticalautomationCryptographySnowflakeTerraformdebugginganalyticsBigQuerysecurityJenkinsFulltime
Key highlights
- 10+ years of required experience in software, data, or platform engineering.
- Expert‑level proficiency with Apache Kafka (or Confluent) and Kafka Connect.
- Competitive compensation package with a strong benefits plan.
- Professional development and training opportunities.
- Work with a Silicon Valley leader in cloud infrastructure and AI‑focused Kubernetes platforms.
- Must have experience with Kubernetes, Terraform, and CI/CD automation.
Role overview
We are looking for an experienced data platform engineer to own the event‑streaming backbone behind the k0rdent‑ai platform — the multi‑tenant control plane for enterprise GPU infrastructure. Every cluster provisioned, every GPU‑hour consumed, and every tenant action produces events that must move reliably from our services into analytics, billing, observability, and downstream systems. You will design and operate that pipeline end to end: the streaming platform itself, the connectors and change‑data‑capture flows that feed it, the schemas that keep producers and consumers compatible, and the automation that lets product teams onboard themselves without waiting on you. Working within an agile framework alongside engineering teams across the US, Europe, and India, you will make event data a dependable product surface rather than a best‑effort side channel.
Responsibilities
- Design, deploy, and operate the production streaming platform on Kubernetes — cluster lifecycle, topics, partitioning and retention strategy, capacity planning, and upgrades.
- Build and maintain change-data-capture and connector pipelines moving data between operational databases, the streaming layer, and analytical stores.
- Own schema governance — a schema registry, compatibility rules, and versioning discipline so producer changes never silently break consumers.
- Develop custom connectors, transformations, and stream-processing logic in Java or Go where off-the-shelf components fall short.
- Build self‑service onboarding so product teams can provision topics, schemas, and access through infrastructure as code rather than tickets.
- Automate the platform with Terraform and GitOps — no manually configured clusters, no undocumented topic definitions.
- Instrument the pipeline for production: metrics, lag and throughput monitoring, alerting, and dashboards that both the platform team and application owners can act on.
- Enforce security and multi‑tenant isolation across the streaming layer — mTLS, authentication, topic‑level RBAC, encryption, and audit trails.
- Guarantee delivery semantics that the business can rely on — ordering, idempotency, exactly‑once where required, replay, and disaster recovery.
- Partner with backend and SRE teams on event schema design, producer patterns, and failure modes; participate in on‑call for the platform you own.
- Mentor engineers on streaming architecture and raise the team's bar through design review and documentation.
Requirements
- 10+ years in software, data, or platform engineering, including deep hands‑on ownership of a production event‑streaming platform.
- Expert‑level Apache Kafka (or Confluent Platform/Cloud) — administration, tuning, troubleshooting, and capacity management at production scale.
- Strong Kafka Connect and CDC experience — Debezium, JDBC, and custom connector or transformation development.
- Proven schema‑registry practice and compatibility management across many independent producers and consumers.
- Solid programming ability in Java, Python, or Go — enough to build connectors, transformations, and platform tooling, not only configure them.
- Strong SQL and relational database knowledge (PostgreSQL, MySQL, or equivalent), including replication and CDC mechanics.
- Kubernetes fluency — deploying, operating, and debugging stateful workloads.
- Infrastructure as code discipline (Terraform or equivalent) and CI/CD pipeline ownership.
- Clear written English for design docs, runbooks, and asynchronous review across global time zones.
- Streaming: Apache Kafka or Confluent, Kafka Connect, Schema Registry, and stream processing.
- Data Movement: CDC tooling (Debezium or equivalent), connector development, and pipelines into analytical stores.
- Kubernetes Native: Kubernetes, Docker, and Helm‑packaged workloads in production.
- Automation: Terraform, CI/CD (Jenkins, GitHub Actions, or equivalent), and GitOps practice.
- Data Stores: PostgreSQL at scale, plus at least one cloud data warehouse or search store (Snowflake, BigQuery, Elasticsearch, or equivalent).
- Cloud: AWS or GCP — networking, IAM, managed Kubernetes.
- Security & Observability: mTLS, OIDC/SSO, RBAC, and Prometheus/Grafana monitoring.
Nice to have
- Confluent certification, or contributions to Kafka‑ecosystem open source.
- Stream processing with Flink, Kafka Streams, or ksqlDB.
- Event‑driven architecture patterns — outbox, saga, event sourcing, CloudEvents.
- Exposure to GPU infrastructure, AI/ML data pipelines, or telemetry at high cardinality.
- Usage metering, billing, or chargeback pipelines built on event data.
- Alternative brokers (NATS JetStream, Pulsar) and a view on the tradeoffs.
- Go, for working directly in our backend services' producer code.
- Data quality, lineage, or catalog tooling.
- Compliance exposure — SOC 2, ISO 27001, or similar audit support.
Benefits
- Work with an established Silicon Valley leader in the cloud infrastructure industry.
- Work with exceptionally passionate, talented and engaging colleagues, helping Fortune 500 and Global 2000 customers implement next‑generation cloud technologies.
- Be a part of cutting‑edge, open‑source innovation.
- Thrive in the high‑energy environment of a young company where openness, collaboration, risk‑taking, and continuous growth are valued.
- Professional development and training.
- Attend conferences and working groups.
- Company outings, happy hours, hackathons, and tech talks.
- Receive a competitive compensation package with a strong benefits plan.
Additional details
- Mirantis, an IREN company, is the Kubernetes‑native AI infrastructure company, enabling organizations to build and operate scalable, secure, and sovereign infrastructure for modern AI, machine learning, and data‑intensive applications.
- By combining open source innovation with deep expertise in Kubernetes orchestration, Mirantis empowers platform engineering teams to deliver composable, production‑ready developer platforms across any environment—on‑premises, in the cloud, at the edge, or in sovereign data centers.
- As enterprises navigate the growing complexity of AI‑driven workloads, Mirantis delivers the automation, GPU orchestration, and policy‑driven control needed to manage infrastructure with confidence and agility.
- Committed to open standards and freedom from lock‑in, Mirantis ensures that customers retain full control of their infrastructure strategy.
- We are a Leader for Container Management in G2 (#2 after AWS)!
- Originally posted on Himalayas.
