Senior Search Engineer, Index Team

Kagi Inc

WorldwideremotePosted 1 month ago

Skill Required

Search-Index-EngineeringBackend-EngineeringSearch-InfrastructureDistributed-Systems-EngineeringSearch-IndexSenior-Search-EngineerPrincipal-Search-EngineerSenior-Solr-EngineerSearch-Platform-EngineerElasticsearch-EngineerObservabilityEngineeringPostgreSQLPrometheusdesigningsimilar)Query OptimizationPythondesignRedisjQueryRustandFulltime

Key highlights

5+ years of backend engineering experience

Role overview

Responsibilities

Embed within the Search team to ensure index improvements translate into better results for user
Design and improve the URL frontier and crawl job scheduling system - how we decide what to index, when, and in what order
Own and evolve the indexing strategy (what we index post-crawl) and ranking implementation
Investigate and fix index quality issues - spam, stale content, deduplication, coverage gaps

Requirements

5+ years of backend engineering experience in production information retrieval/search systems
Prior experience with search index infrastructure, URL frontier systems, or crawl scheduling
Deep expertise across the information retrieval stack: index design, relevance scoring and ranking optimization, near-duplicate detection, query engine architectures, and graph-based signal extraction
Strong database skills - you're comfortable designing and optimizing complex PostgreSQL schemas and queries, including large job/task tables that need to scale
Proficiency in Python and/or Rust - our primary languages for this work
Ability to reason about large-scale data pipelines: throughput, correctness, idempotency, and operational reliability
Experience with observability tooling (Prometheus, Grafana or similar)
Comfort with Redis for distributed state and queue management
Experience with distributed job scheduling or work queue systems - you understand the failure modes and edge cases
Experience operating systems where correctness and data quality are the primary constraints