Posted today · be early
Senior Analytics Engineer
Imagine Worldwide
WorldwideremotePosted 1 day ago
Skill Required
Analytics-EngineerData-EngineerAnalytics-Platform-EngineerData-Platform-EngineerBig-Data-EngineerSenior-Data-Analytics-EngineerAnalytics Engineeranalyticsdata engineeringAirflowEngineeringServerlessnetworkingObservabilityTerraformdesignDevOpsSparkCloudAPIsdbtAWSIAMandFulltime
Key highlights
- Salary: Competitive and commensurate with experience
- Experience Level: At least 7 years total, 3+ years in distributed data processing
- Notable Requirement: Must be based in Africa
- Key Benefit: Full-time remote position
Role overview
The Senior Analytics Engineer builds, runs and improves Imagine’s analytics platform, including the infrastructure, pipelines and core data models that the organisation’s reporting and analysis depend on. The central task is to scale the platform as data volumes grow, keeping costs under control without giving up performance, covering both the underlying infrastructure and the data engineering itself. Reporting to the Director of Analytics, the role involves turning architectural direction into working systems and bringing forward evidence-based proposals for improvement. As the engineer builds context, they take on increasing ownership of the platform’s day-to-day running and technical decisions.
Responsibilities
- Build and maintain the platform’s AWS infrastructure, including storage, compute, networking, the data catalogue, secrets and access management (S3, EC2, Glue Data Catalog, Lake Formation, IAM, Secrets Manager).
- Manage infrastructure as code in Terraform, keeping environments reproducible, version-controlled and reviewable.
- Administer the ClickHouse Cloud warehouse, including service configuration, user and role management, and monitoring of performance and capacity.
- Host and maintain the team’s self-hosted tools, including Apache Superset for BI and Apache Airflow for orchestration, covering deployment, upgrades, authentication and availability.
- Contribute to architectural decisions on how the platform evolves, setting out options and trade-offs for review with the Director of Analytics.
- Build and maintain ingestion pipelines from the platform’s sources, including application databases, APIs and operational systems, using tools such as AWS Glue, Airbyte and AWS Lambda.
- Implement incremental loading patterns that stay efficient and recoverable as volumes grow, landing raw data in the lake in a form that preserves its full history.
- Develop and optimise Spark jobs that process billions of device events into learning sessions and higher-level aggregates, paying close attention to partitioning, skew and compute cost.
- Bring in and model the programme context that gives event data its meaning, such as schools, terms, device allocations and enrolment, and handle how that context changes over time.
- Design and maintain Iceberg tables for performance and cost, including partitioning strategy, file sizing, compaction and snapshot management.
- Validate data at ingestion and surface quality issues early, working with the Analytics Team to trace them back to their source.
- Contribute to evaluating new processing approaches as volumes grow, including moving more work upstream and assessing where streaming would add value.
- Build and maintain Airflow workflows that coordinate ingestion, processing and transformation, with clear dependencies, retries and alerting.
- Develop and maintain dbt staging and core models in ClickHouse, following consistent conventions for naming, structure and materialisation.
- Implement incremental models for high-volume tables, choosing materialisation strategies and table designs that balance freshness, correctness and cost.
- Write and maintain dbt tests and model contracts for core datasets, so that changes that would break downstream reporting are caught before they reach production.
- Document core models such as learning sessions as agreed definitions, including their lineage, and make clear where a model applies an approximation rather than a direct transformation.
- Work with the Senior BI & Data Analyst on the boundary between core and reporting models, and with the Data Scientist on datasets for research and evaluation.
- Build and maintain CI/CD pipelines in GitHub Actions for infrastructure, pipeline and dbt code, including automated checks that stop unsafe changes from reaching production.
- Implement monitoring and alerting across pipelines, the warehouse and hosted tools, so that failures and delays are visible quickly and reach the right people.
- Respond to platform incidents and data issues, carry out root cause analysis, and contribute to the team’s data quality incident process.
- Apply and maintain access controls and security practices across the data lake, warehouse and BI layer, in line with Imagine’s data governance policies.
- Monitor platform costs across AWS and ClickHouse, track the cost and performance of individual workloads over time, and use that evidence to identify savings and prioritise optimisation work.
- Maintain and periodically test backup and recovery procedures for the platform’s core components.
- Take part in code review and write clear runbooks and technical documentation, following the team’s documentation standards, so that knowledge of the platform is shared across the team.
Requirements
- A bachelor’s degree in Computer Science, Software Engineering or a related field, or equivalent practical experience.
- At least 7 years in data engineering, software engineering or platform engineering roles, including at least 3 years building and running distributed data processing (Spark or equivalent) in production.
- A track record of taking responsibility for a production data platform, or major parts of one, including its reliability, performance and day-to-day operation.
- Experience working with datasets in the hundreds of millions to billions of records, and designing processing that stays efficient as volumes grow.
- Experience running data workloads within real budget constraints: making deliberate trade-offs between compute, storage and performance, and being able to show the effect of those decisions.
- Experience managing the cloud infrastructure that data workloads run on, not only using managed services.
- Experience in small teams where engineers cover a broad scope and work with limited supervision.
- Experience working with remote teams across multiple countries and time zones.
- Strong Python, written to production standards, with testing and modular design.
- Advanced SQL, including writing and optimising queries against analytical databases.
- Spark (PySpark), including performance tuning for partitioning, shuffles and data skew.
- A solid understanding of data lake design, including columnar formats such as Parquet, partitioning strategy, and when to process in the lake rather than the warehouse.
- Hands-on AWS experience with data services and the infrastructure around them, such as S3, Glue or EMR, IAM and EC2.
- Infrastructure as code, using Terraform, CloudFormation, CDK or similar.
- Linux administration and Docker, to the level needed to deploy and maintain self-hosted services.
- dbt in production, including incremental models and testing.
- Workflow orchestration with Airflow, Dagster, Prefect or similar.
- Git, CI/CD and code review practice.
Nice to have
- Experience in the non-profit, international development or social sector.
- Apache Iceberg, or another open table format such as Delta Lake or Hudi.
- ClickHouse, or a comparable columnar warehouse such as Snowflake, BigQuery or Redshift.
- Terraform specifically.
- Apache Airflow specifically.
- AWS Glue and Lake Formation.
- Data modelling for analytics, including handling reference data that changes over time.
- Managed Spark platforms such as Databricks.
- Streaming technologies such as Kafka, Flink, Kinesis or Spark Structured Streaming.
- Event, telemetry or IoT data, particularly from devices that sync intermittently.
- Ingestion tools such as Airbyte.
- Administering BI tools such as Apache Superset.
- Software development beyond data pipelines, such as building APIs or system integrations.
Benefits
- Salary is competitive and commensurate with experience.
Additional details
- The role must be based in Africa.
- This is a full-time remote position; the role can be based anywhere in Africa with reliable internet access and alignment with core working hours across time zones.
- The role may involve occasional international travel for in-person team meetings, field support or training, as needed.
- Candidates should be comfortable working independently in a virtual environment and coordinating with colleagues across global teams.
- Applications are reviewed on a rolling basis.
- Imagine Worldwide is an equal opportunity employer.
