TaskUs is a global provider of outsourced digital services and next-generation customer experience, serving fast-growing technology companies across sectors like social media, e-commerce, gaming, and FinTech. With a People First culture, TaskUs prioritizes employee well-being, offering competitive salaries, comprehensive benefits, and opportunities for professional growth. As a Senior Data Engineer, you will be the engine room of the data strategy, building resilient, self-healing systems that transform raw data into high-fidelity data products, turning architectural blueprints into production-grade code to ensure clean and reliable data.
Responsibilities
- Build and maintain high-throughput ETL/ELT pipelines that ingest data from various sources into the Lakehouse.
- Drive the adoption of dbt for transformation and PySpark for heavy lifting.
- Write modular, reusable code that follows strict CI/CD practices.
- Implement 'Data SLAs' by building monitoring and alerting systems that notify the team of data drift or pipeline failures before the business notices.
- Work closely with the BI team and Data Scientists to prepare 'feature-ready' datasets.
- Deep-dive into SQL and Spark query plans to optimize slow-running jobs, reducing cost and latency.
- Implement a workflow where engineers can develop and test complex SQL transformations locally using DuckDB, drastically reducing 'waiting-for-cluster' time and lowering development costs.
- Build specialized pipelines for specific client reporting or data-quality checks that run on single-node containers (e.g., Lambda or small ECS tasks) using DuckDB, avoiding the minimum billing cycles of larger warehouses.
- Explore ways to use DuckDB as an embedded engine for internal tools or 'edge' processing within the BPO’s local site offices where bandwidth to the cloud might be a constraint.
Requirements
- 5+ years of experience in Data Engineering, including on-call experience and building systems that don’t break at 3 AM.
- Expert-level proficiency in Python, SQL, and PySpark.
- Hands-on experience with Databricks (Delta Lake) or Snowflake, including understanding the nuances of the Medallion Architecture (Bronze/Silver/Gold).
- Advanced experience with dbt (Data Build Tool), treating data models like software, including version control, testing, and documentation.
- Experience with Apache Airflow or Prefect, specifically building complex, idempotent DAGs.
- Proven experience using DuckDB for high-speed local development, unit testing data transformations, or as a query engine for 'small-to-medium' datasets (up to 100GB) without the overhead of a distributed cluster.
- Ability to identify when to use heavyweight compute (PySpark/Databricks) versus lightweight compute (DuckDB/Python) to minimize cloud costs and reduce job latency.
- Experience using DuckDB to directly query data stored in S3/Azure Blob (via Parquet or Iceberg files) for rapid ad-hoc analysis or local dashboarding.
- Deep understanding of Kubernetes (K8s), including Pods, Deployments, Services, ConfigMaps, and Secrets management.
- Comfort with Terraform or CloudFormation to manage data infrastructure.
Nice to have
- Familiarity with Kafka, Kinesis, or Spark Streaming.
Benefits
- Competitive industry salaries.
- Comprehensive benefits packages.
- Departments dedicated to Total Rewards, Wellness, HR, and Diversity.
- Inclusive and equitable work environment.
- Encouragement of internal mobility and professional growth at all career stages.
- Positive impact on the community.
Additional details
- Work mode: Remote.
- TaskUs will neither solicit money from you during your application process nor require any form of payment in order to proceed with your application. Kindly ensure that you are always in communication with only authorized recruiters of TaskUs.
- TaskUs believes that innovation and higher performance are brought by people from all walks of life. We welcome applicants of different backgrounds, demographics, and circumstances. Inclusive and equitable practices are our responsibility as a business. TaskUs is committed to providing equal access to opportunities. If you need reasonable accommodations in any part of the hiring process, please let us know.
- Originally posted on Himalayas.
Zobhira
