Senior Data Engineer - ETL & Pipelines
2am.tech
WorldwideremotePosted 16 days ago
Skill Required
Data-EngineeringETL-DevelopmentPipeline-EngineeringData-Infrastructure-EngineeringBackend-EngineeringSenior-Data-Pipeline-EngineerSenior-ETL-EngineerSenior-Data-EngineeringSenior-Data-Integration-EngineerETL-Data-EngineerData EngineerETLGitHub ActionsObservabilityEngineeringTypeScriptKubernetesanalyticsNode.jsRabbitMQbuildingMongoDBfintechDockerdesignGit.NETRedisDesign PatternsjQueryC#dbtSQLAWSandFulltime
Key highlights
- High-volume B2C environment with 200,000+ active users and 30,000+ creators
- 5-10 years of relevant engineering experience required
- Data integrity critical for financial transactions
- Experience with payment or transaction data and financial reconciliation
- Collaboration with payments and platform teams on financial data
Role overview
Our client operates a high-volume B2C creator platform in the entertainment industry, serving more than 200,000 active users and 30,000 creators, with payment events flowing through the platform every second. Data is one of the company's most valuable competitive advantages. In this role, you will work closely with an experienced data analyst to transform a continuous flow of events into clean, reliable, and query-ready data that supports product, payments, and growth decisions. We are looking for a Data Engineer who will take end-to-end ownership of the company's ETL pipelines—the systems responsible for moving data from the transactional platform, including MongoDB, payment streams, and behavioral events, into the analytics environment where it can be turned into actionable insights. This is not simply a dashboard integration role. You will design and maintain pipelines that remain accurate, reliable, and timely while millions of events are processed and financial transactions occur every second. Data integrity is critical. A missing payment event or duplicated transaction is not merely a technical issue—it can have a direct financial impact.
Responsibilities
- Own the pipelines: Design, build, and maintain ETL and streaming pipelines that ingest high-volume event and payment data and deliver it to the analytics warehouse reliably and accurately
- Model the data: Partner closely with the Lead Data Analyst to create warehouse schemas that answer essential business questions related to creator earnings, payment throughput, retention, lifetime value, and cohort behavior
- Engineer for failure: Build idempotent, replayable, and observable jobs that can withstand traffic spikes, retries, and partial failures without compromising downstream data
- Process high-volume events: Move data through queue infrastructure including BullMQ and Redis, AWS SQS, and RabbitMQ, while instrumenting pipelines through OpenTelemetry
- Protect data integrity: Implement reconciliation checks, deduplication, backfills, and reliable handling of late or out-of-order events
- Protect financial transactions: Collaborate with payments and platform teams to ensure that every transaction is captured once, accurately, and on time
Requirements
- Approximately 5–10 years of relevant engineering experience
- Strong experience with batch and/or streaming data pipelines
- Deep knowledge of event-driven architectures and message queues
- A strong understanding of idempotency, at-least-once versus exactly-once processing, retries, and out-of-order event delivery
- Professional experience with TypeScript and Node.js
- Solid SQL and data-modeling capabilities
- The ability to collaborate with analysts and design schemas that make business queries efficient and intuitive
- A strong focus on data integrity, particularly when working with payments and financial transactions
- Experience implementing reconciliation, auditing, monitoring, and data-quality controls
- Previous experience in a high-volume B2C environment such as gaming, gambling, fintech, payments, or another real-money platform
Nice to have
- Experience working with payment or transaction data and financial reconciliation
- Experience with MongoDB change streams or change data capture
- Familiarity with data warehouse tooling and dbt-style modeling
- Experience working with Kubernetes and infrastructure as code
- Experience with .NET or C#
Additional details
- Languages: Node.js and TypeScript as the primary technologies, with some .NET and C#
- Data stores: MongoDB as the primary transactional database and Redis
- Queues and streaming: BullMQ, AWS SQS, and RabbitMQ
- Analytics and warehouse: PostHog, its data warehouse layer, and SQL data modeling
- Platform: AWS, including S3 and SNS, Docker, Kubernetes, and GitHub Actions
- Observability: OpenTelemetry tracing across every pipeline
- The successful candidate will be employed or engaged directly by the client, while 2am.tech will coordinate the recruitment and selection process
- This is a direct-hire opportunity