Big Data Lead
DemandMatrix
IndiaremotePosted 1 month ago
D
Skill Required
Big-Data-EngineeringData-Engineering-LeadBig-Data-EngineerData-Platform-EngineeringBig-Data-DevelopmentLead-Big-Data-EngineerData-Tech-LeadLead-Data-EngineerMachine LearningData StructuresElasticsearchEngineeringKubernetesHadoopMongoDBPythonDockerSparkCI/CDCloudDesign PatternsRustLinuxAWSGCPandGoCICDAIFulltime
Key highlights
- 7+ years of experience in Software Development
- Focus on big data and large data pipelines
- Remote Work / Work From Home
- Expertise in Python, Spark, and Hadoop/MapReduce
Role overview
DemandMatrix aims to disrupt the $100 billion sales and marketing intelligence industry using domain knowledge, machine learning, and AI, serving Fortune 100 companies like Microsoft, Google, Adobe, Amazon, and IBM. The company is looking for a hands-on SME in leveraging big data tech to solve the most complex data issues.
Responsibilities
- Spend almost half of time with hands-on coding
- Large scale text data processing, event driven data pipelines, in-memory computations, optimization considering CPU core to network IO to disk IO
- Using cloud native services in AWS and GCP
Requirements
- Solid grounding in computer engineering, Unix, data structures and algorithms
- Designed and built multiple big data modules and data pipelines to process large volume
- Genuinely excited about technology and worked on projects from scratch
- 7+ years of hands-on experience in Software Development with a focus on big data and large data pipelines
- Minimum 3 years of experience to build services and pipelines using Python
- Expertise with a variety of data processing systems, including streaming, event, and batch (Spark, Hadoop/MapReduce)
- Understanding of at least one NoSQL stores like MongoDB, Elasticsearch, HBase
- Understanding of how data models, sharding and data location strategies for distributed data stores in large scale high-throughput and high-availability environments and their effect in non-structured text data processing
- Experience with running scalable & high available systems with AWS or GCP
Nice to have
- Experience with Docker / Kubernetes
- Exposure with CI/CD
- Knowledge of Crawling/Scraping
Benefits
- Entire Work From Home
- Birthday Leave
- Remote Work
Additional details
- Originally posted on Himalayas