Web Scraping Lead - Global Team (India)
YipitData
IndiaremotePosted 6 days ago
Skill Required
Engineering-Data-EngineeringWeb-ScrapingEngineering-ManagementBrowser-AutomationTeam-LeadershipWeb-Scraping-EngineerWeb-Data-Scraping-EngineerWeb-Data-Extraction-EngineerSenior-Python-Data-Scraping-EngineerWeb-Scraping-DeveloperData-Scraping-EngineerWeb-Scraping-SpecialistPython-Data-Scraping-EngineerWeb-Data-Extraction-Specialistsoftware engineeringtroubleshootingEngineeringPlaywrightautomationObservabilityanalyticsSeleniumbuildingdesignRustHTMLAPIsandCryptographyFulltime
Key highlights
- Annual base salary: $73K‑$90K.
- 7+ years of software engineering experience required.
- Unlimited vacation and flexible work hours.
- Generous 401K match and health benefits with no monthly premium.
- Remote work allowed within the United States (or based in India for IST hours).
- No visa sponsorship will be considered.
Role overview
We are looking for an experienced Web Scraping Manager to lead our web scraping engineering team and own the design, scalability, reliability, and performance of our scraping infrastructure. You’ll lead engineers, drive technical architecture and best practices, solve complex scraping challenges, and collaborate with Engineering, Data, Analytics, and Product teams to deliver reliable, high-quality data.
Responsibilities
- Lead, mentor, and grow a team of web scraping engineers.
- Own the architecture, roadmap, scalability, and reliability of scraping systems.
- Guide the development and maintenance of scrapers using Playwright, Puppeteer, Selenium, etc.
- Solve complex challenges involving dynamic websites, browser automation, fingerprinting, sessions, cookies, headers, and proxies.
- Establish monitoring, logging, alerting, and data-quality practices.
- Drive performance, reliability, and infrastructure improvements.
- Partner with cross-functional teams to prioritize scraping requirements and deliver high-quality data.
- Participate in hiring, technical reviews, and engineering planning.
Requirements
- 7+ years of software engineering experience, with strong web scraping experience.
- Strong hands-on experience with Playwright, Puppeteer, or Selenium.
- Deep understanding of HTTP, REST APIs, HTML/DOM, browser rendering, and TLS/SSL.
- Strong experience with fingerprinting, anti-bot challenges, cookies, headers, sessions, and proxy rotation.
- Experience building scalable, reliable scraping infrastructure.
- Strong knowledge of monitoring, logging, metrics, and production troubleshooting.
- Excellent communication and stakeholder-management skills.
- Strong ownership, leadership, and mentoring abilities.
Nice to have
- Previous engineering management/team leadership experience preferred.
Benefits
- Competitive salary and comprehensive benefit offerings.
- Opportunities to build a career as a manager or individual contributor through a career ladder that promotes based on impact.
- Regular career conversations with your manager to discuss short and long-term career goals.
- Flexible work hours and a hybrid work environment.
- Unlimited vacation.
- Generous 401K match.
- Parental leave.
- Team events.
- Wellness budget.
- Learning reimbursement.
- Monthly work‑from‑home budget for remote workers.
- Choice of health plans starting at no monthly out‑of‑pocket premium.
- Dental, vision, and commuter benefits.
Additional details
- About Us: YipitData is the leading market research and analytics firm for the disruptive economy and recently raised up to $475M from The Carlyle Group at a valuation over $1B.
- We analyze billions of alternative data points every day to provide accurate, detailed insights on ridesharing, e‑commerce marketplaces, payments and more. Our on‑demand insights team uses proprietary technology to identify, license, clean and analyze the data many of the world’s largest investment funds and corporations depend on.
- For three years and counting, we have been recognized as one of Inc’s Best Workplaces.
- We are a fast‑growing technology company backed by The Carlyle Group and Norwest Venture Partners.
- Our offices are located in NYC, Austin, Miami, Denver, Mountain View, Seattle, Hong Kong, Shanghai, Beijing, Guangzhou, and Singapore.
- We cultivate a people‑centric culture focused on mastery, ownership, and transparency.
- This is a fully‑remote opportunity based in India. Standard work hours are from 11am to 8pm IST, but there is flexibility here.
- The role may be performed fully remotely within the United States. If the remote work is performed outside of our US office hubs, income may be subject to New York State tax withholding.
- The annual base salary for this position is anticipated to be $73K‑$90K. The final offer may be determined by a number of factors, including but not limited to the applicant's experience, knowledge, skills, and abilities, as well as internal team benchmarks.
- We are not able to consider candidates who currently or in the future will require visa sponsorship.
- We are committed to equal employment opportunity regardless of race, color, ancestry, religion, sex, national origin, sexual orientation, age, marital status, disability, gender, gender identity or expression, or veteran status.
- Job Applicant Privacy Notice.
- Originally posted on Himalayas.