About the Role You will design, build, and operate large-scale web data acquisition systems used for research and model development. You will build and maintain distributed crawlers, handle anti-bot systems and dynamic JavaScript-heavy sites, and develop pipelines for cleaning, deduplication, filtering, and normalization. You will construct and maintain research datasets, monitor crawl performance and data quality, iterate quickly, and optimize infrastructure for cost, latency, and reliability. You will collaborate with research teams to align data collection with modeling needs and own end-to-end data acquisition pipelines.
Requirements Strong programming experience in Go Rust Python Java or C++ Experience building web crawlers or large-scale data pipelines Solid understanding of HTTP networking and browser behavior Familiarity with distributed systems and parallel processing Experience working with large datasets (TB–PB scale preferred) Ability to debug unstable or adversarial environments Experience with headless browsers (e.g., Chrome DevTools Protocol Playwright Puppeteer) Familiarity with proxy systems IP rotation and large-scale request orchestration Experience running workloads on cloud or bare-metal infrastructure Experience with NLP pipelines dataset curation or LLM pretraining data (preferred)
Responsibilities Build and maintain large-scale web crawlers across diverse domains Design high-throughput fault-tolerant systems for data collection Handle anti-bot systems rate limits and dynamic JS-heavy sites Develop pipelines for cleaning deduplication filtering and normalisation Construct and maintain datasets for research and model training Monitor crawl performance coverage and data quality and iterate quickly Collaborate with research teams to align data collection with modeling needs Optimize infrastructure for cost latency and reliability Own end-to-end data acquisition pipelines
Benefits Equity package Fully remote work Benefits package Funding Investors Junior SRE Endpoint focus TransFICC · 2 days ago Software Development Engineer in Test TransFICC · 2 days ago Senior Applications Engineer Corporate Systems Robinhood Markets, Inc. · 4 days ago Senior Big Data Engineer Amberdata · 4 days ago Lead Engineer Trading Platform Engineering CoinDesk · 4 days ago Funding Investors Junior SRE Endpoint focus TransFICC · 2 days ago Software Development Engineer in Test TransFICC · 2 days ago Senior Applications Engineer Corporate Systems Robinhood Markets, Inc. · 4 days ago Senior Big Data Engineer Amberdata · 4 days ago Lead Engineer Trading Platform Engineering CoinDesk · 4 days ago