Web scraping speed benchmark

How fast is scraping a page, really? Curl is the floor — it returns raw HTML with the JavaScript never run, so on a React app you get an empty shell. Fastcrawl does the whole job (fetch, render if needed, convert to LLM-ready markdown, cache) and on this run lands withintens of milliseconds of curl on static pages — while returning real content on the pages curl can't read.

Median scrape times (seconds)

Reading the numbers

curl is the fastest possible fetch because it does nothing: one request, raw bytes, no browser, no parsing, no markdown. It is also the only tool here that cannotread a client-rendered page. Fastcrawl is a hosted round trip — edge fetch, browser render only when the page actually needs it, clean markdown back — with zero infrastructure on your side.

The interesting row is news.ycombinator.com: curl takes 1.75s(HN is a slow origin, and curl has no cache), while Fastcrawl returns the same page as markdown in 0.27s. Repeat checks of the same URL come back in ~60ms from the cache — that is the "scrape 200,000 pages a month for your own newsroom" path, and it is how we run our own content pipeline.

Methodology

Five URLs covering static, dynamic, JavaScript-heavy, single-page-app and PDF cases. Three runs per tool per URL, median reported. Fastcrawl figures use maxAge: 0 to force a fresh render (never a cached result); curl uses time_total viacurl -w with the same Googlebot user-agent. Percentages are rounded to 2 decimals. Captured 2026-09-13 against the production API.

Note on local scrapers: we previously published a crawl4ai column. We removed it because a local Chromium run isn't reproducible on the host we publish from, and we would rather show two sets of numbers we measured ourselves than three where one is second-hand. The honest framing: a local scraper has no network round trip, but you own the browser, the proxies and the CPU — that trade is the whole point of a hosted API.