Documentation

One key. REST and MCP.

Base URL https://fastcrawl.net (or your custom domain). Auth headerAuthorization: Bearer YOUR_KEY. Failed calls are never charged.

1. Get a key

Sign up at /app. The dashboard shows your key once.

2. Scrape

curl -X POST https://fastcrawl.net/api/v1/scrape/ \
  -H "Authorization: Bearer ***" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com","formats":["markdown","json"],"timeout":30,"fetchMode":"auto"}'

Returns markdown, json (structured data: title, headings, paragraphs, links, images, word count), metadata, duration_ms. Add "html" to formats for raw HTML, "links" for outbound links, "changeTracking" to get a diff vs your previous scrape of the same URL (metadata.change_information: first / unchanged / added / removed / different + line diff). timeout (5-30s, default 30) fails fast on blocking pages. PDF URLs are detected automatically and parsed via document extraction — no separate call needed.

Failures are business results: 422 with an actionable error_code (target_dns_error, target_not_found, target_blocked, target_rate_limited, target_timeout, scrape_empty_content, …) and a retryable flag — retry only when it's true. Failed requests are never charged.

fetchMode: auto (default) tries HTTP first (~100ms) and escalates to a browser render only when the page needs JavaScript; http never opens a browser (~100ms, for static/SSR pages). There is no forced-render mode — auto already escalates, and forcing a render spends seconds and browser-hours on pages plain HTTP fetches in milliseconds.

3. Search + content

Search, and optionally the first results' pages, in one call — the agent loop of "search → read top N" without a second round trip. include_content (1-5) fetches that many results in order through the same pipeline as /scrape, so a bundled page is served from the same cache a direct scrape of it would use.

curl -X POST https://fastcrawl.net/api/v1/search/ \
  -H "Authorization: Bearer ***" \
  -H "Content-Type: application/json" \
  -d '{"query":"model context protocol server registry","limit":5,"include_content":2}'

Response (real call, markdown trimmed):

{
  "success": true,
  "query": "model context protocol server registry",
  "results": [
    { "title": "MCP Server Registry | Discover & Govern AI Agent Tools | Kong I…",
      "url": "https://konghq.com/products/mcp-registry",
      "snippet": "The Model Context Protocol (MCP) has simplified how agents connect to tools. …",
      "content": "# Kong AI Registry: Govern What AI Agents Can Discover  | Kong Inc.… (20,893 chars)",
      "content_error": null },
    { "title": "Awesome MCP Servers",
      "url": "https://mcpservers.org/",
      "snippet": "A lightweight MCP (Model Context Protocol) server for Blender. …",
      "content": "# Awesome MCP Servers … (15,537 chars)",
      "content_error": null },
    { "title": "GitHub - mapbox/mcp-server: Mapbox Model Context Protocol (MCP) …",
      "url": "https://github.com/mapbox/mcp-server",
      "snippet": "…",
      "content": null, "content_error": null }
  ]
}

Fields: content is the page as markdown for the first include_content results (rows past N keep it null), content_error is null on success. Results 1 credit; each content page 1 credit, cache hits free — a repeat of a page we already scraped inside the 48h window costs nothing. A page that fails (404, empty body, anti-bot) never fails the request: that row gets content: null and a short content_error (e.g. target_not_found, scrape_empty_content) while the call still returns 200. include_content above 5 is a 400, and the whole request has a 60s budget — rows that miss it come back with content_error: "budget exceeded". REST only: the MCP search tool stays one credit per call.

4. Screenshot & PDF

Both return a data: URI you can write straight to a file. Set viewport for mobile widths or tall full-page captures.

# PNG screenshot
curl -X POST https://fastcrawl.net/api/v1/screenshot/ \
  -H "Authorization: Bearer YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com","viewport":{"width":1280,"height":800}}'

# Rendered PDF
curl -X POST https://fastcrawl.net/api/v1/pdf/ \
  -H "Authorization: Bearer YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com"}'

Each returns { "success": true, "data": "data:image/png;base64,…" } (screenshot) or data:application/pdf;base64,… (pdf). 1 credit per call.

5. HTML to PNG image

Markup in, hosted PNG out — render a chart, invoice, social card or OG image from HTML+CSS with no browser on your side. This route takes html only; for a URL capture use /screenshot.

curl -X POST https://fastcrawl.net/api/v1/image/ \
  -H "Authorization: Bearer ***" \
  -H "Content-Type: application/json" \
  -d '{"html":"<html><body><h1>Hello from fastcrawl</h1></body></html>","css":"body{font-family:sans-serif;padding:40px} h1{color:#2563eb}","viewport_width":800,"viewport_height":600}'

Returns { "success": true, "url": "https://…/v1/image/…" } and that URL serves content-type: image/png. Fields: html (required, 1MB max), css, viewport_width (default 800), viewport_height (default 600), google_fonts. PNG only — no PDF/JPEG, and no width/height in the response because the renderer does not report them. 1 credit per image; an identical repeat within 30 minutes is a cache hit, returns "cached": true and is free. Render capacity saturation answers 429 — back off and retry, failed calls are never charged.

6. MCP

{
  "mcpServers": {
    "fastcrawl": {
      "url": "https://mcp.fastcrawl.net/mcp",
      "headers": { "Authorization": "Bearer YOUR_KEY" }
    }
  }
}

14 tools: scrape, crawl, map, search, extract, screenshot, pdf, image, parse, verify_email, monitor_create, monitor_list, monitor_run, monitor_delete.

Autonomous agents can discover this server via the A2A agent card at /.well-known/agent-card.json (no auth required).

7. Verify emails

Is this address worth sending to? Syntax and gibberish, disposable and webmail domains, MX records over DNS-over-HTTPS, then a real SMTP handshake at the mailbox (RCPT TO — no message is ever sent). One credit per address, up to 10 addresses per call, and the same key you scrape with. Failed calls are never charged.

# one address
curl -X POST https://fastcrawl.net/api/v1/email/verify \
  -H "Authorization: Bearer ***" \
  -H "Content-Type: application/json" \
  -d '{"email":"[email protected]"}'

# or up to 10 in one call
  -d '{"emails":["[email protected]","[email protected]"]}'

Every result carries a status plus the evidence behind it — score, mx_records, smtp_check, accept_all, disposable, webmail — so your own threshold can decide, rather than ours. valid send it · invalid remove it · risky a catch-all domain, so the mailbox may or may not exist · unknown unproven, because the large providers rate-limit the probe IP. Retry unknown, never treat it as invalid — that one rule is what keeps a list clean. A batch row is not always a verdict: skipped means the 80-second budget ran out before that address was probed and error means the probe threw — retry both, they say nothing about the address.

Checking a single address by hand? The free email checker runs the syntax, typo, disposable and MX layers in your browser with no key. The mailbox-level handshake is what the API adds.

8. Crawl callbacks & webhooks

Stop polling. Pass callback_url to POST /api/v1/crawl (or the MCP crawl tool) and we POST a signed crawl.completed event the moment the job reaches succeeded or failed. Monitors do the same for their webhook_url whenever content changes. Delivery is free — billing is unchanged (one credit per crawled page, nothing for the callback, no extra usage rows).

Every delivery is signed with a secret that belongs to your account — read it under Dashboard → Webhooks (or GET /api/v1/webhooks/secret; it is created the first time you read it, and POST /api/v1/webhooks/secret/rotate replaces it). No need to email anyone.

curl -X POST https://fastcrawl.net/api/v1/crawl/ \
  -H "Authorization: Bearer ***" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com","max_pages":10,"callback_url":"https://your-app.example/hooks/fastcrawl"}'

# POSTed to your callback_url when the job finishes (pages = pages crawled):
{"event":"crawl.completed","job_id":"…","status":"succeeded","status_url":"/api/v1/crawl/…/","pages":10}

Retries: up to 4 deliveries — immediately, then after ~1s, ~4s and ~15s — then we log the failure and stop. A failed job posts the same event with status: failed and no pages field.

Where callbacks may point: http or https only, no user:pass@ credentials, and the host must not resolve to a loopback (127.0.0.0/8, ::1), link-local (169.254.0.0/16, fe80::/10), private (10/8, 172.16/12, 192.168/16, fc00::/7), CGNAT (100.64/10) or multicast address. That is checked when you submit — a rejected URL is a 400 before any job or monitor exists — and re-checked at dial time, so a host that re-resolves to a private address never gets a connection.

# Every delivery is signed (Stripe-style, replay-protected):
#   X-Fastcrawl-Signature: t=<unix_ts>,v1=<hex hmac-sha256(secret, "<ts>.<body>")>

# verify.py — paste into your handler (your signing secret: Dashboard → Webhooks,
# or GET /api/v1/webhooks/secret)
import hmac, hashlib, time

def verify(header: str, body: bytes, secret: str, tolerance: int = 300) -> bool:
    parts = dict(p.split("=", 1) for p in header.split(","))
    ts = int(parts["t"])
    if abs(time.time() - ts) > tolerance:      # reject replays older than 5 min
        return False
    want = hmac.new(secret.encode(), f"{ts}.".encode() + body, hashlib.sha256).hexdigest()
    return hmac.compare_digest(want, parts["v1"])

# IMPORTANT: body must be the RAW request bytes — never re-serialize the parsed JSON
# ok = verify(request.headers["X-Fastcrawl-Signature"], request.get_data(), SECRET)

Endpoints

Limits

Past 5,000 credits on Go: $1.00 per 1,000 extra, metered — requests are never blocked. Disable overage anytime in the dashboard (Overview) to hard-cap at 5,000. Extract is 1 credit/page. Failed requests are free.

CLI · Free tools · llms.txt · openapi.json · agent skill