Skip to main content

Crawl

POST /crawl starts a BFS crawl from a seed URL. It discovers internal links recursively and extracts Markdown content from each page found.

Endpoint

Authentication

X-API-Key header required. See Authentication.

Parameters

string
required
Seed URL to start the crawl. SSRF guard applied to the seed and every discovered link.
integer
required
Maximum pages to crawl (1–50). Recommended value: 10. No default — omitting it returns 422.

Cost

1 credit per successful page.
From 5 estimated credits, the Playground asks for user confirmation before starting. Failed pages (block, timeout) do not consume credits.

Asynchronous flow

  1. Send POST /crawl → receive 202 with job_id
  2. Poll via GET /jobs/{job_id} every 2s
  3. When status: "SUCCESS", results are in the results field
  4. stopped_reason may be max_pages if the limit was reached

Example

Successful response (via polling)

Limits

Errors