boolean
Endpoints
POST /crawl
BFS crawl from a seed URL (up to 50 pages).
POST
/
crawl
POST /crawl
Starts a BFS crawl from a seed URL and returns202 with a job_id.
The request accepts the full advanced crawl contract; PDF parsing and OCR
stay within the same max_pages × 1 budget.
Authentication
X-API-Key header required.
Parameters
string
required
Seed URL. SSRF guard applies to the seed and every discovered link.
integer
required
Maximum pages to crawl. Public API range:
1–50. Recommended value: 10.integer
Optional crawl depth cap. Default:
null. Must be >= 0 when provided.boolean
Follow subdomains of the seed host. Default:
false.string
Optional RE2 regex to filter discovered absolute URLs. Default:
"".
Maximum length: 512 characters.Extract only the main content. Default:
true.boolean
Keep links in the final Markdown. Default:
true.boolean
Keep images in the final Markdown. Default:
true.boolean
Keep frame references in the final Markdown. Default:
false.boolean
Deterministically remove or replace long
data: URLs. Default: false.boolean
Enable PDF parsing. Default:
true.boolean
OCR fallback for PDFs without useful text. Default:
false.
Requires parse_pdf=true.integer
Post-load wait in milliseconds. Default:
0. Range: 0–30000.
Must remain below timeout_ms.boolean
Disable animations and transitions before the configured wait. Default:
false.integer
Total page timeout in milliseconds. Default:
60000. Range: 1000–300000.string[]
Normalized tags persisted on billed usage records. Default:
[].Cost
1 credit per successful page. PDF parsing and OCR do not add a separate tariff. The ceiling remainsmax_pages × 1.
Partial results and reasons
PollGET /jobs/{job_id} to inspect progress. Crawls can end early with a
partial result set. stopped_reason documents the terminal reason:
frontier_empty, max_pages, timeout, insufficient_credits, or
invalid_seed.
Individual result items can also expose a stable reason such as
pdf_parse_disabled, pdf_ocr_timeout, or pdf_ocr_resource_limit.
Example
curl -X POST https://api.messora.dev/crawl \
-H "Content-Type: application/json" \
-H "X-API-Key: YOUR_API_KEY" \
-d '{
"url": "https://docs.messora.dev/en/introduction",
"max_pages": 10,
"url_regex": "^https://docs\\.messora\\.dev/en(/.*)?$",
"parse_pdf": true,
"pdf_ocr": false,
"wait_for_ms": 0,
"tags": ["crawl", "api-reference"]
}'
import requests
response = requests.post(
"https://api.messora.dev/crawl",
headers={
"Content-Type": "application/json",
"X-API-Key": "YOUR_API_KEY",
},
json={
"url": "https://docs.messora.dev/en/introduction",
"max_pages": 10,
"url_regex": "^https://docs\\.messora\\.dev/en(/.*)?$",
"parse_pdf": True,
"pdf_ocr": False,
"wait_for_ms": 0,
"tags": ["crawl", "api-reference"],
},
)
job_id = response.json()["job_id"]
print(job_id)
const res = await fetch("https://api.messora.dev/crawl", {
method: "POST",
headers: {
"Content-Type": "application/json",
"X-API-Key": "YOUR_API_KEY",
},
body: JSON.stringify({
url: "https://docs.messora.dev/en/introduction",
max_pages: 10,
url_regex: "^https://docs\\.messora\\.dev/en(/.*)?$",
parse_pdf: true,
pdf_ocr: false,
wait_for_ms: 0,
tags: ["crawl", "api-reference"],
}),
});
const { job_id } = await res.json();
console.log(job_id);
Responses
202 returns job_id. Polling via GET /jobs/{job_id} returns partial or
final results. Successful pages contribute credits_used: 1 each.
Errors
401: missingX-API-Key403: invalid or revoked API key422:max_pagesmissing/out of range, internal seed, or invalid params429: rate limit exceeded