> ## Documentation Index
> Fetch the complete documentation index at: https://docs.messora.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Crawl

> Crawl entire sites via BFS from a seed URL, up to 50 pages per job.

# Crawl

`POST /crawl` starts a BFS crawl from a seed URL. It discovers internal links
recursively and extracts Markdown content from each page found.

## Endpoint

```
POST /crawl
```

## Authentication

`X-API-Key` header required. See [Authentication](/en/authentication).

## Parameters

<ParamField body="url" type="string" required>
  Seed URL to start the crawl. SSRF guard applied to the seed and every discovered link.
</ParamField>

<ParamField body="max_pages" type="integer" required>
  Maximum pages to crawl (1–50). Recommended value: 10.
  No default — omitting it returns `422`.
</ParamField>

## Cost

**1 credit per successful page.**

<Note>
  From 5 estimated credits, the Playground asks for user confirmation before starting.
  Failed pages (block, timeout) do not consume credits.
</Note>

## Asynchronous flow

1. Send `POST /crawl` → receive `202` with `job_id`
2. Poll via `GET /jobs/{job_id}` every 2s
3. When `status: "SUCCESS"`, results are in the `results` field
4. `stopped_reason` may be `max_pages` if the limit was reached

## Example

<CodeGroup>
  ```bash cURL theme={null}
  curl -X POST https://api.messora.dev/crawl \
    -H "Content-Type: application/json" \
    -H "X-API-Key: YOUR_API_KEY" \
    -d '{
      "url": "https://example.com",
      "max_pages": 10
    }'
  ```

  ```python Python theme={null}
  import requests
  import time

  # 1. Start the crawl
  response = requests.post(
      "https://api.messora.dev/crawl",
      headers={
          "Content-Type": "application/json",
          "X-API-Key": "YOUR_API_KEY",
      },
      json={
          "url": "https://example.com",
          "max_pages": 10,
      },
  )

  job_id = response.json()["job_id"]
  print(f"Job created: {job_id}")

  # 2. Poll until complete
  while True:
      status = requests.get(
          f"https://api.messora.dev/jobs/{job_id}",
          headers={"X-API-Key": "YOUR_API_KEY"},
      ).json()

      if status["status"] in ("SUCCESS", "FAILURE"):
          break

      # Partial progress
      if "meta" in status:
          print(f"Progress: {status['meta']}")

      time.sleep(2)

  # 3. Results
  for page in status.get("results", []):
      print(f"{page['url']}: {page['scrape_status']}")
  ```

  ```javascript JavaScript theme={null}
  // 1. Start the crawl
  const res = await fetch("https://api.messora.dev/crawl", {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      "X-API-Key": "YOUR_API_KEY",
    },
    body: JSON.stringify({
      url: "https://example.com",
      max_pages: 10,
    }),
  });

  const { job_id } = await res.json();
  console.log(`Job created: ${job_id}`);

  // 2. Poll until complete
  let status;
  do {
    await new Promise((r) => setTimeout(r, 2000));
    const poll = await fetch(
      `https://api.messora.dev/jobs/${job_id}`,
      { headers: { "X-API-Key": "YOUR_API_KEY" } }
    );
    status = await poll.json();
  } while (!["SUCCESS", "FAILURE"].includes(status.status));

  // 3. Results
  for (const page of status.results || []) {
    console.log(`${page.url}: ${page.scrape_status}`);
  }
  ```
</CodeGroup>

## Successful response (via polling)

```json theme={null}
{
  "status": "SUCCESS",
  "job_id": "550e8400-e29b-41d4-a716-446655440003",
  "results": [
    {
      "url": "https://example.com",
      "scrape_status": "success",
      "markdown": "# Example Domain\n\nThis domain is for use in illustrative examples...",
      "credits_used": 1
    },
    {
      "url": "https://example.com/about",
      "scrape_status": "success",
      "markdown": "# About\n\nMore information...",
      "credits_used": 1
    }
  ],
  "credits_used": 2,
  "stopped_reason": null
}
```

## Limits

| Context                     | Max `max_pages` |
| --------------------------- | --------------- |
| Public API (`X-API-Key`)    | 50              |
| Playground (cookie session) | 10              |

## Errors

| HTTP Status | Description                                                                    |
| ----------- | ------------------------------------------------------------------------------ |
| `401`       | Missing `X-API-Key`                                                            |
| `403`       | Invalid or revoked API key                                                     |
| `422`       | Missing `max_pages`, outside 1–50, internal seed (SSRF), or invalid parameters |
| `429`       | Rate limit exceeded (10 req/min)                                               |