> ## Documentation Index
> Fetch the complete documentation index at: https://docs.prefetch.io/llms.txt
> Use this file to discover all available pages before exploring further.

# POST /crawl

> Crawl a whole site and get every page back as clean markdown. Costs 3 credits to submit, plus 3 per page returned.

## Overview

`/crawl` walks a site breadth-first from a start URL and renders every page it finds, exactly the way [`GET /scrape`](/api-reference/endpoint/scrape) would render it.

Crawls run for minutes, so this is the one asynchronous endpoint in the API. You submit a crawl, get an id back straight away, then poll [`GET /crawl/{id}`](/api-reference/endpoint/crawl-status) for progress and results.

<Steps>
  <Step title="Submit">
    `POST /crawl` returns `202` with a crawl id.
  </Step>

  <Step title="Poll">
    `GET /crawl/{id}` returns progress plus a page of results. Follow `next` until it is null.
  </Step>

  <Step title="Stop early (optional)">
    `DELETE /crawl/{id}` cancels a running crawl. Pages already fetched stay readable.
  </Step>
</Steps>

## Example request

<CodeGroup>
  ```bash cURL theme={null}
  curl -X POST "https://api.prefetch.io/crawl" \
    -H "X-API-Key: $PREFETCH_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{
      "url": "https://docs.stripe.com",
      "limit": 50,
      "max_depth": 2,
      "include_paths": ["^/docs/"],
      "exclude_paths": ["^/docs/changelog/"],
      "scrape_options": { "formats": ["markdown"] }
    }'
  ```

  ```javascript Node.js theme={null}
  const res = await fetch("https://api.prefetch.io/crawl", {
    method: "POST",
    headers: {
      "X-API-Key": process.env.PREFETCH_API_KEY,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      url: "https://docs.stripe.com",
      limit: 50,
      max_depth: 2,
      include_paths: ["^/docs/"],
      scrape_options: { formats: ["markdown"] },
    }),
  });

  const { data } = await res.json();
  console.log(data.id); // crw_9f2c1a7b4e0d4c3a8b1e5f7d2c9a0b3e
  ```

  ```python Python theme={null}
  import requests, os

  r = requests.post(
      "https://api.prefetch.io/crawl",
      headers={"X-API-Key": os.environ["PREFETCH_API_KEY"]},
      json={
          "url": "https://docs.stripe.com",
          "limit": 50,
          "max_depth": 2,
          "include_paths": ["^/docs/"],
          "scrape_options": {"formats": ["markdown"]},
      },
  )
  crawl_id = r.json()["data"]["id"]
  ```
</CodeGroup>

## Example response

```json theme={null}
{
  "success": true,
  "data": {
    "id": "crw_9f2c1a7b4e0d4c3a8b1e5f7d2c9a0b3e",
    "status": "pending",
    "url": "https://docs.stripe.com",
    "limit": 50,
    "max_depth": 2,
    "created_at": "2026-08-20T09:14:03.221Z",
    "expires_at": "2026-08-21T09:14:03.221Z"
  },
  "meta": {
    "requestId": "c5f4e3d2-9b8c-4d5e-af3a-2b4c6d8e0f21",
    "durationMs": 84
  }
}
```

The status is `202 Accepted`, not `200` — the crawl has been queued, not completed.

## Controlling scope

A crawl stays inside the boundary you draw. By default that means the site you pointed it at, two links deep, at most 25 pages.

| Parameter              | Default | What it does                                               |
| ---------------------- | ------- | ---------------------------------------------------------- |
| `limit`                | 25      | Maximum pages fetched. Hard ceiling: 500.                  |
| `max_depth`            | 2       | How many links deep to follow. Hard ceiling: 5.            |
| `include_paths`        | —       | Regular expressions. A URL's path must match at least one. |
| `exclude_paths`        | —       | Regular expressions. A matching URL is skipped.            |
| `allow_subdomains`     | `false` | Follow links onto subdomains.                              |
| `allow_external_links` | `false` | Follow links onto other sites.                             |

Patterns are matched against the **path and query**, not the whole URL — so `^/docs/` means the docs section, and a pattern containing the hostname will never match. Exclusions win over inclusions. An invalid regular expression is rejected at submission with a `400`, not silently ignored halfway through a crawl.

```json theme={null}
{
  "url": "https://docs.stripe.com",
  "include_paths": ["^/docs/payments/"],
  "exclude_paths": ["^/docs/payments/legacy/", "\\?locale="]
}
```

## Sitemap seeding and depth

With `use_sitemap` on (the default), the crawl seeds itself from the site's sitemap as well as following links.

Sitemap-seeded pages sit at **depth 0** — the site handed them over, you did not follow a link to reach them. That makes one particularly useful combination:

```json theme={null}
{ "url": "https://docs.stripe.com", "max_depth": 0, "use_sitemap": true, "limit": 200 }
```

This crawls exactly what the sitemap lists and follows nothing else.

## Politeness

Crawls obey `robots.txt` by default — both `Disallow` rules and `Crawl-delay`. Set `respect_robots_txt: false` to ignore the rules.

<Note>
  Turning the rules off never stops `robots.txt` being read. Its `Sitemap:` entries always seed the crawl, because a sitemap tells you where a site's pages are whether or not you are honouring its restrictions. Use `use_sitemap: false` to skip seeding.
</Note>

`delay_ms` adds a pause between pages, and `concurrency` (1–5, default 2) sets how many are fetched in parallel. A `Crawl-delay` in `robots.txt` wins whenever it asks for more space than `delay_ms`.

## Rendering each page

`scrape_options` takes the same options as [`GET /scrape`](/api-reference/endpoint/scrape), so a crawled page and a scraped page are the same object.

```json theme={null}
{
  "url": "https://docs.stripe.com",
  "scrape_options": {
    "formats": ["markdown", "links"],
    "only_main_content": true,
    "exclude_selectors": [".sidebar", ".version-banner"]
  }
}
```

<Warning>
  Formats are charged per page. Requesting `summary` or `json` on a 500-page crawl means 500 LLM calls. Start with a small `limit` to check the output before scaling up.
</Warning>

## Credits

You are charged **3 credits to submit**, plus **3 credits per page the crawl returns**.

Pages are billed as they complete, on the status endpoint, so:

* You pay for pages actually fetched, never for the `limit` you asked for.
* Pages that failed (`ok: false`) are not charged.
* Polling repeatedly never charges twice for the same page.
* A cancelled crawl is charged only for what it fetched before stopping.

## Next

<CardGroup cols={2}>
  <Card title="GET /crawl/{id}" icon="list-check" href="/api-reference/endpoint/crawl-status">
    Poll for progress and read the crawled pages.
  </Card>

  <Card title="DELETE /crawl/{id}" icon="ban" href="/api-reference/endpoint/crawl-cancel">
    Stop a running crawl. Free.
  </Card>
</CardGroup>


## OpenAPI

````yaml POST /crawl
openapi: 3.1.0
info:
  title: Prefetch API
  description: >-
    Extract brand identity, company data, screenshots, IAB classifications, and
    clean page content from any URL.
  version: 1.0.0
  contact:
    email: support@prefetch.io
servers:
  - url: https://api.prefetch.io
    description: Production
security:
  - apiKey: []
tags:
  - name: Endpoints
    description: Data extraction endpoints. All require authentication and consume credits.
  - name: Health
    description: Health and readiness probes. No authentication required.
paths:
  /crawl:
    post:
      tags:
        - Endpoints
      summary: Start a crawl
      description: >-
        Queues a breadth-first crawl and returns its id immediately. Crawls run
        for minutes, so nothing waits on this request — poll `GET /crawl/{id}`
        for progress and results.


        **Credit cost: 3 to submit, plus 3 per page the crawl returns.** Pages
        are charged as they complete, so you pay for what was fetched rather
        than for the `limit` you asked for.
      operationId: startCrawl
      requestBody:
        required: true
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/CrawlRequest'
      responses:
        '202':
          description: Crawl accepted and queued.
          content:
            application/json:
              schema:
                type: object
                properties:
                  success:
                    type: boolean
                    enum:
                      - true
                  data:
                    $ref: '#/components/schemas/CrawlJobData'
                  meta:
                    $ref: '#/components/schemas/Meta'
                required:
                  - success
                  - data
                  - meta
              example:
                success: true
                data:
                  id: crw_9f2c1a7b4e0d4c3a8b1e5f7d2c9a0b3e
                  status: pending
                  url: https://docs.stripe.com
                  limit: 50
                  max_depth: 2
                  created_at: '2026-08-20T09:14:03.221Z'
                  expires_at: '2026-08-21T09:14:03.221Z'
                meta:
                  requestId: c5f4e3d2-9b8c-4d5e-af3a-2b4c6d8e0f21
                  durationMs: 84
        '400':
          description: Invalid or missing parameters.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
        '401':
          description: Missing API key.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
        '403':
          description: Revoked key, expired key, credit limit, or blocklisted URL.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
        '422':
          description: The URL hostname could not be resolved.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
        '429':
          description: Rate limit exceeded.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
        '503':
          description: Service temporarily busy. Retry shortly.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
        '504':
          description: Request timed out.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
components:
  schemas:
    CrawlRequest:
      type: object
      required:
        - url
      properties:
        url:
          type: string
          description: Where the crawl starts.
          example: https://docs.stripe.com
        limit:
          type: integer
          minimum: 1
          maximum: 500
          default: 25
          description: Maximum pages to crawl.
        max_depth:
          type: integer
          minimum: 0
          maximum: 5
          default: 2
          description: >-
            How many links deep to follow from the start URL. Sitemap-seeded
            pages sit at depth 0, so `max_depth: 0` with `use_sitemap: true`
            crawls exactly what the sitemap lists and follows nothing.
        include_paths:
          type: array
          items:
            type: string
          description: >-
            Regular expressions matched against a URL's path and query. A URL
            must match at least one to be crawled.
          example:
            - ^/docs/
        exclude_paths:
          type: array
          items:
            type: string
          description: >-
            Regular expressions. A URL matching any of them is skipped.
            Exclusions win over inclusions.
          example:
            - ^/docs/changelog/
        allow_subdomains:
          type: boolean
          default: false
          description: Follow links onto subdomains of the start URL.
        allow_external_links:
          type: boolean
          default: false
          description: Follow links onto other sites.
        ignore_query_params:
          type: boolean
          default: true
          description: Treat URLs that differ only by query string as one page.
        respect_robots_txt:
          type: boolean
          default: true
          description: >-
            Obey the rules in `robots.txt` — Disallow and Crawl-delay. Sitemap
            discovery happens either way.
        use_sitemap:
          type: boolean
          default: true
          description: Seed the crawl from the site's sitemap as well as from links.
        delay_ms:
          type: integer
          minimum: 0
          maximum: 30000
          default: 0
          description: >-
            Politeness delay between pages. A larger `Crawl-delay` in robots.txt
            wins.
        concurrency:
          type: integer
          minimum: 1
          maximum: 5
          default: 2
          description: Pages fetched in parallel.
        scrape_options:
          $ref: '#/components/schemas/CrawlScrapeOptions'
    CrawlJobData:
      type: object
      properties:
        id:
          type: string
          example: crw_9f2c1a7b4e0d4c3a8b1e5f7d2c9a0b3e
        status:
          type: string
          enum:
            - pending
            - running
            - completed
            - failed
            - cancelled
        url:
          type: string
        limit:
          type: integer
        max_depth:
          type: integer
        created_at:
          type: string
          format: date-time
        expires_at:
          type: string
          format: date-time
          description: After this, the crawl and its pages are no longer readable.
    Meta:
      type: object
      properties:
        requestId:
          type: string
          format: uuid
          description: Unique identifier for this request.
          example: a3f2c1d4-7b6e-4f2a-9c1d-8e3f2a1b4c5d
        durationMs:
          type: integer
          nullable: true
          description: Time taken to process the request in milliseconds.
          example: 1842
      required:
        - requestId
        - durationMs
    ErrorResponse:
      type: object
      properties:
        success:
          type: boolean
          enum:
            - false
        error:
          type: string
          description: Human-readable error message.
          example: Credit limit exceeded
        meta:
          $ref: '#/components/schemas/Meta'
      required:
        - success
        - error
        - meta
    CrawlScrapeOptions:
      type: object
      description: How each crawled page is rendered. The same options `GET /scrape` takes.
      properties:
        formats:
          type: array
          items:
            type: string
            enum:
              - markdown
              - summary
              - html
              - raw_html
              - links
              - images
              - json
          default:
            - markdown
        only_main_content:
          type: boolean
          default: true
        include_links:
          type: boolean
          default: true
        include_images:
          type: boolean
          default: true
        include_base64_images:
          type: boolean
          default: false
        include_selectors:
          type: array
          items:
            type: string
        exclude_selectors:
          type: array
          items:
            type: string
        json_prompt:
          type: string
        json_schema:
          type: object
        summary_max_words:
          type: integer
          minimum: 20
          maximum: 400
          default: 120
  securitySchemes:
    apiKey:
      type: apiKey
      in: header
      name: X-API-Key
      description: Your Prefetch API key. Obtain one from the dashboard.

````