> ## Documentation Index
> Fetch the complete documentation index at: https://docs.prefetch.io/llms.txt
> Use this file to discover all available pages before exploring further.

# GET /crawl/{id}

> Poll a crawl for progress and read its pages. Charges 3 credits for each page completed since your last call.

## Overview

Returns where a crawl has got to, plus a page of results in completion order. Poll it until `status` is `completed`.

Results are paginated. Each response carries a `next` link; follow it until `next` is `null` to read every page. Results stay readable for **24 hours** after a crawl finishes, then expire.

## Example request

<CodeGroup>
  ```bash cURL theme={null}
  curl "https://api.prefetch.io/crawl/crw_9f2c1a7b4e0d4c3a8b1e5f7d2c9a0b3e" \
    -H "X-API-Key: $PREFETCH_API_KEY"
  ```

  ```javascript Node.js theme={null}
  const id = "crw_9f2c1a7b4e0d4c3a8b1e5f7d2c9a0b3e";
  const headers = { "X-API-Key": process.env.PREFETCH_API_KEY };

  // Poll until the crawl finishes, then read every page of results.
  let status;
  do {
    await new Promise((r) => setTimeout(r, 5000));
    const res = await fetch(`https://api.prefetch.io/crawl/${id}`, { headers });
    ({ data: status } = await res.json());
    console.log(`${status.completed}/${status.total} pages`);
  } while (status.status === "pending" || status.status === "running");

  const pages = [...status.data];
  let next = status.next;
  while (next) {
    const res = await fetch(`https://api.prefetch.io${next}`, { headers });
    const { data } = await res.json();
    pages.push(...data.data);
    next = data.next;
  }
  console.log(`${pages.length} pages collected`);
  ```

  ```python Python theme={null}
  import os, time, requests

  crawl_id = "crw_9f2c1a7b4e0d4c3a8b1e5f7d2c9a0b3e"
  headers = {"X-API-Key": os.environ["PREFETCH_API_KEY"]}
  base = "https://api.prefetch.io"

  while True:
      data = requests.get(f"{base}/crawl/{crawl_id}", headers=headers).json()["data"]
      print(f"{data['completed']}/{data['total']} pages")
      if data["status"] not in ("pending", "running"):
          break
      time.sleep(5)

  pages, nxt = list(data["data"]), data["next"]
  while nxt:
      data = requests.get(base + nxt, headers=headers).json()["data"]
      pages.extend(data["data"])
      nxt = data["next"]
  ```
</CodeGroup>

## Example response

```json theme={null}
{
  "success": true,
  "data": {
    "id": "crw_9f2c1a7b4e0d4c3a8b1e5f7d2c9a0b3e",
    "status": "running",
    "url": "https://docs.stripe.com",
    "total": 38,
    "completed": 12,
    "failed": 1,
    "credits_used": 36,
    "created_at": "2026-08-20T09:14:03.221Z",
    "started_at": "2026-08-20T09:14:04.010Z",
    "finished_at": null,
    "expires_at": "2026-08-21T09:14:03.221Z",
    "error": null,
    "next": "/crawl/crw_9f2c1a7b4e0d4c3a8b1e5f7d2c9a0b3e?skip=25&limit=25",
    "data": [
      {
        "url": "https://docs.stripe.com/payments",
        "depth": 0,
        "status_code": 200,
        "ok": true,
        "error": null,
        "final_url": "https://docs.stripe.com/payments",
        "domain": "docs.stripe.com",
        "metadata": {
          "title": "Payments",
          "description": "Accept payments online.",
          "language": "en"
        },
        "markdown": {
          "content": "# Payments\n\nAccept payments online, in person, and around the world...",
          "char_count": 2841,
          "word_count": 402,
          "truncated": false,
          "main_content_only": true
        }
      }
    ]
  },
  "meta": {
    "requestId": "d6a5f4e3-ac9d-4e6f-b04b-3c5d7e9f1a32",
    "durationMs": 96
  }
}
```

## Statuses

| Status      | Meaning                                                               |
| ----------- | --------------------------------------------------------------------- |
| `pending`   | Queued, not started.                                                  |
| `running`   | In progress. `data` already holds the pages finished so far.          |
| `completed` | Finished.                                                             |
| `failed`    | Stopped early — see `error`. Pages already stored are still readable. |
| `cancelled` | You called `DELETE /crawl/{id}`.                                      |

<Note>
  `total` is the frontier size **as currently known**, not a final count. A crawl discovers pages as it goes, so this number rises until the frontier is exhausted or `limit` is reached. Do not treat `completed === total` as "finished" — check `status`.
</Note>

## Page objects

Each entry in `data` carries the crawl's bookkeeping plus the same fields `GET /scrape` returns for that page:

| Field                                    | Meaning                                                                                    |
| ---------------------------------------- | ------------------------------------------------------------------------------------------ |
| `url`                                    | The page that was crawled.                                                                 |
| `depth`                                  | How many links from the start URL. Sitemap-seeded pages are `0`.                           |
| `ok`                                     | `false` when the page could not be fetched.                                                |
| `error`                                  | Why it failed, when `ok` is `false`.                                                       |
| `status_code`, `metadata`, `markdown`, … | Exactly as in [`GET /scrape`](/api-reference/endpoint/scrape), shaped by `scrape_options`. |

Failed pages are listed rather than silently dropped, so you can see what a crawl could not reach. You are not charged for them.

## Billing

This endpoint is where crawled pages are charged: each call bills for the pages that completed **since your previous call**, at 3 credits each.

That makes polling safe — calling it ten times while a crawl runs costs exactly the same as calling it once at the end. `credits_used` on the response shows the running total for the crawl.

## Related

<CardGroup cols={2}>
  <Card title="POST /crawl" icon="spider" href="/api-reference/endpoint/crawl">
    Start a crawl and configure its scope.
  </Card>

  <Card title="DELETE /crawl/{id}" icon="ban" href="/api-reference/endpoint/crawl-cancel">
    Stop a running crawl. Free.
  </Card>
</CardGroup>


## OpenAPI

````yaml GET /crawl/{id}
openapi: 3.1.0
info:
  title: Prefetch API
  description: >-
    Extract brand identity, company data, screenshots, IAB classifications, and
    clean page content from any URL.
  version: 1.0.0
  contact:
    email: support@prefetch.io
servers:
  - url: https://api.prefetch.io
    description: Production
security:
  - apiKey: []
tags:
  - name: Endpoints
    description: Data extraction endpoints. All require authentication and consume credits.
  - name: Health
    description: Health and readiness probes. No authentication required.
paths:
  /crawl/{id}:
    get:
      tags:
        - Endpoints
      summary: Get crawl status and results
      description: >-
        Returns progress plus a page of crawled pages, oldest first. Follow
        `next` until it is null to read every page. Results stay readable for 24
        hours after the crawl finishes.


        This is also where crawled pages are billed: each call charges for the
        pages that completed since the previous call, so polling repeatedly
        never charges twice for the same page.
      operationId: getCrawl
      parameters:
        - $ref: '#/components/parameters/crawlId'
        - $ref: '#/components/parameters/crawlSkip'
        - $ref: '#/components/parameters/crawlPageLimit'
      responses:
        '200':
          description: Crawl status.
          content:
            application/json:
              schema:
                type: object
                properties:
                  success:
                    type: boolean
                    enum:
                      - true
                  data:
                    $ref: '#/components/schemas/CrawlStatusData'
                  meta:
                    $ref: '#/components/schemas/Meta'
                required:
                  - success
                  - data
                  - meta
              example:
                success: true
                data:
                  id: crw_9f2c1a7b4e0d4c3a8b1e5f7d2c9a0b3e
                  status: running
                  url: https://docs.stripe.com
                  total: 38
                  completed: 12
                  failed: 1
                  credits_used: 36
                  created_at: '2026-08-20T09:14:03.221Z'
                  started_at: '2026-08-20T09:14:04.010Z'
                  finished_at: null
                  expires_at: '2026-08-21T09:14:03.221Z'
                  error: null
                  next: /crawl/crw_9f2c1a7b4e0d4c3a8b1e5f7d2c9a0b3e?skip=25&limit=25
                  data:
                    - url: https://docs.stripe.com/payments
                      depth: 0
                      status_code: 200
                      ok: true
                      error: null
                      final_url: https://docs.stripe.com/payments
                      domain: docs.stripe.com
                      metadata:
                        title: Payments
                        description: Accept payments online.
                        language: en
                      markdown:
                        content: >-
                          # Payments


                          Accept payments online, in person, and around the
                          world...
                        char_count: 2841
                        word_count: 402
                        truncated: false
                        main_content_only: true
                meta:
                  requestId: d6a5f4e3-ac9d-4e6f-b04b-3c5d7e9f1a32
                  durationMs: 96
        '400':
          description: Invalid or missing parameters.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
        '401':
          description: Missing API key.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
        '403':
          description: Revoked key, expired key, credit limit, or blocklisted URL.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
        '404':
          description: Crawl not found, expired, or owned by another key.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
        '422':
          description: The URL hostname could not be resolved.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
        '429':
          description: Rate limit exceeded.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
        '503':
          description: Service temporarily busy. Retry shortly.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
        '504':
          description: Request timed out.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
components:
  parameters:
    crawlId:
      name: id
      in: path
      required: true
      description: The crawl id returned by `POST /crawl`.
      schema:
        type: string
        example: crw_9f2c1a7b4e0d4c3a8b1e5f7d2c9a0b3e
    crawlSkip:
      name: skip
      in: query
      required: false
      description: >-
        Number of result pages to skip. Use the `next` link rather than building
        this by hand.
      schema:
        type: integer
        minimum: 0
        default: 0
    crawlPageLimit:
      name: limit
      in: query
      required: false
      description: Number of crawled pages to return per request.
      schema:
        type: integer
        minimum: 1
        maximum: 100
        default: 25
  schemas:
    CrawlStatusData:
      type: object
      properties:
        id:
          type: string
        status:
          type: string
          enum:
            - pending
            - running
            - completed
            - failed
            - cancelled
        url:
          type: string
        total:
          type: integer
          description: >-
            Pages the crawl expects to fetch, as currently known. Rises as it
            discovers more.
        completed:
          type: integer
        failed:
          type: integer
        credits_used:
          type: integer
        created_at:
          type: string
          format: date-time
        started_at:
          type: string
          format: date-time
          nullable: true
        finished_at:
          type: string
          format: date-time
          nullable: true
        expires_at:
          type: string
          format: date-time
        error:
          type: string
          nullable: true
        next:
          type: string
          nullable: true
          description: URL for the next page of results, or null when there are no more.
        data:
          type: array
          items:
            $ref: '#/components/schemas/CrawlPage'
          description: Crawled pages, in completion order.
    Meta:
      type: object
      properties:
        requestId:
          type: string
          format: uuid
          description: Unique identifier for this request.
          example: a3f2c1d4-7b6e-4f2a-9c1d-8e3f2a1b4c5d
        durationMs:
          type: integer
          nullable: true
          description: Time taken to process the request in milliseconds.
          example: 1842
      required:
        - requestId
        - durationMs
    ErrorResponse:
      type: object
      properties:
        success:
          type: boolean
          enum:
            - false
        error:
          type: string
          description: Human-readable error message.
          example: Credit limit exceeded
        meta:
          $ref: '#/components/schemas/Meta'
      required:
        - success
        - error
        - meta
    CrawlPage:
      type: object
      description: >-
        One crawled page. Carries the same fields `GET /scrape` returns, plus
        the crawl's own bookkeeping.
      allOf:
        - type: object
          properties:
            url:
              type: string
            depth:
              type: integer
              description: >-
                How many links from the start URL. Sitemap-seeded pages are
                depth 0.
            status_code:
              type: integer
              nullable: true
            ok:
              type: boolean
              description: >-
                False when the page could not be fetched. You are not charged
                for it.
            error:
              type: string
              nullable: true
        - $ref: '#/components/schemas/ScrapePayload'
    ScrapePayload:
      type: object
      properties:
        url:
          type: string
        final_url:
          type: string
        domain:
          type: string
          nullable: true
          example: stripe.com
        status_code:
          type: integer
          nullable: true
        metadata:
          $ref: '#/components/schemas/ScrapeMetadata'
        markdown:
          $ref: '#/components/schemas/MarkdownBlock'
        summary:
          $ref: '#/components/schemas/SummaryBlock'
        html:
          $ref: '#/components/schemas/HtmlBlock'
        raw_html:
          $ref: '#/components/schemas/HtmlBlock'
        links:
          $ref: '#/components/schemas/LinksBlock'
        images:
          $ref: '#/components/schemas/ScrapeImagesBlock'
        json:
          $ref: '#/components/schemas/JsonBlock'
        warnings:
          type: array
          items:
            type: string
          description: Non-fatal problems, such as a format that could not be produced.
      description: >-
        The page content fields. Which of them are present depends on the
        `formats` requested.
    ScrapeMetadata:
      type: object
      description: Always present, whatever formats were requested.
      properties:
        source_url:
          type: string
          nullable: true
          description: The URL you requested.
        final_url:
          type: string
          nullable: true
          description: The URL after redirects.
        status_code:
          type: integer
          nullable: true
        title:
          type: string
          nullable: true
          example: Stripe | Financial Infrastructure
        description:
          type: string
          nullable: true
        language:
          type: string
          nullable: true
          example: en
        canonical_url:
          type: string
          nullable: true
        favicon:
          type: string
          nullable: true
        author:
          type: string
          nullable: true
        site_name:
          type: string
          nullable: true
        published_time:
          type: string
          nullable: true
        modified_time:
          type: string
          nullable: true
        robots:
          type: string
          nullable: true
          description: The page's own robots meta tag.
        keywords:
          type: array
          items:
            type: string
        open_graph:
          type: object
          additionalProperties:
            type: string
        twitter:
          type: object
          additionalProperties:
            type: string
        json_ld:
          type: array
          items:
            type: object
          description: Parsed JSON-LD blocks, up to 10.
        headings:
          type: array
          description: The page's heading outline, up to 100 entries.
          items:
            type: object
            properties:
              level:
                type: integer
              text:
                type: string
    MarkdownBlock:
      type: object
      properties:
        content:
          type: string
          description: >-
            GitHub Flavored Markdown. Links and images are resolved to absolute
            URLs.
        char_count:
          type: integer
        word_count:
          type: integer
        truncated:
          type: boolean
          description: True when the content hit the 1 MB size cap and was cut.
        main_content_only:
          type: boolean
          description: >-
            True when boilerplate was successfully isolated away. False means
            the full page body was returned.
    SummaryBlock:
      type: object
      properties:
        text:
          type: string
          nullable: true
          description: Null when the summary could not be produced; see `warnings`.
        model:
          type: string
          nullable: true
    HtmlBlock:
      type: object
      properties:
        content:
          type: string
        char_count:
          type: integer
        truncated:
          type: boolean
        main_content_only:
          type: boolean
    LinksBlock:
      type: object
      properties:
        internal:
          type: array
          description: Links on the same site, subdomains included.
          items:
            $ref: '#/components/schemas/ScrapeLinkItem'
        external:
          type: array
          items:
            $ref: '#/components/schemas/ScrapeLinkItem'
        count:
          type: integer
    ScrapeImagesBlock:
      type: object
      properties:
        assets:
          type: array
          items:
            $ref: '#/components/schemas/ScrapeImageItem'
        og_image:
          type: string
          nullable: true
        twitter_image:
          type: string
          nullable: true
        count:
          type: integer
    JsonBlock:
      type: object
      properties:
        data:
          type: object
          nullable: true
          description: The extracted object, shaped by your prompt or schema.
        model:
          type: string
          nullable: true
    ScrapeLinkItem:
      type: object
      properties:
        url:
          type: string
        text:
          type: string
          nullable: true
          description: The anchor text.
        title:
          type: string
          nullable: true
        rel:
          type: string
          nullable: true
    ScrapeImageItem:
      type: object
      properties:
        url:
          type: string
        alt:
          type: string
          nullable: true
        width:
          type: integer
          nullable: true
        height:
          type: integer
          nullable: true
        loading:
          type: string
          nullable: true
  securitySchemes:
    apiKey:
      type: apiKey
      in: header
      name: X-API-Key
      description: Your Prefetch API key. Obtain one from the dashboard.

````