> ## Documentation Index
> Fetch the complete documentation index at: https://docs.prefetch.io/llms.txt
> Use this file to discover all available pages before exploring further.

# GET /scrape

> Turn any page into clean markdown, an LLM summary, structured JSON, links, or images. Costs 3 credits.

## Overview

`/scrape` fetches one URL and gives it back to you in whichever formats you ask for. It handles the parts that make scraping tedious:

* **JavaScript rendering** — pages that need a browser get one, automatically. Static pages take the fast path.
* **Bot walls** — blocked requests escalate through stealth and proxy tiers before giving up.
* **Boilerplate** — navigation, sidebars, footers, and cookie banners are stripped by default.
* **Absolute URLs** — every link and image is resolved against the page, so the output still works once you store it somewhere else.

Pass `formats` as a comma-separated list. Each format you request appears as its own object on `data`. Formats you do not request are **absent**, not null — so `if (data.markdown)` is a reliable check.

| Format     | You get                                                                |
| ---------- | ---------------------------------------------------------------------- |
| `markdown` | GitHub Flavored Markdown, plus character and word counts. The default. |
| `summary`  | A natural-language summary of the page.                                |
| `html`     | Cleaned HTML: scripts, styles, and inline event handlers removed.      |
| `raw_html` | The source exactly as fetched.                                         |
| `links`    | Every link, split into `internal` and `external`.                      |
| `images`   | Every image, including `srcset` and lazy-loaded sources.               |
| `json`     | Structured data matching your own prompt or JSON Schema.               |

`metadata` is always present, whatever you request.

## Example requests

<CodeGroup>
  ```bash Markdown (default) theme={null}
  curl "https://api.prefetch.io/scrape?url=https://stripe.com" \
    -H "X-API-Key: $PREFETCH_API_KEY"
  ```

  ```bash Markdown + links theme={null}
  curl "https://api.prefetch.io/scrape?url=https://stripe.com&formats=markdown,links" \
    -H "X-API-Key: $PREFETCH_API_KEY"
  ```

  ```bash Full page, no boilerplate stripping theme={null}
  curl "https://api.prefetch.io/scrape?url=https://stripe.com&only_main_content=false" \
    -H "X-API-Key: $PREFETCH_API_KEY"
  ```

  ```javascript Node.js theme={null}
  const params = new URLSearchParams({
    url: "https://stripe.com",
    formats: "markdown,links",
  });

  const res = await fetch(`https://api.prefetch.io/scrape?${params}`, {
    headers: { "X-API-Key": process.env.PREFETCH_API_KEY },
  });
  const { data } = await res.json();

  console.log(data.markdown.content);
  console.log(`${data.links.count} links found`);
  ```

  ```python Python theme={null}
  import requests, os

  r = requests.get(
      "https://api.prefetch.io/scrape",
      params={"url": "https://stripe.com", "formats": "markdown,links"},
      headers={"X-API-Key": os.environ["PREFETCH_API_KEY"]},
  )
  data = r.json()["data"]
  print(data["markdown"]["content"])
  ```
</CodeGroup>

## Example response

```json theme={null}
{
  "success": true,
  "data": {
    "url": "https://stripe.com",
    "final_url": "https://stripe.com",
    "domain": "stripe.com",
    "status_code": 200,
    "metadata": {
      "source_url": "https://stripe.com",
      "final_url": "https://stripe.com",
      "status_code": 200,
      "title": "Stripe | Financial Infrastructure to Grow Your Revenue",
      "description": "Stripe powers online payment processing for internet businesses.",
      "language": "en",
      "canonical_url": "https://stripe.com",
      "favicon": "https://stripe.com/favicon.ico",
      "site_name": "Stripe",
      "keywords": [],
      "open_graph": { "title": "Stripe", "type": "website" },
      "twitter": { "card": "summary_large_image" },
      "json_ld": [],
      "headings": [
        { "level": 1, "text": "Financial infrastructure to grow your revenue" }
      ]
    },
    "markdown": {
      "content": "# Financial infrastructure to grow your revenue\n\nJoin the millions of companies that use Stripe...",
      "char_count": 4821,
      "word_count": 702,
      "truncated": false,
      "main_content_only": true
    },
    "links": {
      "internal": [
        { "url": "https://stripe.com/pricing", "text": "Pricing", "title": null, "rel": null }
      ],
      "external": [],
      "count": 84
    }
  },
  "meta": {
    "requestId": "a3f2c1d4-7b6e-4f2a-9c1d-8e3f2a1b4c5d",
    "durationMs": 2140
  }
}
```

## Main content isolation

`only_main_content` defaults to `true`. It removes navigation, headers, footers, sidebars, share widgets, and cookie bars before rendering `markdown`, `html`, `summary`, and `json`.

When a page is not article-shaped — a product grid, a landing page — isolation would have to guess which fragment is "the content". Rather than guess, it returns the full page body, sets `main_content_only: false`, and tells you in `warnings`:

```json theme={null}
{
  "markdown": { "main_content_only": false, "...": "..." },
  "warnings": ["Main content could not be isolated; returned the full page body instead."]
}
```

<Note>
  `links` and `images` are always read from the **full** document, even when `only_main_content` is true. If you are mapping a site, the navigation links are exactly the ones you want.
</Note>

To take control yourself, use `include_selectors` to name the content, or `exclude_selectors` to name what to drop:

```bash theme={null}
curl "https://api.prefetch.io/scrape?url=https://example.com/post&include_selectors=article,.post-body" \
  -H "X-API-Key: $PREFETCH_API_KEY"
```

## Structured extraction with `json`

Ask for the `json` format with a plain-language `json_prompt`, a `json_schema`, or both. Requesting `json` without either returns a `400`.

<CodeGroup>
  ```bash Prompt theme={null}
  curl -G "https://api.prefetch.io/scrape" \
    --data-urlencode "url=https://example.com/product" \
    --data-urlencode "formats=json" \
    --data-urlencode "json_prompt=product name, price and availability" \
    -H "X-API-Key: $PREFETCH_API_KEY"
  ```

  ```bash JSON Schema theme={null}
  curl -G "https://api.prefetch.io/scrape" \
    --data-urlencode "url=https://example.com/product" \
    --data-urlencode "formats=json" \
    --data-urlencode 'json_schema={"type":"object","properties":{"name":{"type":"string"},"price":{"type":"number"}}}' \
    -H "X-API-Key: $PREFETCH_API_KEY"
  ```
</CodeGroup>

```json theme={null}
{
  "json": {
    "data": {
      "product_name": "Widget Pro",
      "price": "$49.00",
      "availability": "In stock"
    },
    "model": "gpt-4.1-mini"
  }
}
```

Values come only from the page. A field that is not on the page comes back `null` rather than being filled in from the model's own knowledge.

## Partial failures

The LLM-backed formats (`summary` and `json`) can fail on their own without losing the rest of the scrape. When that happens the format returns null values and the reason appears in `warnings`:

```json theme={null}
{
  "markdown": { "content": "# Still here...", "...": "..." },
  "summary": { "text": null, "model": null },
  "warnings": ["Summary failed: Request timed out"]
}
```

The response is still `success: true`, and you are still charged — the page was fetched.

## Content size

`markdown`, `html`, and `raw_html` are each capped at 1 MB. Content that hits the cap is cut on a character boundary and flagged:

```json theme={null}
{ "raw_html": { "char_count": 1000000, "truncated": true, "...": "..." } }
```

## Related

<CardGroup cols={2}>
  <Card title="GET /map" icon="sitemap" href="/api-reference/endpoint/map">
    Find every URL on a site before you scrape it. **3 credits.**
  </Card>

  <Card title="POST /crawl" icon="spider" href="/api-reference/endpoint/crawl">
    Scrape a whole site in one job. **3 credits + 3 per page.**
  </Card>
</CardGroup>


## OpenAPI

````yaml GET /scrape
openapi: 3.1.0
info:
  title: Prefetch API
  description: >-
    Extract brand identity, company data, screenshots, IAB classifications, and
    clean page content from any URL.
  version: 1.0.0
  contact:
    email: support@prefetch.io
servers:
  - url: https://api.prefetch.io
    description: Production
security:
  - apiKey: []
tags:
  - name: Endpoints
    description: Data extraction endpoints. All require authentication and consume credits.
  - name: Health
    description: Health and readiness probes. No authentication required.
paths:
  /scrape:
    get:
      tags:
        - Endpoints
      summary: Scrape a page
      description: >-
        Fetches one URL and returns it in the formats you request: clean
        markdown, an LLM summary, cleaned or raw HTML, links, images, or JSON
        matching your own schema. Pages that need JavaScript are rendered in a
        real browser automatically. **Credit cost: 3.**
      operationId: scrapeUrl
      parameters:
        - $ref: '#/components/parameters/url'
        - $ref: '#/components/parameters/scrapeFormats'
        - $ref: '#/components/parameters/onlyMainContent'
        - $ref: '#/components/parameters/includeLinks'
        - $ref: '#/components/parameters/includeImages'
        - $ref: '#/components/parameters/includeBase64Images'
        - $ref: '#/components/parameters/includeSelectors'
        - $ref: '#/components/parameters/excludeSelectors'
        - $ref: '#/components/parameters/jsonPrompt'
        - $ref: '#/components/parameters/jsonSchema'
        - $ref: '#/components/parameters/summaryMaxWords'
      responses:
        '200':
          description: Page scraped.
          content:
            application/json:
              schema:
                type: object
                properties:
                  success:
                    type: boolean
                    enum:
                      - true
                  data:
                    $ref: '#/components/schemas/ScrapeData'
                  meta:
                    $ref: '#/components/schemas/Meta'
                required:
                  - success
                  - data
                  - meta
              example:
                success: true
                data:
                  url: https://stripe.com
                  final_url: https://stripe.com
                  domain: stripe.com
                  status_code: 200
                  metadata:
                    source_url: https://stripe.com
                    final_url: https://stripe.com
                    status_code: 200
                    title: Stripe | Financial Infrastructure to Grow Your Revenue
                    description: >-
                      Stripe powers online payment processing for internet
                      businesses.
                    language: en
                    canonical_url: https://stripe.com
                    favicon: https://stripe.com/favicon.ico
                    author: null
                    site_name: Stripe
                    published_time: null
                    modified_time: null
                    robots: null
                    keywords: []
                    open_graph:
                      title: Stripe
                      type: website
                    twitter:
                      card: summary_large_image
                    json_ld: []
                    headings:
                      - level: 1
                        text: Financial infrastructure to grow your revenue
                  markdown:
                    content: >-
                      # Financial infrastructure to grow your revenue


                      Join the millions of companies that use Stripe to accept
                      payments online...
                    char_count: 4821
                    word_count: 702
                    truncated: false
                    main_content_only: true
                  links:
                    internal:
                      - url: https://stripe.com/pricing
                        text: Pricing
                        title: null
                        rel: null
                    external: []
                    count: 84
                meta:
                  requestId: a3f2c1d4-7b6e-4f2a-9c1d-8e3f2a1b4c5d
                  durationMs: 2140
        '400':
          description: Invalid or missing parameters.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
        '401':
          description: Missing API key.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
        '403':
          description: Revoked key, expired key, credit limit, or blocklisted URL.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
        '422':
          description: The URL hostname could not be resolved.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
        '429':
          description: Rate limit exceeded.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
        '503':
          description: Service temporarily busy. Retry shortly.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
        '504':
          description: Request timed out.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
components:
  parameters:
    url:
      name: url
      in: query
      required: true
      description: >-
        The website URL to process. `https://` is prepended automatically if no
        protocol is provided.
      schema:
        type: string
        example: https://stripe.com
    scrapeFormats:
      name: formats
      in: query
      required: false
      description: >-
        Comma-separated list of output formats. Each requested format appears as
        its own object on `data`; formats you do not request are absent, not
        null.
      schema:
        type: string
        default: markdown
        example: markdown,links
    onlyMainContent:
      name: only_main_content
      in: query
      required: false
      description: >-
        Strip navigation, sidebars, footers, and cookie banners before rendering
        `markdown`, `html`, `summary`, and `json`. `links` and `images` always
        come from the full document.
      schema:
        type: boolean
        default: true
    includeLinks:
      name: include_links
      in: query
      required: false
      description: >-
        Keep hyperlinks in the `markdown` and `html` output. When false, link
        text is kept and the target is dropped.
      schema:
        type: boolean
        default: true
    includeImages:
      name: include_images
      in: query
      required: false
      description: Keep images in the `markdown` and `html` output.
      schema:
        type: boolean
        default: true
    includeBase64Images:
      name: include_base64_images
      in: query
      required: false
      description: >-
        Keep inline `data:` images. Off by default because base64 payloads are
        large and carry no meaning as text.
      schema:
        type: boolean
        default: false
    includeSelectors:
      name: include_selectors
      in: query
      required: false
      description: >-
        Comma-separated CSS selectors to keep. Overrides main-content detection
        entirely.
      schema:
        type: string
        example: article,.post-body
    excludeSelectors:
      name: exclude_selectors
      in: query
      required: false
      description: Comma-separated CSS selectors to remove before rendering.
      schema:
        type: string
        example: .cookie-bar,#promo
    jsonPrompt:
      name: json_prompt
      in: query
      required: false
      description: >-
        What to extract, in plain language. Required for the `json` format
        unless `json_schema` is given.
      schema:
        type: string
        example: product name, price and availability
    jsonSchema:
      name: json_schema
      in: query
      required: false
      description: >-
        A JSON Schema, passed as a JSON string, describing the object the `json`
        format should return.
      schema:
        type: string
    summaryMaxWords:
      name: summary_max_words
      in: query
      required: false
      description: Word budget for the `summary` format.
      schema:
        type: integer
        minimum: 20
        maximum: 400
        default: 120
  schemas:
    ScrapeData:
      allOf:
        - $ref: '#/components/schemas/ScrapePayload'
      required:
        - url
        - final_url
        - metadata
    Meta:
      type: object
      properties:
        requestId:
          type: string
          format: uuid
          description: Unique identifier for this request.
          example: a3f2c1d4-7b6e-4f2a-9c1d-8e3f2a1b4c5d
        durationMs:
          type: integer
          nullable: true
          description: Time taken to process the request in milliseconds.
          example: 1842
      required:
        - requestId
        - durationMs
    ErrorResponse:
      type: object
      properties:
        success:
          type: boolean
          enum:
            - false
        error:
          type: string
          description: Human-readable error message.
          example: Credit limit exceeded
        meta:
          $ref: '#/components/schemas/Meta'
      required:
        - success
        - error
        - meta
    ScrapePayload:
      type: object
      properties:
        url:
          type: string
        final_url:
          type: string
        domain:
          type: string
          nullable: true
          example: stripe.com
        status_code:
          type: integer
          nullable: true
        metadata:
          $ref: '#/components/schemas/ScrapeMetadata'
        markdown:
          $ref: '#/components/schemas/MarkdownBlock'
        summary:
          $ref: '#/components/schemas/SummaryBlock'
        html:
          $ref: '#/components/schemas/HtmlBlock'
        raw_html:
          $ref: '#/components/schemas/HtmlBlock'
        links:
          $ref: '#/components/schemas/LinksBlock'
        images:
          $ref: '#/components/schemas/ScrapeImagesBlock'
        json:
          $ref: '#/components/schemas/JsonBlock'
        warnings:
          type: array
          items:
            type: string
          description: Non-fatal problems, such as a format that could not be produced.
      description: >-
        The page content fields. Which of them are present depends on the
        `formats` requested.
    ScrapeMetadata:
      type: object
      description: Always present, whatever formats were requested.
      properties:
        source_url:
          type: string
          nullable: true
          description: The URL you requested.
        final_url:
          type: string
          nullable: true
          description: The URL after redirects.
        status_code:
          type: integer
          nullable: true
        title:
          type: string
          nullable: true
          example: Stripe | Financial Infrastructure
        description:
          type: string
          nullable: true
        language:
          type: string
          nullable: true
          example: en
        canonical_url:
          type: string
          nullable: true
        favicon:
          type: string
          nullable: true
        author:
          type: string
          nullable: true
        site_name:
          type: string
          nullable: true
        published_time:
          type: string
          nullable: true
        modified_time:
          type: string
          nullable: true
        robots:
          type: string
          nullable: true
          description: The page's own robots meta tag.
        keywords:
          type: array
          items:
            type: string
        open_graph:
          type: object
          additionalProperties:
            type: string
        twitter:
          type: object
          additionalProperties:
            type: string
        json_ld:
          type: array
          items:
            type: object
          description: Parsed JSON-LD blocks, up to 10.
        headings:
          type: array
          description: The page's heading outline, up to 100 entries.
          items:
            type: object
            properties:
              level:
                type: integer
              text:
                type: string
    MarkdownBlock:
      type: object
      properties:
        content:
          type: string
          description: >-
            GitHub Flavored Markdown. Links and images are resolved to absolute
            URLs.
        char_count:
          type: integer
        word_count:
          type: integer
        truncated:
          type: boolean
          description: True when the content hit the 1 MB size cap and was cut.
        main_content_only:
          type: boolean
          description: >-
            True when boilerplate was successfully isolated away. False means
            the full page body was returned.
    SummaryBlock:
      type: object
      properties:
        text:
          type: string
          nullable: true
          description: Null when the summary could not be produced; see `warnings`.
        model:
          type: string
          nullable: true
    HtmlBlock:
      type: object
      properties:
        content:
          type: string
        char_count:
          type: integer
        truncated:
          type: boolean
        main_content_only:
          type: boolean
    LinksBlock:
      type: object
      properties:
        internal:
          type: array
          description: Links on the same site, subdomains included.
          items:
            $ref: '#/components/schemas/ScrapeLinkItem'
        external:
          type: array
          items:
            $ref: '#/components/schemas/ScrapeLinkItem'
        count:
          type: integer
    ScrapeImagesBlock:
      type: object
      properties:
        assets:
          type: array
          items:
            $ref: '#/components/schemas/ScrapeImageItem'
        og_image:
          type: string
          nullable: true
        twitter_image:
          type: string
          nullable: true
        count:
          type: integer
    JsonBlock:
      type: object
      properties:
        data:
          type: object
          nullable: true
          description: The extracted object, shaped by your prompt or schema.
        model:
          type: string
          nullable: true
    ScrapeLinkItem:
      type: object
      properties:
        url:
          type: string
        text:
          type: string
          nullable: true
          description: The anchor text.
        title:
          type: string
          nullable: true
        rel:
          type: string
          nullable: true
    ScrapeImageItem:
      type: object
      properties:
        url:
          type: string
        alt:
          type: string
          nullable: true
        width:
          type: integer
          nullable: true
        height:
          type: integer
          nullable: true
        loading:
          type: string
          nullable: true
  securitySchemes:
    apiKey:
      type: apiKey
      in: header
      name: X-API-Key
      description: Your Prefetch API key. Obtain one from the dashboard.

````