> ## Documentation Index
> Fetch the complete documentation index at: https://docs.prefetch.io/llms.txt
> Use this file to discover all available pages before exploring further.
# POST /batch/scrape
> Scrape a list of URLs in one job. Free to submit, then 1 credit per page returned.
## Overview
`/batch/scrape` scrapes up to 500 URLs you already have, as one job. Each page is fetched and rendered exactly as [`GET /scrape`](/api-reference/endpoint/scrape) would render it, configured through `scrape_options`. Nothing is followed and no sitemap is read: you get the pages you listed.
It works like a [crawl](/api-reference/endpoint/crawl): you get an id straight away, then read progress and pages from [`GET /batch/scrape/{id}`](/api-reference/endpoint/batch-scrape-status), or receive them by webhook.
Use it instead of many `/scrape` calls when you have a list of URLs: one request instead of hundreds, and no concurrency limit to manage on your side.
## Example request
```bash cURL theme={"theme":"css-variables"}
curl -X POST "https://api.prefetch.io/batch/scrape" \
-H "X-API-Key: $PREFETCH_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"urls": ["https://stripe.com/pricing", "https://stripe.com/docs"],
"scrape_options": { "formats": ["markdown"], "only_main_content": true }
}'
```
```javascript Node.js theme={"theme":"css-variables"}
const res = await fetch("https://api.prefetch.io/batch/scrape", {
method: "POST",
headers: { "X-API-Key": process.env.PREFETCH_API_KEY, "Content-Type": "application/json" },
body: JSON.stringify({
urls: ["https://stripe.com/pricing", "https://stripe.com/docs"],
scrape_options: { formats: ["markdown"] },
}),
});
const { data } = await res.json();
console.log(data.id); // poll GET /batch/scrape/{id}
```
## Invalid URLs
Every URL gets the checks a single `/scrape` would. By default one invalid or blocklisted URL rejects the whole batch with a `400` that names it. Set `ignore_invalid_urls: true` to drop those URLs instead; they are listed in the response's `invalid_urls`. Duplicate URLs, including the same page under another spelling (`www.`, a trailing slash), are scraped once.
## Webhooks
`webhook` works as it does for crawls, with events named `batch_scrape.started`, `batch_scrape.page`, `batch_scrape.completed` and `batch_scrape.failed`. See [crawl webhooks](/api-reference/endpoint/crawl#webhooks) for the payload and how to verify the signature. Not available through RapidAPI.
## Credits
Submitting is free. You are charged **1 credit per page returned**, or **4** when `scrape_options` asks for `summary`, `json` or `check_prompt_injection`. Pages are billed as they complete — when you poll the status endpoint, or when the batch ends — and failed pages are never charged.
To run brand, enrich, company, classify, screenshot or styleguide over a list instead, see [`POST /batch/{endpoint}`](/api-reference/endpoint/batch).
Over MCP, `prefetch_batch_scrape` submits the batch and waits for it, returning the pages when it finishes in time.
## Related
Read progress and pages.
Which pages failed, and why.
## OpenAPI
````yaml POST /batch/scrape
openapi: 3.1.0
info:
title: Prefetch API
description: >-
Extract brand identity, company data, screenshots, IAB classifications, and
clean page content from any URL.
version: 1.0.0
contact:
email: support@prefetch.io
servers:
- url: https://api.prefetch.io
description: Production
security:
- apiKey: []
tags:
- name: Endpoints
description: Data extraction endpoints. All require authentication and consume credits.
- name: Health
description: Health and readiness probes. No authentication required.
paths:
/batch/scrape:
post:
tags:
- Endpoints
summary: Scrape a list of URLs
description: >-
Queues a list of pages as one job and returns its id immediately. Each
page is fetched and rendered exactly as `GET /scrape` would, configured
through `scrape_options`; nothing is followed, and no sitemap or
robots.txt is read. Poll `GET /batch/scrape/{id}` for progress and
results, or pass a `webhook`.
**Credit cost: nothing to submit, 1 per page returned (4 when
`scrape_options` asks for `summary`, `json` or
`check_prompt_injection`).** Pages are charged as they complete.
operationId: startBatchScrape
requestBody:
required: true
content:
application/json:
schema:
$ref: '#/components/schemas/BatchScrapeRequest'
responses:
'202':
description: Batch queued.
content:
application/json:
schema:
type: object
properties:
success:
type: boolean
enum:
- true
data:
$ref: '#/components/schemas/BatchJobData'
meta:
$ref: '#/components/schemas/Meta'
required:
- success
- data
- meta
example:
success: true
data:
id: crw_5a8d2e7f1c4b4a9e8d3c6b1a0f9e8d7c
status: pending
url_count: 2
invalid_urls: []
created_at: '2026-10-11T09:14:03.221Z'
expires_at: '2026-10-12T09:14:03.221Z'
meta:
requestId: 9b8a7c6d-5e4f-4a3b-8c2d-1e0f9a8b7c6d
durationMs: 58
'400':
description: Invalid or missing parameters.
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponse'
'401':
description: Missing API key.
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponse'
'403':
description: Revoked key, expired key, credit limit, or blocklisted URL.
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponse'
'422':
description: The URL hostname could not be resolved.
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponse'
'429':
description: Rate limit exceeded.
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponse'
'503':
description: Service temporarily busy. Retry shortly.
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponse'
'504':
description: Request timed out.
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponse'
components:
schemas:
BatchScrapeRequest:
type: object
required:
- urls
properties:
urls:
type: array
items:
type: string
minItems: 1
maxItems: 500
description: The pages to scrape (1–500). Duplicates are scraped once.
example:
- https://stripe.com/pricing
- https://stripe.com/docs
concurrency:
type: integer
minimum: 1
maximum: 5
default: 2
description: Pages fetched in parallel.
ignore_invalid_urls:
type: boolean
default: false
description: >-
Drop URLs that fail validation (listed in `invalid_urls`) instead of
rejecting the whole batch.
webhook:
description: >-
Where to send progress: an https URL, or an object. Events:
`batch_scrape.started`, `batch_scrape.page` (one per page),
`batch_scrape.completed`, `batch_scrape.failed`. Each delivery is
signed — `X-Prefetch-Signature: t=,v1=`, HMAC-SHA256 of
`.` with the `secret` returned when the batch is
created — the same scheme as monitor webhooks.
oneOf:
- type: string
example: https://example.com/hooks/prefetch
- type: object
required:
- url
properties:
url:
type: string
events:
type: array
items:
type: string
enum:
- started
- page
- completed
- failed
headers:
type: object
additionalProperties:
type: string
metadata:
type: object
description: Echoed back in every payload (up to 4 KB).
scrape_options:
$ref: '#/components/schemas/CrawlScrapeOptions'
BatchJobData:
type: object
properties:
id:
type: string
example: crw_4b1e9c0a7d2f4e6b
endpoint:
type: string
description: What each URL is run through.
example: scrape
status:
type: string
enum:
- pending
- running
- completed
- failed
- cancelled
url_count:
type: integer
description: >-
Pages queued, after removing duplicates (and invalid URLs, with
ignore_invalid_urls).
invalid_urls:
type: array
items:
type: string
description: URLs dropped by ignore_invalid_urls.
created_at:
type: string
format: date-time
expires_at:
type: string
format: date-time
webhook:
type: object
description: Present when the request set `webhook`.
properties:
url:
type: string
events:
type: array
items:
type: string
secret:
type: string
description: >-
Verifies the `X-Prefetch-Signature` on each delivery. Shown only
here — store it.
required:
- id
- status
- url_count
- invalid_urls
- created_at
- expires_at
Meta:
type: object
properties:
requestId:
type: string
format: uuid
description: Unique identifier for this request.
example: a3f2c1d4-7b6e-4f2a-9c1d-8e3f2a1b4c5d
durationMs:
type: integer
nullable: true
description: Time taken to process the request in milliseconds.
example: 1842
required:
- requestId
- durationMs
ErrorResponse:
type: object
properties:
success:
type: boolean
enum:
- false
description: >-
Always false. Absent, like `meta`, on authentication and rate-limit
errors, which are answered before a request is processed.
error:
type: string
description: >-
Human-readable error message. The wording can change: branch on
`error_code`.
example: Credit limit exceeded
error_code:
type: string
enum:
- INVALID_REQUEST
- PAYLOAD_TOO_LARGE
- NOT_FOUND
- CONFLICT
- UNAUTHORIZED
- FORBIDDEN
- CREDITS_EXHAUSTED
- RATE_LIMITED
- CONCURRENCY_LIMITED
- URL_BLOCKED
- URL_NOT_PERMITTED
- DNS_RESOLUTION_FAILED
- TARGET_BLOCKED
- TARGET_CONNECTION_FAILED
- TARGET_TIMEOUT
- TARGET_SSL_ERROR
- TARGET_SERVER_ERROR
- TARGET_CLIENT_ERROR
- TARGET_NOT_A_PAGE
- TARGET_UNLOADABLE
- RESEARCH_INCONCLUSIVE
- TIMEOUT
- SERVICE_BUSY
- SERVICE_UNAVAILABLE
- INTERNAL_ERROR
description: >-
What went wrong, for machines. Stable: a code keeps its meaning once
published. See [Error handling](/concepts/errors).
example: CREDITS_EXHAUSTED
retryable:
type: boolean
description: >-
Whether the same request can succeed if sent again after a short
wait (honour `Retry-After` when present). False for problems with
the request or the target.
example: false
retryAfterSeconds:
type: integer
description: 'On `RATE_LIMITED`: seconds until a request will be accepted.'
limit:
type: integer
description: >-
On `CONCURRENCY_LIMITED`: how many requests this key may have in
flight.
meta:
$ref: '#/components/schemas/Meta'
required:
- error
- error_code
- retryable
CrawlScrapeOptions:
type: object
description: >-
How each crawled page is fetched and rendered. The same options `GET
/scrape` takes.
properties:
formats:
type: array
items:
type: string
enum:
- markdown
- summary
- html
- raw_html
- links
- images
- json
- screenshot
default:
- markdown
only_main_content:
type: boolean
default: true
include_links:
type: boolean
default: true
include_images:
type: boolean
default: true
include_base64_images:
type: boolean
default: false
include_selectors:
type: array
items:
type: string
exclude_selectors:
type: array
items:
type: string
json_prompt:
type: string
json_schema:
type: object
summary_max_words:
type: integer
minimum: 20
maximum: 400
default: 120
screenshot_full_page:
type: boolean
default: false
screenshot_format:
type: string
enum:
- jpeg
- png
- webp
default: jpeg
wait_for:
type: integer
minimum: 0
maximum: 30000
wait_for_selector:
type: string
headers:
type: object
additionalProperties:
type: string
description: >-
Sent with every page request. Pages fetched with headers are never
cached.
mobile:
type: boolean
default: false
max_age:
type: integer
minimum: 0
maximum: 2592000000
default: 0
description: >-
Accept stored copies of pages fetched at most this many milliseconds
ago.
store_in_cache:
type: boolean
default: true
check_prompt_injection:
type: boolean
default: false
description: >-
Check each page for prompt injection, as on `/scrape`. Adds the AI
surcharge to every page.
securitySchemes:
apiKey:
type: apiKey
in: header
name: X-API-Key
description: Your Prefetch API key. Obtain one from the dashboard.
````
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.