Skip to main content
POST
Start a crawl
Starts a crawl and returns 202 with a crawl id. Read the pages with GET /crawl/{id}. Free to submit. 1 credit per page returned, 4 with summary, json or check_prompt_injection. Failed requests are never charged.

Guide: Crawl a site

Scope, sitemap seeding, politeness, webhooks and examples.

Authorizations

X-API-Key
string
header
required

Your Prefetch API key. Obtain one from the dashboard.

Body

application/json
url
string
required

Where the crawl starts.

Example:

"https://docs.stripe.com"

limit
integer
default:25

Maximum pages to crawl.

Required range: 1 <= x <= 500
max_depth
integer
default:2

How many links deep to follow from the start URL. Sitemap-seeded pages sit at depth 0, so max_depth: 0 with use_sitemap: true crawls exactly what the sitemap lists and follows nothing.

Required range: 0 <= x <= 5
include_paths
string[]

Regular expressions matched against a URL's path and query. A URL must match at least one to be crawled.

Example:
exclude_paths
string[]

Regular expressions. A URL matching any of them is skipped. Exclusions win over inclusions.

Example:
allow_subdomains
boolean
default:false

Follow links onto subdomains of the start URL.

Follow links onto other sites.

ignore_query_params
boolean
default:true

Treat URLs that differ only by query string as one page.

respect_robots_txt
boolean
default:true

Obey the rules in robots.txt — Disallow and Crawl-delay. Sitemap discovery happens either way.

use_sitemap
boolean
default:true

Seed the crawl from the site's sitemap as well as from links.

crawl_entire_domain
boolean
default:false

Follow links anywhere on the site. By default a crawl stays at or below the path it starts from: one started at /docs/ crawls /docs/… only. A crawl from the root covers the whole site either way.

webhook

Where to send progress: an https URL, or an object. Events: crawl.started, crawl.page (one per page), crawl.completed, crawl.failed. Each delivery is signed — X-Prefetch-Signature: t=<unix>,v1=<hex>, HMAC-SHA256 of <t>.<raw body> with the secret returned when the crawl is created — the same scheme as monitor webhooks.

Example:

"https://example.com/hooks/prefetch"

delay_ms
integer
default:0

Politeness delay between pages. A larger Crawl-delay in robots.txt wins.

Required range: 0 <= x <= 30000
concurrency
integer
default:2

Pages fetched in parallel.

Required range: 1 <= x <= 5
scrape_options
object

How each crawled page is fetched and rendered. The same options GET /scrape takes.

Response

Crawl accepted and queued.

success
enum<boolean>
required
Available options:
true
data
object
required
meta
object
required