> ## Documentation Index > Fetch the complete documentation index at: https://docs.prefetch.io/llms.txt > Use this file to discover all available pages before exploring further. # GET /crawl/{id}/errors > Why a crawl returned fewer pages than expected: failed pages with error codes, and skipped URLs by reason. Free. ## Overview Explains a crawl that came back smaller than you expected. It returns two things: * **`failed`** — pages that were fetched and failed, oldest first, each with an [`error_code`](/concepts/errors#error-codes). Paged with `skip` and `limit`; follow `next` until it is `null`. * **`skipped`** — every URL the crawl found but did not fetch, counted by reason, with a few example URLs for each. | Reason | Meaning | | - | - | | `limit` | The crawl had already queued `limit` pages. | | `depth` | Deeper than `max_depth`. | | `out-of-scope` | Above the start URL's path. Set `crawl_entire_domain` to follow these. | | `excluded` | Matched `exclude_paths`. | | `not-included` | Matched none of `include_paths`. | | `robots` | Disallowed by the site's `robots.txt`. | | `off-site` | On another site or a subdomain you did not allow. | | `file` | A file rather than a page — an image, an archive, a stylesheet. | | `invalid` | Not a URL the API will fetch, such as a private address. | | `blocklist` | On your key's blocklist. | | `duplicate` | A page the crawl already had, under another URL or through a redirect. | Works while the crawl is running and until it expires. Free. ## Example request ```bash cURL theme={"theme":"css-variables"} curl "https://api.prefetch.io/crawl/crw_9f2c1a7b4e0d4c3a8b1e5f7d2c9a0b3e/errors" \ -H "X-API-Key: $PREFETCH_API_KEY" ``` ## Related Start a crawl. What each error code means, and which are worth retrying. ## OpenAPI ````yaml GET /crawl/{id}/errors openapi: 3.1.0 info: title: Prefetch API description: >- Extract brand identity, company data, screenshots, IAB classifications, and clean page content from any URL. version: 1.0.0 contact: email: support@prefetch.io servers: - url: https://api.prefetch.io description: Production security: - apiKey: [] tags: - name: Endpoints description: Data extraction endpoints. All require authentication and consume credits. - name: Health description: Health and readiness probes. No authentication required. paths: /crawl/{id}/errors: get: tags: - Endpoints summary: Get crawl errors and skipped URLs description: >- Why a crawl came back with less than you expected: the pages that were fetched and failed, oldest first (paged with `skip`/`limit`), each with an `error_code`, and `skipped`, every URL the crawl found but did not fetch, counted by reason with a few examples of each. Free. operationId: getCrawlErrors parameters: - $ref: '#/components/parameters/crawlId' - $ref: '#/components/parameters/crawlSkip' - $ref: '#/components/parameters/crawlPageLimit' responses: '200': description: Failed pages and skip reasons. content: application/json: schema: type: object properties: success: type: boolean enum: - true data: $ref: '#/components/schemas/CrawlErrorsData' meta: $ref: '#/components/schemas/Meta' required: - success - data - meta example: success: true data: id: crw_9f2c1a7b4e0d4c3a8b1e5f7d2c9a0b3e status: completed failed: - url: https://docs.stripe.com/legacy/v1 depth: 2 status_code: null error: Failed to load page (HTTP 503) error_code: TARGET_SERVER_ERROR failed_total: 1 skipped: out-of-scope: count: 41 samples: - https://docs.stripe.com/blog/2024 - https://docs.stripe.com/jobs robots: count: 3 samples: - https://docs.stripe.com/search next: null meta: requestId: 4c1e2a9b-7d3f-4b8e-9a6c-1f2e3d4c5b6a durationMs: 41 '400': description: Invalid or missing parameters. content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' '401': description: Missing API key. content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' '403': description: Revoked key, expired key, credit limit, or blocklisted URL. content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' '404': description: Crawl not found, expired, or owned by another key. content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' '422': description: The URL hostname could not be resolved. content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' '429': description: Rate limit exceeded. content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' '503': description: Service temporarily busy. Retry shortly. content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' '504': description: Request timed out. content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' components: parameters: crawlId: name: id in: path required: true description: The crawl id returned by `POST /crawl`. schema: type: string example: crw_9f2c1a7b4e0d4c3a8b1e5f7d2c9a0b3e crawlSkip: name: skip in: query required: false description: >- Number of result pages to skip. Use the `next` link rather than building this by hand. schema: type: integer minimum: 0 default: 0 crawlPageLimit: name: limit in: query required: false description: Number of crawled pages to return per request. schema: type: integer minimum: 1 maximum: 100 default: 25 schemas: CrawlErrorsData: type: object properties: id: type: string status: type: string enum: - pending - running - completed - failed - cancelled failed: type: array items: type: object properties: url: type: string depth: type: integer status_code: type: integer nullable: true error: type: string nullable: true error_code: type: string nullable: true description: >- Why the page failed, as one of the API's [error codes](/concepts/errors#error-codes). example: TARGET_SERVER_ERROR required: - url - depth - status_code - error - error_code description: Pages that were fetched and failed, oldest first. failed_total: type: integer skipped: $ref: '#/components/schemas/SkippedUrls' next: type: string nullable: true required: - id - status - failed - failed_total - skipped - next Meta: type: object properties: requestId: type: string format: uuid description: Unique identifier for this request. example: a3f2c1d4-7b6e-4f2a-9c1d-8e3f2a1b4c5d durationMs: type: integer nullable: true description: Time taken to process the request in milliseconds. example: 1842 required: - requestId - durationMs ErrorResponse: type: object properties: success: type: boolean enum: - false description: >- Always false. Absent, like `meta`, on authentication and rate-limit errors, which are answered before a request is processed. error: type: string description: >- Human-readable error message. The wording can change: branch on `error_code`. example: Credit limit exceeded error_code: type: string enum: - INVALID_REQUEST - PAYLOAD_TOO_LARGE - NOT_FOUND - CONFLICT - UNAUTHORIZED - FORBIDDEN - CREDITS_EXHAUSTED - RATE_LIMITED - CONCURRENCY_LIMITED - URL_BLOCKED - URL_NOT_PERMITTED - DNS_RESOLUTION_FAILED - TARGET_BLOCKED - TARGET_CONNECTION_FAILED - TARGET_TIMEOUT - TARGET_SSL_ERROR - TARGET_SERVER_ERROR - TARGET_CLIENT_ERROR - TARGET_NOT_A_PAGE - TARGET_UNLOADABLE - RESEARCH_INCONCLUSIVE - TIMEOUT - SERVICE_BUSY - SERVICE_UNAVAILABLE - INTERNAL_ERROR description: >- What went wrong, for machines. Stable: a code keeps its meaning once published. See [Error handling](/concepts/errors). example: CREDITS_EXHAUSTED retryable: type: boolean description: >- Whether the same request can succeed if sent again after a short wait (honour `Retry-After` when present). False for problems with the request or the target. example: false retryAfterSeconds: type: integer description: 'On `RATE_LIMITED`: seconds until a request will be accepted.' limit: type: integer description: >- On `CONCURRENCY_LIMITED`: how many requests this key may have in flight. meta: $ref: '#/components/schemas/Meta' required: - error - error_code - retryable SkippedUrls: type: object additionalProperties: type: object properties: count: type: number samples: type: array items: type: string required: - count - samples description: >- URLs the crawl found but did not fetch, by reason: `limit`, `depth`, `out-of-scope`, `excluded`, `not-included`, `robots`, `off-site`, `file`, `invalid`, `blocklist`, `duplicate` (a redirect onto a page already crawled). securitySchemes: apiKey: type: apiKey in: header name: X-API-Key description: Your Prefetch API key. Obtain one from the dashboard. ```` This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.