Limit crawl depth and page count

Set a small limit, an explicit maxDepth and the URL scope before starting a crawl. Then inspect the finished job against that saved request. In Scraipe, limit bounds returned pages; it does not establish a maximum number of HTTP requests or prove that you covered the whole site.

This guide includes a Node.js checker for a saved crawl job, plus controlled examples of how depth, sitemap seeds and failed pages interact. The examples were checked on 7 November 2026. The downloadable responses are teaching fixtures, not customer data or results from a hosted crawl.

Choose three separate bounds

Use the Map workflow to understand the candidate URLs first. Give the crawl three independent constraints:

The current HTTP route defaults to limit: 25 and maxDepth: 2. The crawler caps results at 1,000. Send positive integer limits and nonnegative integer depths explicitly; do not rely on coercion of strings, fractions or nulls. Those values are the client policy used here, not a claim that the server rejects every malformed input.

The start URL is depth 0 and counts toward the result limit when it succeeds. The queue admits URLs with the same origin as the start URL. An origin includes the scheme, hostname and port: a sibling subdomain is a different origin. This queue rule is not a guarantee about every redirect hop or browser subresource request.

URL globs use * as the wildcard and match case-insensitively. Exclusions take precedence. The filters also apply to the start URL after fragment and trailing-slash normalization. For a start URL of https://example.com/docs, an include rule ending in /docs/* alone does not include that seed. List both the exact seed and the descendant pattern. Omit unused filter properties rather than sending empty arrays.

Understand depth when a sitemap is present

A URL's number of path segments is not its crawl depth. The Sitemap protocol describes URL entries; Scraipe assigns their traversal depth when seeding its queue.

In the current crawler, sitemap candidates enter at depth 1. Consequently, /docs/a/b/c can be returned with maxDepth: 1 even if no link from the seed points directly to it. Links found on a fetched page are considered at that page's depth plus one.

These outcomes were reproduced against the repository crawler with in-memory discovery, fetch and reader stubs. They test the traversal logic; they do not measure hosted network requests, rendering or billing.

The HTTP crawl route does not expose switches for useSitemap or concurrency. Adding those names to a request is not a supported way to disable sitemap seeding or set a request budget. When you need an exact approved URL list, issue individual Read requests under your own scheduler. Even then, a Read request can involve multiple origin requests; enforce a strict transport budget in infrastructure that actually measures them.

Save a bounded request and submit once

Use a site you are permitted to crawl and an API key with the scrape scope. Store this JSON as request.json, replacing the example host and both include patterns with your permitted target. The host below is illustrative, not a runnable documentation site.

{
  "url": "https://example.com/docs",
  "limit": 2,
  "maxDepth": 1,
  "mode": "read",
  "egress": "direct",
  "include": [
    "https://example.com/docs",
    "https://example.com/docs/*"
  ]
}

Set SCRAIPE_API_KEY in your shell environment without committing it. Submit the saved request once:

export API='https://scraipe-fc66a552921e.herokuapp.com'
curl --fail-with-body --silent --show-error \
  "$API/v1/crawl" \
  -H "Authorization: Bearer $SCRAIPE_API_KEY" \
  -H 'Content-Type: application/json' \
  --data-binary @request.json > started.json

A successful submission returns HTTP 202 with ok: true, jobId and a poll path. It does not contain the finished pages. Inspect the response and copy its jobId into JOB_ID; keep the ID and request together. Repeating the POST starts another job.

export JOB_ID='job_REPLACE_WITH_RETURNED_ID'
curl --fail-with-body --silent --show-error \
  "$API/v1/jobs/$JOB_ID" \
  -H "Authorization: Bearer $SCRAIPE_API_KEY" > job.json

Inspect job.status. Poll the GET again after a delay while it is queued or running; for a pilot, use a five-second interval and a deliberate observation deadline. Reaching that deadline does not cancel the crawl. Investigate before submitting another one. The API also documents cancellation through DELETE, which removes the job, so save any evidence you need first. Finished jobs have limited retention: download the result before the seven-day expiry. See the API contract for job states and authentication.

Audit the saved job offline

Download the budget checker, its tests, the request fixture and the job fixture into one empty directory. Node.js 22 is sufficient; there are no dependencies to install and these commands make no network requests.

node --test audit-crawl-budget.test.mjs
node audit-crawl-budget.mjs crawl-budget-request.json crawl-budget-job.json

The 13 tests cover malformed budgets, request mismatches, unfinished jobs, result/depth overruns, duplicate and cross-origin URLs, missing content and error accounting. The fixture produces:

{
  "decision": "within_recorded_bounds",
  "reasons": [],
  "pagesCrawled": 2,
  "maxObservedDepth": 1,
  "urlsDiscovered": 4,
  "errorCount": 0,
  "limitReached": true,
  "completeness": "not_established",
  "originRequestCount": "not_measured"
}

Next run node audit-crawl-budget.mjs request.json job.json on your own saved files. Exit 0 means the recorded pages passed this bounded review; exit 2 means hold the result for inspection; exit 1 means the inputs are malformed or outside the checker's supported read/direct workflow.

The checker compares saved request fields with job.input, checks result counters, page count, recorded depths, URL origins, duplicate normalized URLs and nonempty content. A completed job with page errors is held. The crawler retains at most 50 error details, so the checker preserves errorCount even when it exceeds the sample size.

This is an after-the-fact check. It cannot stop requests already made, independently reconstruct traversal depth, verify that content is accurate, audit redirect destinations or re-evaluate include/exclude patterns. Inspect returned URLs against your intended section separately. It also does not verify billing or grant permission to crawl. An edited local JSON file is not authenticated evidence from the service.

Interpret limits and failures before scaling

A failed page gives its result slot back. Discovery can also fetch robots or sitemap information, and the fetcher can make more than one origin request for a page. Therefore neither limit nor pagesCrawled + errorCount is a reliable HTTP-request total. A depth bound does not place a wall-clock deadline on a job.

After a pilot passes review, increase one constraint at a time. Keep the request, job ID, returned URLs, errors and usage evidence for each run. Check coverage against your own inventory, and inspect representative page bodies before sending them to an agent or retrieval pipeline. The source-preservation guide explains what to retain downstream.

Keep the credit budget separate

This example explicitly uses mode: "read" and egress: "direct". Current Scraipe pricing makes direct Read pages free of model and residential credits. It does not make the job instantaneous or exempt it from access, capacity or origin restrictions.

Extraction can incur compilation credits; an explicitly requested residential path can incur network credits. This guide does not test residential availability. Do not change those options as an automatic response to a refused page. Check the current pricing and the job's credits together with its request receipts. A result limit is one input to a budget, not an exact cost quotation.

Start with the small direct pilot, inspect its saved result, then use the playground to explore individual pages before expanding the crawl.