Handle partial crawl failures
Review page errors in a completed Scraipe crawl, reconcile missing URLs against an inventory and stage successful pages with a tested offline Node.js example.
By Codex. Checked 2026-11-10.
A crawl of five expected documentation pages can finish with two pages of Markdown and three page errors. Those counts add up to five, but they tell you little about which documents you can use. One of the two returned pages might be a glossary discovered along the way, while a required FAQ is missing from both lists.
Save the complete job response and compare its URLs with the inventory you intended to collect. Keep the successful pages available for content review, retain the error evidence and decide what to do with each unresolved URL before updating your index.
Read the job and its page outcomes
A successful crawl submission returns HTTP 202, a job ID and a polling path. Save the ID beside the original request. Poll that job with the same account and a key with read access. Repeating the submission starts another crawl. When the job is queued or running, keep its ID and poll again after a delay. If it failed, was cancelled or was interrupted, preserve the job record and inspect its status before submitting more work. Finished jobs have a seven-day retention window. See the Scraipe API reference.
For a completed job, inspect these fields together.
| Field | Decision it supports |
|---|---|
job.result.pages and pagesCrawled | Identify the returned pages and check the count against the array length. Inspect their actual content. |
job.result.errors and errorCount | Match visible failures by URL and reason. Subtract the number of visible rows from the total to count failures with missing detail. |
job.result.urlsDiscovered | Understand how many candidates entered discovery. The result cap, filters and page failures affect how many become usable records. |
Per-page requestId, when present | Find the corresponding receipt when investigating an outcome or reconciling usage. |
job.credits | Check the total charge recorded for the job, including any successful paid work. |
The current crawler keeps up to 50 error rows and counts every page error. We exercised the repository crawler locally with an invented sitemap, an in-memory fetcher and a reader fixture. Of 54 discovered URLs, one page succeeded and 53 failed. The result contained 50 error rows. Three failure identities were unavailable from that array. A required URL absent from both arrays could be among those three or could have gone unprocessed. Preserve that uncertainty.
Reconcile the intended URLs
Before submitting a crawl, save the exact start URL, filter settings, page limit and the list of URLs that matter to your application. Attach the returned job ID to that record. A product catalogue requiring every current price needs a different release decision from a documentation search index that can keep its last reviewed copy of a missing page. Use the Map guide to build an initial inventory and the crawl limits guide to choose the pilot size.
The fixture below expects five URLs under the same origin. It illustrates why comparing just the counts can hide a gap.
| Expected URL suffix | Saved evidence | Next step |
|---|---|---|
/docs/intro | Returned Markdown and a receipt ID | Check that the body contains the intended document, then stage that observation |
/docs/api | A page error with reason timeout | Inspect the receipt and prior attempts before considering one bounded follow-up |
/docs/archive | A page error with reason robots_disallowed | Stop fetching this URL under the current policy |
/docs/limits | A page error with reason rate_limited | Check the available wait instruction and defer follow-up |
/docs/faq | Absent from the page rows and visible errors | Record it as unaccounted for and inspect the seed, filters and discovery limits |
The second returned page is /docs/glossary. It belongs in the report as an additional result. It cannot fill the FAQ slot. Preserve URL spellings and query strings through this comparison. If a slash or redirect alias accounts for a mismatch, document the verified mapping before merging the records.
Run the partial-crawl reviewer
Download these six files to one empty directory. The plan and job are synthetic teaching fixtures. Node 22 can run the reviewer offline using the files directly.
bash
node --test review-partial-crawl.test.mjs
status=0
node review-partial-crawl.mjs partial-crawl-plan.json partial-crawl-job.json > review.json || status=$?
printf 'Reviewer exit: %s\n' "$status"
node -p 'require("./review.json").expectedRows.map(row => row.outcome)'Expect exit 2 and five outcomes in order: review_content, failed, failed, failed, unaccounted. Open additionalRows to see the glossary. The fixture reports three errors, all with visible detail. The test suite also covers 53 failures with 50 visible details, an unknown reason, a cancelled run, an inconsistent count, empty Markdown and a query variant.
For your own pilot, replace both files with a plan and a full saved poll from the same job. Use exact absolute URLs on the seed origin. Each input file can be up to 1 MiB. Exit 2 produces a hold report. Exit 0 means the planned rows are ready for content review. Exit 1 means the input is malformed, inconsistent or outside this example's supported scope. To exercise exit 1, change the fixture's errorCount to 4 while keeping its three error rows. Keep the source job intact and make this change in a copy.
The helper supports read-mode Markdown and checks job identity, counts, origin and duplicate outcome rows. A nonempty body still needs a document-quality check. A branded error page can contain plenty of text. For extraction jobs, validate the returned fields and schema before accepting an individual row.
Decide which failures merit a follow-up
Branch on the recorded reason. A timeout or network error can merit a later attempt after you check the receipt, confirm permission and set a per-URL attempt budget. A robots refusal or site opt-out ends the attempt under the current access policy. Preserve an unfamiliar reason for investigation. The example writes a review report and makes zero network requests.
For rate limits, use the wait information actually available to the caller. An HTTP Retry-After header can give delay seconds or an HTTP date. A header on the poll response applies to that request. The saved crawl error may contain only the URL, reason, message and receipt information. A missing per-page delay leaves the timing unknown. Consult the receipt and your origin policy before scheduling another attempt. See RFC 9110, section 10.2.3.
If one permitted document needs a fresh read, save its original job and receipt IDs alongside the new observation. A whole-crawl rerun can revisit successful pages and change the set discovered. Use a new crawl when the scope itself needs another traversal. Direct Read conversion is free under the current contract. Extraction or paid networking may incur charges on successful pages. Compare the job's credits and its request receipts before widening the retry budget.
Update the index after review
For an index that tolerates partial updates, stage each accepted page with its source URL, job ID and receipt. Keep the last reviewed version of an unresolved document and record when it was last seen. For a snapshot that requires complete coverage, hold the proposed version until each required URL has met your acceptance rule. Keep a separate list of additional URLs for later scope decisions.
The strict importer in the crawl deduplication guide holds captures containing page errors. Preserve that evidence when preparing a separately reviewed partial update. Clearing error counts to feed a partial capture into a clean-job importer would lose the reason the snapshot was held.