Deduplicate crawl results

Two documentation URLs can return the same Markdown today and change independently tomorrow. If you keep only the first URL, an agent can lose the source it should cite. Store each distinct Markdown body once and keep an observation for every returned page. Each observation points from the job and URL to the body seen on that run.

Choose what counts as a duplicate

Consider two crawls of a documentation site. On the first run, the setup and quickstart pages contain identical instructions. On the next run, only the setup page gains a configuration step. The importer needs three different decisions.

Scraipe already handles some duplicates during traversal. Its crawler strips fragments and removes one trailing slash from non-root paths before adding a URL to a per-crawl set. Two different admitted URLs can still return identical content. The result's page URL identifies the queued address. Keep it as supplied, along with any page receipt ID and the job ID. A verified redirect destination or publisher-declared canonical URL requires its own evidence.

Group exact Markdown and retain every source

For each accepted page, encode its returned content string as UTF-8 and calculate SHA-256. Use that digest as the lookup key for the stored body. When the key already exists, compare the stored text with the incoming text before reusing it. Keep a separate row for the observation with the source URL, body digest, job ID, job finish time, crawl depth and request ID when available.

Preserve line endings, case, punctuation and whitespace in the string you hash. A trailing space can matter to Markdown rendering. A raw HTML page and its converted Markdown have different identities. Record the representation when designing persistent storage. The example below accepts read-mode Markdown jobs and rejects malformed Unicode strings before encoding them. Node documents the hashing operation in its crypto reference.

Two pages with identical Markdown may have different titles, licences, permissions or relevance to a query. Keep the original job responses for any metadata your retrieval system needs. Use the shared body only within a corpus whose access policy you control. When serving a citation, select an observation the reader is allowed to see. A content digest alone carries no source or access information.

Run the two-crawl example

Use Node 22 and download these six files into one empty directory. The two snapshots are invented, with the shape of completed crawl job responses. The offline run requires no API key, packages or network calls.

• Importer
• Seven tests
• First saved job
• Second saved job
• Bounded JSON loader
• Loader dependency

node --test deduplicate-crawl-results.test.mjs
node deduplicate-crawl-results.mjs dedup-job-a.json dedup-job-b.json dedup-job-a.json > corpus.json
node -p 'JSON.stringify(require("./corpus.json").summary)'

The third argument deliberately replays the first capture. The result contains two distinct jobs, one replay, six observations and three distinct bodies.

{"jobs":2,"replays":1,"observations":6,"distinctBodies":3}

Open corpus.json and follow the body hashes for the two observations whose URL ends in /setup. They lead to different instructions. Both quickstart observations still lead to the original text. The billing observations share a body, and the later observation retains its ?edition=2 query string. An agent can use the text once while retaining all four source addresses across the two jobs.

To exercise the error path, make a copy of the first fixture, edit one page's content and leave its job ID unchanged. Pass the original and edited file together. The command exits 1 and reports a conflicting snapshot. Save a successful result to a new file before replacing an existing corpus. Shell output redirection opens its destination even if the command later fails.

Import your own finished jobs

Save the full GET /v1/jobs/:id responses from your own bounded read-mode crawls. A successful submission gives you a job ID and polling path. Wait for completion before importing. Save the snapshots within the documented seven-day job retention period, and keep the receipts if you need to reconcile usage later. See the crawl and jobs contract and the guide to crawl depth and page count.

Run the importer with your saved filenames in place of the fixtures. Each file can be up to 1 MiB. The importer holds jobs with page errors, empty or inconsistent results, unsupported representations, blank content or ambiguous repeated URLs within one job. Inspect the original response to decide whether particular surviving pages are appropriate for a separate partial import. The example keeps the full corpus in memory and rebuilds it from the supplied captures on each invocation.

For a persistent importer, use unique constraints for the body digest and the observation identity within your corpus. Commit the job record and its observations together after validating the complete capture. This allows a retry after an interrupted write to preserve the same observation count. The runnable example compares repeated job objects by value and stops if even their metadata differs. Retain both source files for investigation in that case.

Decide which work can be reused

Exact matching can save downstream processing when its inputs match. If embeddings depend on headings, the source URL or another field outside the body, include those inputs in the reuse key. Also record the chunking policy and embedding model version. Otherwise a pipeline can silently reuse an old vector after a model or chunk-size change.

A digest changes when a timestamp, navigation block or single space changes. Review the reader output and the content you need before introducing a second normalized or near-duplicate representation. Keep the original body digest and citations so you can inspect what the normalization merged. The source-URL guide shows how to replay a citation against a saved observation.

This deduplication happens after fetching. The fixture's three unique bodies measure its own storage inputs. For a real job, measure returned pages, unique bodies, retained provenance and actual downstream work separately. Use the request receipts to account for the original crawl. Direct Read conversion is free under the current API contract, while other modes and network choices have their own costs.