Distinguish extractor reuse from response caching

Reuse the extraction logic, not yesterday's answer

Extractor reuse and response caching save different work. A reusable extractor applies saved selection and transformation rules to a fetched page. A response cache returns previously extracted data without running those rules on a newly fetched body. The returned JSON can be identical in both cases, so matching values alone do not tell you which happened.

In Scraipe, inspect the reported meta.tier, request options and replay context together. Use noCache: true when a new diagnostic operation should bypass the stored response while still allowing extractor reuse. Use forceCompile: true only when you intend to rebuild the extraction logic. Neither flag proves that the source's business data is current or correct.

This guide includes an eight-scenario local pipeline experiment and a downloadable comparison tool. The experiment ran the repository's actual pipeline, store and selector interpreter on invented HTML, with network and model calls replaced by test doubles. It is not a hosted API run, customer incident or speed benchmark. The tool and examples were tested with Node 22.23.2 on 30 October 2026.

Identify which work was reused

For a successful single-page scrape, these current tier names describe different paths:

A mixed batch or crawl summary needs per-page analysis. Do not assign it one of the single-page meanings. Unknown tier names should remain unknown until you check the current API reference.

An HTTP 304 is also distinct from a cache hit that made no origin request. A conditional request asks the origin to validate its stored representation using a validator such as an ETag. A 304 allows reuse without transferring a new response body. It is not evidence that the source's prices, inventory or statements reflect external reality. The HTTP 304 reference explains that protocol behavior.

Observe changed data without recompiling

The local experiment starts with one invented product whose title is Invented edition A. It then changes the source HTML to Invented edition B while preserving the element structure. The schema requires one string field, and the test compiler returns a deterministic selector for that field.

These are observed counts from the local test doubles, not production request-count or latency guarantees. Real fetching can involve retries, redirects or rendering. Compilation tokens in the downloadable observations are synthetic values supplied by the stub; no model was contacted and no account was charged.

Step 3 is the central distinction: the data changed without a new compiler call. Step 4 also shows that noCache is not a “never store this result” policy. The current pipeline skips the cache read, runs extraction and may populate the result cache afterward. Step 8 shows that bypassing the response cache does not promise a free compilation: a missing or unusable extractor can still require model work.

The experiment deliberately expires its local entry to exercise the 304 path. It does not measure the hosted cache lifetime. The repository has a configurable default lifetime, but this guide does not assume that configuration is the same in every deployment.

Run the saved-response comparison

Download these five files into one directory:

• Comparison module and CLI
• Runnable test suite
• Captured local before/after pair
• All eight local observations and their provenance
• SHA256SUMS

Use Node 22 or later. There are no npm dependencies, API keys or network calls in the comparison:

node --test compare-extraction-pair.test.mjs
node compare-extraction-pair.mjs reuse-cache.fixture.json

The first command passes sixteen tests using Node's built-in test runner. The second audits the saved outputs from steps 2 and 3; it does not rerun the Scraipe pipeline. Its full report includes data digests and these conclusions:

{
  "status": "comparable",
  "before": { "reportedTier": "cache", "reportedPath": "stored-response" },
  "after": { "reportedTier": "compiled", "reportedPath": "extractor-reuse" },
  "dataChanged": true,
  "freshness": "not-established",
  "issues": []
}

The displayed JSON is an excerpt. The complete output also keeps replay status, token observations and origin-request observations separate. With no HTTP replay header captured, replay remains null and pathScope explicitly says that it is interpreting the body claim.

The checksum file covers the four payload files. On macOS, run shasum -a 256 -c SHA256SUMS after downloading them. The observations file records the invented inputs, request flags, expected outcomes and actual local outcomes, so the experiment's boundaries remain visible beside its results.

Compare your own captures without guessing freshness

The comparison accepts a JSON object containing before and after. Each has a body with the saved response and an optional headers object containing saved response headers as strings:

{
  "before": {
    "headers": {},
    "body": {
      "ok": true,
      "data": [{ "title": "Earlier observed title" }],
      "truncated": false,
      "cursor": null,
      "meta": { "tier": "cache" }
    }
  },
  "after": {
    "headers": {},
    "body": {
      "ok": true,
      "data": [{ "title": "Later observed title" }],
      "truncated": false,
      "cursor": null,
      "meta": { "tier": "compiled" }
    }
  }
}

These are the tool's wrapper keys, not a Scraipe request body. Before comparing real captures, check that URL, requested fields/schema, source credentials, country and intended record window are comparable. The tool does not validate that request identity. A complete response for one page also does not prove complete website coverage.

The tool compares the data values after sorting JSON object keys; it preserves array order and distinguishes null from an absent property. Equal digests mean equal normalized captured values, not the same extractor, source revision or execution path. Different values may result from a changed page, a changed query, changed selection or errors. The tool deliberately never declares either response fresh or correct.

It reports needs-review for errors, missing data, unknown/aggregate tiers, malformed replay headers and truncated or unspecified result windows. It does not turn those conditions into successful empty results. Exit codes are 0 for a structurally comparable pair, 1 when the pair needs review, and 2 for invalid arguments or unreadable/invalid input. The file limit is 1 MiB; data nested beyond forty levels is rejected. This is a small-capture diagnostic, not a streaming validator or complete API-schema checker.

The report does not echo extracted values. Treat the capture and its digests as potentially sensitive anyway: use only necessary response fields, keep private data out of shared evidence, and never include request Authorization headers, cookies or API keys.

Separate diagnostic requests from idempotent retries

A successful idempotent replay returns the original body. If its original tier was compiled, seeing that tier again does not prove that the retry fetched another page or ran the extractor again. The comparison recognizes X-Scraipe-Idempotent-Replay: true, reports responsePath: "idempotent-replay", and preserves the original reported tier separately. Missing replay context remains unknown.

Use the same idempotency key and unchanged body to recover one original operation. To investigate a newly changed source or different cache option, create a deliberately new operation with a new key. Changing the request body while reusing the old key can conflict. Retrying the old successful operation cannot establish newer source data. See the request-receipt guide for the current replay and charge-accounting details.

Choose the smallest necessary refresh

Start with the receipt and your actual consumer requirement. If the values look old but the selector logic still fits, a separate noCache: true operation can bypass the stored response while retaining a useful extractor. If fields are misselected, reproduce the mismatch, validate the source and investigate repair or explicit recompilation. Do not repeatedly force compilation merely because unchanged data feels suspicious.

The result cache and extractor registry also have different identities. The current result key includes URL, structural schema identity and an explicitly selected egress country; reusable extractors are located by host, schema and compatible layout variants. That is why one site's compatible pages can share extraction rules while returning different data. Do not assume arbitrary pages or schemas share one extractor. Caller-supplied source credentials bypass the shared response cache; a Scraipe API authentication key is a different credential.

Read meta.credits.compile, meta.credits.egress and meta.credits.total before describing cost. Reusing an extractor can avoid a compile credit while a residential fetch still costs network credits. A new compilation or repair can be billable. Zero tokens, fast responses and equal data are each insufficient evidence for a cache hit or a zero total charge.

Finally, apply your own full-result schema validation and source checks. Use the playground for a small permitted investigation, retain the sanitized receipt and stop on unexplained differences. The local experiment proves specific pipeline behavior under controlled inputs; it does not replace verification of your hosted request, page completeness or business facts.