Skip to content

Menu

Choose cache freshness for recurring workflows

Choose an observation-age limit, schedule cache-bypassed refreshes and hold stale results with a tested Node.js freshness policy checker.

By Codex. Checked 2026-11-17.

A recurring job needs an explicit answer to this question: how old may the last successful observation be when another system uses it? A five-minute schedule alone cannot answer it. The job may receive a cached response, replay an earlier operation, fail to refresh or spend several minutes processing its result.

Choose a maximum observation age, leave room for refresh work, and check the age again before downstream use. Scraipe’s current scrape API exposes noCache to bypass its response cache and forceCompile to rebuild an extractor. It does not expose a per-request cache TTL. This guide implements a caller-side freshness decision with a dependency-free Node 22 example. Current scrape contract

Define what freshness means for this workflow

Keep three clocks separate. Your observation clock records when your request began. HTTP caching has its own age and validation rules. The underlying business fact, such as stock availability, may have changed before either clock started. A newly fetched page can still contain an old price or a loading placeholder.

The policy here limits time since an accepted observation attempt began. It does not certify the age of the source’s business data or implement an HTTP cache. HTTP response age includes cache residence and transit time; see the primary HTTP caching specification, section 4.2.3. Establish separate source-specific checks when a decision requires a current business fact.

Write down these three values for each extraction definition. Keep separate histories when the URL, schema, credentials, country or rendering requirements differ.

Policy valueMeaningInvented example
Maximum observation ageDeadline after which downstream use must stop900 seconds
Poll intervalTime between policy evaluations300 seconds
Processing budgetAllowance for fetching, validating and committing a replacement60 seconds

These values illustrate a policy, rather than recommend a universal interval. Start with the consequence of using an old value. Select an age that the downstream task can tolerate, then measure refresh duration and scheduling delays on representative authorized pages. Include queueing and validation in the processing allowance. Respect the source’s access and rate limits.

Leave positive margin: the poll interval plus processing budget must be less than the maximum age. Reevaluate at every tick and before use. When the remaining lifetime is at most the interval plus budget, refresh now. This gives a healthy serial worker an opportunity to replace the observation before a later tick would be too late. Outages, scheduler delays and unexpectedly long work can still cause a hold.

Request a new observation without rebuilding working rules

For a fresh single-page observation, send a new scrape operation with the same extraction definition and noCache: true. The request body below uses an invented URL; replace it with a page you have permission to access.

json

{
  "url": "https://catalog.example/invented-stock",
  "fields": ["stock"],
  "noCache": true
}

Use a new idempotency key for each intended observation. Retain that key when retrying the same operation. If the response has X-Scraipe-Idempotent-Replay: true, it belongs to the original observation; receiving it again must not restart the freshness clock. Persist the request’s start time before sending it and keep the original request ID with the result. Receipt and replay guidance

The repository’s actual pipeline produced the following sequence on an invented stock page. Network fetching and compiler calls were replaced by deterministic test doubles. The source changed from Available to Sold out after the first request.

StepOptions or eventResultFetch callsCompiler calls
First observationDefault requestcompiled-fresh; Available11
Source changesDefault requestcache; Available00
Observe the changenoCache=truecompiled; Sold out10
Fetcher failsnoCache=trueerror; no new result10
Default request afterwardPrevious response still cachedcache; Sold out00

This is local pipeline evidence, not a hosted availability test or latency benchmark. It shows why a failed refresh must leave the old observation’s deadline unchanged. A subsequent cache hit can return that old result without establishing another observation.

noCache skips the result-cache read; a successful extraction can still populate the cache afterward. An existing working extractor can be reused. Rebuilding it with forceCompile is a separate action with a compile cost. A cold or broken extractor can also require compilation during a cache-bypassed request. For the underlying mechanisms and a runnable HTTP experiment, see extractor reuse and response caching.

Run the freshness policy checker

Save the checker, teaching timeline and sixteen tests into one directory. Use Node 22. The example reads local JSON only and makes no network requests or account changes.

bash

node --test check.test.mjs
node check.mjs scenario.json

The timeline is explicitly invented. It records a successful observation starting at 13:00 UTC, an extra refresh attempt that fails at 13:05, and a policy check at 13:10. The output includes:

json

{
  "decision": "refresh",
  "mayUse": true,
  "refreshNow": true,
  "requestId": "teaching-first",
  "expiresAt": "2026-11-17T13:15:00.000Z",
  "ageSeconds": 600,
  "remainingSeconds": 300
}

This is an excerpt of the full output. Five minutes remain, so the old observation can still be used while a refresh is attempted. The checker chooses refresh because 300 seconds is within the 360-second scheduling and processing allowance. The failed attempt did not advance the 13:15 deadline.

Change now in a copy to 2026-11-17T13:15:00.000Z. The result becomes hold, mayUse becomes false, and the program exits with status 1. Exact expiry is excluded. The old data may be retained for diagnostics, but this policy no longer permits downstream use.

Exit 0 means the observation is still usable; inspect refreshNow to distinguish use from refresh. Exit 1 means hold and attempt a replacement. Exit 2 means malformed or inconsistent input, with no decision report. For example, setting the interval to 840 seconds leaves no positive margin after the 60-second budget and produces exit 2. Treat invalid policy evidence as a hold in your caller.

Advance the clock only after acceptance

For your own timeline, replace the invented entries with accurately recorded attempts. Supply UTC timestamps with milliseconds, an explicit noCache boolean, the response’s ok, truncated and meta.tier fields, and the original request ID. Set replayed from the observed replay header. Set validated only after your own schema, identity, required-field and business-rule checks pass.

The example accepts a new anchor only for a non-replayed, successful, complete and validated cache-bypassed result with a known body-fetch tier: compiled, compiled-fresh, healed or llm. Failed requests, cached responses, unknown tiers and validation failures remain in the history without renewing the deadline. Missing explicit input fields cause an input error where the script cannot safely interpret the record.

This intentionally conservative policy also excludes conditional responses. HTTP 304 revalidation can be useful, but accepting it requires a separate policy for the identity and validators of the stored representation. The checker does not implement that policy. Likewise, split a crawl or batch into separately tracked resources; it does not assign one observation time to a mixed summary or truncated result.

The checker trusts the recorded metadata and your validation decision. It cannot verify a saved header’s authenticity, ensure every attempt used the same extraction definition, or inspect business meaning from a tier label. A successful tier alone is insufficient acceptance evidence. Use missing-field validation and source-specific checks before marking a result validated.

Attempts must be serial, chronological and complete by now. A start time after completion, an overlapping attempt, a future completion or a duplicate accepted request ID is rejected. Use a single writer per extraction definition or explicitly serialize your observation history before evaluating it. Keep clocks synchronized and reevaluate with the current time immediately before use.

Age starts at request start, so time spent fetching and validating consumes the allowance. A result that takes 16 minutes to finish is already expired under a 15-minute policy. processingBudgetExceeded flags a successful refresh whose observed duration exceeded the configured allowance. Investigate that signal instead of extending the deadline from completion time.

Budget for refresh attempts and failure recovery

More frequent observations create more fetch opportunities. They do not necessarily create more compilations: the tested changed-source request reused its existing extractor. Direct fetching and result caching have no residential charge, while paid network fetching and compiler work have their own costs. Check the actual receipt components for each attempt. Current pricing contract

Estimate a normal day and a recovery day using the number of intended observations, likely network modes, retries and repair events. Use the workload estimator to compare explicit assumptions, then reconcile them with real receipts. A local freshness budget is not an enforced account spending cap.

When replacement fails, keep the last accepted observation and its unchanged expiry, record the failure, and apply a bounded retry policy. At expiry, hold the downstream action or route it to an explicitly designed manual fallback. Resume only after a qualifying replacement passes validation. Track hold duration, replay responses and refresh duration alongside cost so that a lower request count does not conceal missing or outdated observations.

Sources and verification

Try the next step

Try the playground

Continue building