Choose cache freshness for recurring workflows
Choose an observation-age limit, schedule cache-bypassed refreshes and hold stale results with a tested Node.js freshness policy checker.
By Codex. Checked 2026-11-17.
A recurring job needs an explicit answer to this question: how old may the last successful observation be when another system uses it? A five-minute schedule alone cannot answer it. The job may receive a cached response, replay an earlier operation, fail to refresh or spend several minutes processing its result.
Choose a maximum observation age, leave room for refresh work, and check the age again before downstream use. Scraipe’s current scrape API exposes noCache to bypass its response cache and forceCompile to rebuild an extractor. It does not expose a per-request cache TTL. This guide implements a caller-side freshness decision with a dependency-free Node 22 example. Current scrape contract
Define what freshness means for this workflow
Keep three clocks separate. Your observation clock records when your request began. HTTP caching has its own age and validation rules. The underlying business fact, such as stock availability, may have changed before either clock started. A newly fetched page can still contain an old price or a loading placeholder.
The policy here limits time since an accepted observation attempt began. It does not certify the age of the source’s business data or implement an HTTP cache. HTTP response age includes cache residence and transit time; see the primary HTTP caching specification, section 4.2.3. Establish separate source-specific checks when a decision requires a current business fact.
Write down these three values for each extraction definition. Keep separate histories when the URL, schema, credentials, country or rendering requirements differ.
| Policy value | Meaning | Invented example |
|---|---|---|
| Maximum observation age | Deadline after which downstream use must stop | 900 seconds |
| Poll interval | Time between policy evaluations | 300 seconds |
| Processing budget | Allowance for fetching, validating and committing a replacement | 60 seconds |
These values illustrate a policy, rather than recommend a universal interval. Start with the consequence of using an old value. Select an age that the downstream task can tolerate, then measure refresh duration and scheduling delays on representative authorized pages. Include queueing and validation in the processing allowance. Respect the source’s access and rate limits.
Leave positive margin: the poll interval plus processing budget must be less than the maximum age. Reevaluate at every tick and before use. When the remaining lifetime is at most the interval plus budget, refresh now. This gives a healthy serial worker an opportunity to replace the observation before a later tick would be too late. Outages, scheduler delays and unexpectedly long work can still cause a hold.
Request a new observation without rebuilding working rules
For a fresh single-page observation, send a new scrape operation with the same extraction definition and noCache: true. The request body below uses an invented URL; replace it with a page you have permission to access.
json
{
"url": "https://catalog.example/invented-stock",
"fields": ["stock"],
"noCache": true
}Use a new idempotency key for each intended observation. Retain that key when retrying the same operation. If the response has X-Scraipe-Idempotent-Replay: true, it belongs to the original observation; receiving it again must not restart the freshness clock. Persist the request’s start time before sending it and keep the original request ID with the result. Receipt and replay guidance
The repository’s actual pipeline produced the following sequence on an invented stock page. Network fetching and compiler calls were replaced by deterministic test doubles. The source changed from Available to Sold out after the first request.
| Step | Options or event | Result | Fetch calls | Compiler calls |
|---|---|---|---|---|
| First observation | Default request | compiled-fresh; Available | 1 | 1 |
| Source changes | Default request | cache; Available | 0 | 0 |
| Observe the change | noCache=true | compiled; Sold out | 1 | 0 |
| Fetcher fails | noCache=true | error; no new result | 1 | 0 |
| Default request afterward | Previous response still cached | cache; Sold out | 0 | 0 |
This is local pipeline evidence, not a hosted availability test or latency benchmark. It shows why a failed refresh must leave the old observation’s deadline unchanged. A subsequent cache hit can return that old result without establishing another observation.
noCache skips the result-cache read; a successful extraction can still populate the cache afterward. An existing working extractor can be reused. Rebuilding it with forceCompile is a separate action with a compile cost. A cold or broken extractor can also require compilation during a cache-bypassed request. For the underlying mechanisms and a runnable HTTP experiment, see extractor reuse and response caching.
Run the freshness policy checker
Save the checker, teaching timeline and sixteen tests into one directory. Use Node 22. The example reads local JSON only and makes no network requests or account changes.
bash
node --test check.test.mjs
node check.mjs scenario.jsonThe timeline is explicitly invented. It records a successful observation starting at 13:00 UTC, an extra refresh attempt that fails at 13:05, and a policy check at 13:10. The output includes:
json
{
"decision": "refresh",
"mayUse": true,
"refreshNow": true,
"requestId": "teaching-first",
"expiresAt": "2026-11-17T13:15:00.000Z",
"ageSeconds": 600,
"remainingSeconds": 300
}This is an excerpt of the full output. Five minutes remain, so the old observation can still be used while a refresh is attempted. The checker chooses refresh because 300 seconds is within the 360-second scheduling and processing allowance. The failed attempt did not advance the 13:15 deadline.
Change now in a copy to 2026-11-17T13:15:00.000Z. The result becomes hold, mayUse becomes false, and the program exits with status 1. Exact expiry is excluded. The old data may be retained for diagnostics, but this policy no longer permits downstream use.
Exit 0 means the observation is still usable; inspect refreshNow to distinguish use from refresh. Exit 1 means hold and attempt a replacement. Exit 2 means malformed or inconsistent input, with no decision report. For example, setting the interval to 840 seconds leaves no positive margin after the 60-second budget and produces exit 2. Treat invalid policy evidence as a hold in your caller.
Advance the clock only after acceptance
For your own timeline, replace the invented entries with accurately recorded attempts. Supply UTC timestamps with milliseconds, an explicit noCache boolean, the response’s ok, truncated and meta.tier fields, and the original request ID. Set replayed from the observed replay header. Set validated only after your own schema, identity, required-field and business-rule checks pass.
The example accepts a new anchor only for a non-replayed, successful, complete and validated cache-bypassed result with a known body-fetch tier: compiled, compiled-fresh, healed or llm. Failed requests, cached responses, unknown tiers and validation failures remain in the history without renewing the deadline. Missing explicit input fields cause an input error where the script cannot safely interpret the record.
This intentionally conservative policy also excludes conditional responses. HTTP 304 revalidation can be useful, but accepting it requires a separate policy for the identity and validators of the stored representation. The checker does not implement that policy. Likewise, split a crawl or batch into separately tracked resources; it does not assign one observation time to a mixed summary or truncated result.
The checker trusts the recorded metadata and your validation decision. It cannot verify a saved header’s authenticity, ensure every attempt used the same extraction definition, or inspect business meaning from a tier label. A successful tier alone is insufficient acceptance evidence. Use missing-field validation and source-specific checks before marking a result validated.
Attempts must be serial, chronological and complete by now. A start time after completion, an overlapping attempt, a future completion or a duplicate accepted request ID is rejected. Use a single writer per extraction definition or explicitly serialize your observation history before evaluating it. Keep clocks synchronized and reevaluate with the current time immediately before use.
Age starts at request start, so time spent fetching and validating consumes the allowance. A result that takes 16 minutes to finish is already expired under a 15-minute policy. processingBudgetExceeded flags a successful refresh whose observed duration exceeded the configured allowance. Investigate that signal instead of extending the deadline from completion time.
Budget for refresh attempts and failure recovery
More frequent observations create more fetch opportunities. They do not necessarily create more compilations: the tested changed-source request reused its existing extractor. Direct fetching and result caching have no residential charge, while paid network fetching and compiler work have their own costs. Check the actual receipt components for each attempt. Current pricing contract
Estimate a normal day and a recovery day using the number of intended observations, likely network modes, retries and repair events. Use the workload estimator to compare explicit assumptions, then reconcile them with real receipts. A local freshness budget is not an enforced account spending cap.
When replacement fails, keep the last accepted observation and its unchanged expiry, record the failure, and apply a bounded retry policy. At expiry, hold the downstream action or route it to an explicitly designed manual fallback. Resume only after a qualifying replacement passes validation. Track hold duration, replay responses and refresh duration alongside cost so that a lower request count does not conceal missing or outdated observations.