Handle layout changes in an extraction workflow

Keep the last good result while you investigate

A layout change should start an investigation, not silently replace trusted records with empty or misselected data. Keep the last accepted output with its original observation time, quarantine the new capture, and determine whether the problem is source access, changed markup, changed selection or changed business data. Old data is still old: preserving it does not make it current.

This guide provides a local capture-triage tool and fourteen tests. It separates unusable source captures from extraction failures and structural changes. Its strongest possible result is continue-checks, never automatic promotion. The example captures are invented and the tool is independent of Scraipe's hosted extractor; it does not contact a site or repair rules for you.

Check the source before changing selectors

Save the failed operation's request ID, requested URL, final URL, response status, content type, representation, options and sanitized output. If permitted, retain the HTML actually supplied to extraction. Match the session, locale, egress country and pagination window to the baseline. A login page, bot challenge, truncated response or client-rendered shell can look like a new layout while describing a different problem.

A successful HTTP status is only one input. A challenge can return 200. The downloadable tool requires caller-supplied sourceState: "content" and complete: true; it cannot establish either assertion itself. Use these fields only after your capture process has checked them. If the evidence is unknown, keep the candidate on hold.

Run the capture-triage example

Download these six files into a fresh directory:

• Triage module and CLI
• Fourteen tests
• Invented before-and-after captures
• Package manifest
• Dependency lockfile
• SHA256SUMS

Use Node 22. Cheerio 1.2.0 is pinned. Installation contacts the npm registry; execution uses only local files and needs no API key or extraction credits.

npm ci --ignore-scripts --no-audit --no-fund
npm test
node triage-layout.mjs capture-pair.json

The tests pass. The last command deliberately exits 1: the fixture changes a product field from class="title" to class="name", while the old extractor still requires one direct child title. Its report includes this decision and the baseline digests:

{
  "action": "review-extraction",
  "reason": "candidate-contract",
  "promote": false
}

Exit 0 means continue-checks, exit 1 means hold or review, and exit 2 means invalid CLI input. A review result is a successful diagnosis of the supplied failure, not a broken installation. Verify the five payload files on macOS with shasum -a 256 -c SHA256SUMS.

The input has baseline and candidate objects, each containing status, contentType, complete, sourceState, finalUrl and html. These are the tool's capture fields, not a Scraipe response schema. Keep credentials and private source data out of shared examples. The CLI reads at most a 2 MiB file; each HTML body is limited to 1 MiB.

Interpret the evidence without overpromising

The toy extraction contract requires exactly one main > .catalog, one or more direct .product children, unique SKU attributes and exactly one nonempty direct .title per product. It collapses title whitespace. The Cheerio selector reference explains the distinction between direct children and descendants.

For usable captures, the tool records separate digests for the full HTML, the selected catalog structure and the extracted rows. The structure includes tag names, sorted class tokens and element children inside the catalog; it excludes text and other attributes. It is this tool's fingerprint, not Scraipe's skeleton algorithm. Traversal stops beyond forty levels or 20,000 elements and requests review rather than accepting an incomplete signature.

A cosmetic class can change the signature while extraction remains correct. A changed SKU can alter the data without changing the signature. A notice outside the catalog is invisible to that structural signature even though the full HTML digest changes. The tests demonstrate all three cases. No hash proves freshness, completeness or factual correctness.

The tool conservatively holds all empty candidates for review. Your application may accept a genuinely empty catalog after independent evidence; this tool does not make that decision. It also does not render JavaScript, discover pagination, verify source metadata, compare expected business values or detect all renamed records. Use the held-out evaluation pack to check every expected identity and value, and the nested-list guide when child records can disappear silently.

Repair one version and test both layouts

Keep four artifacts: the old rules, the failed capture, the proposed rules and the evaluation report. Give the proposed rules a new revision. Add the failed page to the regression set, then test the repair against old supported layouts and untouched evaluation pages. A selector that fixes one wrapper but mixes recommendations into parent records has moved the error elsewhere.

Use independent expected identities and field values. Compare record counts, missing and extra IDs, duplicates, required fields and relevant cross-field constraints. Run full-result validation; a few valid leading rows cannot establish that all records survived. If you use a failed evaluation page to tune the repair, it becomes regression evidence rather than fresh held-out evidence.

Promote the candidate only after the declared consumer checks pass and an authorized release decision is recorded. Keep the previous version available. In storage, write candidate output separately and switch the accepted revision only after validation; do not let a failed job overwrite the accepted snapshot. Decide how consumers see stale data: retain its timestamp and stale status, or stop serving it if freshness is mandatory. This article describes that workflow; the downloadable diagnostic does not implement a database or a release service.

What Scraipe can report during a layout change

The current implementation tries existing layout variants and validates their output before deciding that compilation or healing is needed. A changed structural fingerprint can be accepted with an existing working extractor and reported as skeletonRebaselined on the compiled path. A new fingerprint is therefore not, by itself, evidence that a new model call occurred. The sample-based runtime safeguards still do not replace your own whole-result checks.

Read meta.tier, the layout variant, repair information when present, request/replay context and returned charges. A healed result identifies a repair path; it is not independent proof that every returned business value is right. A known empty layout can also be a legitimate result. Review the current API contract for the response and error fields rather than inferring execution from latency or zero tokens.

For a deliberately new diagnostic operation, noCache: true bypasses the stored response but does not prohibit compilation or healing. forceCompile: true asks for rebuilding and may incur a compile charge. It is not a universal fix for access failures or missing client-rendered content. See extractor reuse versus response caching before interpreting a repeat request as a new source observation.

Bound repair attempts and spend. Treat heal_limit, layout_limit and source refusals according to the returned reason and retry guidance; do not keep changing keys to evade limits. Keep compilation and egress charges separate when assessing cost. No authenticated hosted extraction, repair or billing experiment was performed for this guide: the observed failure cases are local fixtures, and the product-path description is based on the current contract and source review.

Start with one failing capture and one known baseline. Diagnose access and representation first, preserve the old evidence, evaluate a versioned repair, and then use a small permitted playground request if hosted confirmation is needed. That sequence makes a layout change reviewable without confusing a repaired selector with verified production data.