Test an extractor on held-out pages

Passing the development page is only the starting point

Test an extractor against pages that did not influence its selectors, prompts or repair rules. Keep those pages out of the development loop, freeze the candidate, and compare every extracted record with independently labeled expected values. A valid JSON shape is necessary, but it cannot tell you that the sixth product disappeared or that a recommendation was mistaken for a catalog item.

This guide supplies a runnable local evaluation: two small HTML extractors, two development fixtures, six evaluation fixtures and a sixteen-test runner suite. Both extractors pass the development fixtures. On the six evaluation fixtures, the broad selector passes three and the scoped selector passes four. Both are rejected by the example's all-cases-must-pass rule. A higher count does not erase a known failure.

The HTML and labels are invented teaching data, authored together with the extractors. The files demonstrate a held-out workflow; they are not a blind independent study, a Scraipe accuracy benchmark or evidence about customer websites. The runner performs real local HTML extraction with Cheerio. It does not call Scraipe, a model or a target website.

Define the record before collecting the split

The example's contract is deliberately small: a catalog contains records with a unique sku and a nonempty title. Expected records refer only to catalog products. Promotions and recommendations are excluded. HTML entities are decoded and title whitespace is collapsed. Record order is ignored because comparison uses SKU identity; title strings otherwise need an exact match. If your consumer needs ranking order, that is a different acceptance rule and must be tested separately.

For a real project, record the request URL, capture time, relevant locale/session, source representation and expected record boundary. Use pages you are permitted to capture. Label expected values from the source independently of the candidate output. In particular, do not accept a candidate's returned IDs as the complete list of expected IDs: that makes omissions invisible.

Separate the roles before tuning:

Choose a small, useful matrix before chasing a percentage: ordinary pages, genuine empty lists, more than five records, optional content, nested recommendations, excluded promotions and known layout families. Keep near-duplicate captures of the same item or template revision together when you split. An exact byte-hash check catches identical files, but cannot establish that two different files are independent. A tiny hand-selected corpus also cannot support a population-wide accuracy estimate.

Freeze the candidate and run the pack

Download these six payload files and their checksum list into one new directory:

• Extractor and evaluation CLI
• Sixteen runner tests
• Invented HTML corpus and explicit expected records
• Pinned package manifest
• Portable dependency lockfile
• Recorded example results
• SHA256SUMS

Use Node 22. The dependency is pinned to Cheerio 1.2.0; the lockfile pins its dependency tree. Installation contacts the npm registry, but running the example is local and requires no API key or extraction credits. From that directory:

npm ci --ignore-scripts --no-audit --no-fund
npm test
node evaluate-extractor.mjs corpus.json --candidate scoped --split development
node evaluate-extractor.mjs corpus.json --candidate scoped --split heldout

The development command exits 0. The held-out command deliberately exits 1, with this summary in its JSON report:

{
  "candidate": "scoped",
  "split": "heldout",
  "total": 6,
  "passed": 4,
  "failed": 2,
  "decision": "reject-on-selected-fixtures"
}

That is a successful reproduction of a rejected candidate, not an installation failure. The test suite itself passes because it checks that the runner identifies the known failures correctly. Do not turn the evaluator's exit code into a success to make a CI badge green. Exit 2 means invalid arguments, unreadable input or an invalid corpus. Successful fixture evaluation says passes-selected-fixtures; it never says production-ready.

For the alternative extractor, run:

node evaluate-extractor.mjs corpus.json --candidate broad --split heldout

It exits 1 with three passing and three failing cases. The full reports include per-case issue codes and HTML digests, plus the exact runner and corpus SHA256. The saved observations include the runtime version and these results, not a fabricated hosted receipt. On macOS, verify the six payloads with shasum -a 256 -c SHA256SUMS.

Before a real evaluation, also record the lockfile hash, environment, expected-data revision, schema, normalization policy and acceptance rule. Hashes show which bytes were evaluated; they do not prove that the candidate author never saw the answers. This example stores both splits in one visible file for convenience. For a blind evaluation, give a separate evaluator control of the held-out inputs and labels.

Read the failures before choosing an extractor

The broad extractor selects .product anywhere in the document and searches descendants for .title. The scoped candidate selects direct catalog products and direct child titles. Cheerio's selector documentation explains how descendant selectors differ from direct-child selectors. Neither strategy is universally correct.

For the wrapped title, the scoped extractor cannot find exactly one direct title and reports extraction-error. For the renamed sixth record, the selected five rows look valid, but expected identity L6 is absent. The runner reports missing-record. It also checks unexpected records, duplicate IDs and incorrect title values, so schema-shaped but wrong output does not pass.

Every expected row is compared; the tool does not sample the first few. A missing catalog container is an extraction failure, while the explicitly labeled empty catalog can pass. However, an empty catalog shell with unseen client-rendered products would also look empty to this parser. The independent expected labels must expose that problem. The runner does not execute JavaScript, fetch pagination, inspect screenshots or discover missing pages.

The corpus validator rejects duplicate case IDs, duplicate exact HTML, malformed expected rows and a missing split. CLI input is limited to 1 MiB, with at most 100 cases and 1,000 expected rows per case. These bounds suit small captured fixtures; this is not a streaming evaluation service. Reports omit raw HTML and title values, but include record IDs and digests. Keep sensitive captures and reports private.

Apply the same distinction to Scraipe

Scraipe's API contract accepts up to five sample URLs in POST /v1/compile. These are compile-time checks. A URL supplied there has participated in development and must not also be counted as untouched held-out evidence. alreadyWarm also means an existing extractor was available; a request to warm a site is not proof that a new candidate was built and evaluated.

Inspection of the current implementation shows that its additional-sample brittleness check examines the first extracted record for empty fields. Its runtime schema check samples up to five records. Those are useful safeguards with a different scope from the whole-result identity and value comparison demonstrated here. They do not prove complete record coverage or semantic correctness. See full-result schema validation for consumer-side type and cross-field checks, and parent-scoped nested extraction for record-boundary failures.

A live API evaluation also has changing execution paths. Cached data, reused rules, fresh compilation and healing are different experiments. For a deliberately new diagnostic operation, noCache: true bypasses the response cache but can still allow new compilation or repair. forceCompile: true requests rebuilding; it does not pin an unchanged extractor. The public workflow described here does not assume a frozen-spec execution switch.

Decide what you are measuring. To evaluate fixed selection rules, run a versioned extractor you control against frozen captures, as this pack does. To evaluate the hosted adaptive service, retain each request's options, source context, response headers, meta.tier, layout variant, returned data and charge receipt. Report repair or recompilation as part of that service behavior instead of describing every result as reuse of one unchanged candidate. Reject partial or failed responses before comparing values. Do not replay a successful operation's old idempotency key and count the stored answer as a new evaluation.

The reuse versus response-cache guide explains those distinctions. Hosted compilation, repair and network choices can incur credits; inspect the current quote and returned compile, egress and total charges before scaling an experiment. This offline pack has no such charges, and no authenticated hosted extraction was performed for this guide.

Decide before repairing

Our teaching rule requires every selected case to pass. In a real workflow, declare the rule before seeing results: for example, zero omitted required records, zero unexpected records and exact critical-field values, with a separate explicitly bounded policy for optional fields. Record both the numerator and denominator, failures by layout family and any excluded or unavailable captures. Never silently remove a difficult page from the denominator.

When a held-out case fails, preserve the failing candidate, corpus hashes and report. Investigate source completeness, labeling errors and selection logic separately. Move any case used to guide a fix into the regression set, give the candidate a new version, and reserve new untouched evaluation pages. Repassing the same six exposed cases is useful regression evidence, but it is no longer a fresh held-out result.

Start with the pack, then substitute a small permitted corpus and your own explicit consumer contract. Use the playground for a bounded hosted check if needed. Keep local fixture results, hosted behavior and production acceptance as separate records so the final decision rests on the evidence you actually collected.