Choose representative sample pages

Select variation before collecting more URLs

Choose sample pages that exercise different extraction risks, then keep a separate set for evaluation. Ten nearly identical product URLs can tell you less than a standard product, one without a price, a multi-variant product and a page from the site's older template. URL count is a collection measure; coverage is an argument about what could fail.

Start with a small, explicit scope: one permitted site, a versioned output schema, a capture method and the reader task. A product-detail extractor and a category-list extractor have different record boundaries. Treat them as separate families until you have evidence that one extractor handles both. A URL path or visual resemblance is a useful clue, not proof of a shared layout.

This guide provides a local inventory audit. It checks declared coverage and catches some development/evaluation overlap. It does not fetch pages, discover layouts, run an extractor or estimate accuracy. Its ten inventory entries are synthetic teaching metadata, including hashes of invented strings. They are not observations from customers or sampled websites.

Write a coverage matrix for the failure modes

Inspect permitted pages in the same representation your extractor will receive. For each family, list observable conditions that affect the result. Save the capture timestamp, final URL, HTTP/access state, representation, capture hash and expected records in your own private inventory. Exclude login walls and incomplete captures from normal content examples; preserve them as separate rejection tests.

Choose conditions because they matter to your consumer. Do not expand the matrix with synonyms to make it look comprehensive. If two risks interact, add an explicit combination such as sale+unavailable; coverage of the individual tags does not establish coverage of their combination. If a condition is outside scope, record the exclusion and the behavior when it appears.

Use two complementary sets. A deliberately varied risk set tests known failure modes. A separately sampled slice of the actual workload helps reveal common cases and previously missed conditions. This small audit handles the first set only. It does not weight cases by traffic or produce a statistically representative success rate.

Keep related captures together before evaluation

Assign related captures a stable group ID before splitting. For example, a product's URL aliases, locale variants or repeated captures may belong to the same group when their shared content would make evaluation too easy. Keep that group entirely in development or entirely in evaluation. If your question is future drift, use an appropriate later capture window as well; a random split alone does not answer it.

The scikit-learn cross-validation guide explains why dependent observations need group-aware splits and why time-dependent data needs a different evaluation design. We apply that principle to capture inventories here; the downloadable tool does not use scikit-learn or implement its algorithms.

Group IDs are global in this example. Exact matching capture hashes also trigger an overlap finding across the two selected sets, even if the group names differ. Different hashes do not prove independence: small HTML changes can hide duplicate content. Review near-duplicates and grouping manually. Conversely, identical generic empty pages may deserve a documented exception in your own process; this example conservatively flags them for review.

Do not tune selectors after reading evaluation failures and then call the same evaluation blind. Move those cases into the regression suite and reserve new independent cases for the next evaluation. The companion guide on held-out page testing covers executing the frozen candidate and comparing every expected record.

Run the inventory audit

Download all four files into one directory:

• Audit module and command
• Fourteen tests
• Synthetic inventory
• Payload checksums

Use Node 22.23.2 or a compatible Node 22 installation. There are no npm dependencies, API keys, network calls or credit charges. The Node test runner runs the suite directly:

shasum -a 256 -c SHA256SUMS
node --test audit-samples.test.mjs
node audit-samples.mjs inventory.json

The supplied inventory has two families and four family/condition requirements. Eight selected pages cover those requirements once in development and once in evaluation; two duplicate candidates remain in reserve. The report has status: "covered", selected: 8, reserve: 2, eight coverage cells and no issues. Each cell lists the IDs that supply its evidence.

The count of eight is a teaching example, not a recommended sample size. The decision depends on the source's variation, the consequences of omissions and the amount of independent labeling you can do.

Replace the inventory with your own reviewed metadata. A page entry looks like this:

{
  "id": "prod-d-standard",
  "family": "product",
  "group": "product-001",
  "sha256": "64-lowercase-hex-characters-from-your-capture",
  "split": "development",
  "tags": ["standard"]
}

The hash string above is a placeholder explaining the field, not a runnable value. Use the actual SHA256 of the bytes you retained. IDs, groups, families and tags must be short lowercase identifiers; the script rejects malformed hashes and duplicate page IDs. Keep URLs, credentials and private extracted records out of shared metadata when they are unnecessary. Group IDs should not expose private identifiers.

Reproduce a coverage failure

Make a copy of inventory.json. Change the list-e-empty entry from evaluation to reserve, then run the audit on that copy. It reports status: "review" and a coverage-gap for family listing, tag empty, split evaluation, and exits with code 1. It does not silently replace that missing case with another standard page.

Other useful negative controls are included in the tests: one entity assigned to both sets, identical bytes under different group IDs, a combined risk that has never been tagged, and an inventory whose labels are explicitly unverified. That last case still reports covered: it demonstrates the tool's central limitation. A metadata checker cannot establish that its input labels are true.

Exit 0 means declared coverage with no detected overlap. Exit 1 means review is needed. Exit 2 means invalid input or command usage. Inputs are limited to 1 MiB, 2,000 pages, 200 requirements and 30 tags per page. This is a bounded local planning aid, not a crawler or a large-data sampling engine. Reserve entries do not contribute coverage and are not evaluated for cross-split overlap until selected.

Use compile samples as development evidence

The current Scraipe API contract allows up to five additional absolute HTTP(S) URLs in POST /v1/compile's samples field. Choose those from the relevant development family, with different content and the important expected conditions. Keep your independent evaluation captures out of that request.

Check brittleness.checked, not just whether the call succeeded. In the current implementation, a sample that cannot be fetched may be omitted. The brittleness check inspects the first extracted record's field presence, with exceptions for fields legitimately absent on the main page. It does not compare every sample record against independently labeled truth. A valid empty collection also needs its own acceptance test rather than assuming this first-record presence check covers it.

An already-warm response may skip compilation. Neither that response nor a positive brittleness result means all requested samples were fully evaluated. Inspect the response and then run your own full-record checks against the frozen candidate. This guide checks the documented contract and repository behavior; it does not claim an authenticated hosted compile run.

Decide what the sample set can support

Before proceeding, you should be able to name the scope, explain every requirement, identify a development and evaluation case for each selected risk, and show that evaluation identities did not leak into development. Preserve the inventory version alongside the extractor and schema versions. If one rare condition has only one usable capture, acknowledge the evaluation gap and collect another independent case; do not duplicate it across sets to make the report green.

Collection, rendering and labeling take time even when the local audit is free. Compile and network charges belong to the actual API receipts; sample count alone is not a quote. Start with a bounded permitted workload and inspect its request receipts. When the source changes, use the layout-change workflow to preserve failures and revise the inventory. The next action is a frozen-candidate test with complete expected records, not a production promotion based solely on covered.