Measure the cost of many layouts versus many pages

Measure charged operations, not just URL count

A thousand pages that share a working extractor can have a different cost profile from a thousand pages spread across incompatible layouts. But layout count is only one explanatory variable. Schema changes, repairs, cache state, network choice, repeated visits and failed outputs also affect the experiment. Measure the actual receipt breakdown and the useful results you accepted.

Use three separate counts: distinct pages visited, declared layout families and actual operations. A repeat visit adds an operation without adding a page. A visually different page may still work with an existing extractor. A second schema on the same page can create different work. Do not equate one URL, one layout, one compile and one charge.

This guide supplies a dependency-free ledger summarizer and four invented scenarios. Running it makes no Scraipe calls and measures no hosted performance. Its value is a reproducible accounting method: include pre-warming, keep rejected outputs in the cost numerator, and leave missing charges unknown.

Define comparable cohorts before making requests

A cohort is the set of operations you will summarize together. Give each cohort a clear workload and a fixed observation window. Keep the schema version, content representation, cache policy, network policy, acceptance rules and intended number of results consistent when investigating layout variation. Record differences that you cannot hold constant.

For a real study, inventory the source with representative sample pages. Label layout families before seeing the cost results, using the extraction boundary and observed structure. These are your analytical labels, not necessarily the service's variant identifier. Retain both separately if you need to investigate a discrepancy.

Use two complementary views:

• A controlled comparison asks what changes when one factor changes while the others stay comparable.
• A representative workload measures the mix you actually expect to operate, including failures and repairs.

Do not treat an artificially cold run as a normal daily average or a warm run as a full onboarding cost. Scraipe may reuse existing shared work; bypassing a response cache does not guarantee a cold extractor. Explicit rebuilding also changes the workload. Record what the receipt says actually happened instead of assuming the desired test state was achieved. The cache and reuse guide explains these distinctions.

Capture the complete cost boundary

The current API contract reports meta.credits.compile, meta.credits.egress and meta.credits.total. Use the combined total when measuring the cost of an operation. The legacy X-Scraipe-Credits-Charged header describes whole compile credits; it is not the combined network-plus-compile amount.

Include pre-warm operations if they were needed for the workload. Keep them in a separate phase so readers can see where the cost occurred, while retaining them in the full numerator. Report the warmup-only and workload-only views as secondary breakdowns, not as substitutes for the full total. A warm request with no compile charge does not erase a prior compile.

Preserve enough context to reconcile each original operation: a local unique operation ID, the account/window boundary, original request or receipt ID, endpoint, phase, final outcome, acceptance result and charge breakdown. Keep secrets and private records out of shareable reports. For batch or crawl jobs, select one accounting level. Adding a parent total to its already-included child totals counts the same work twice.

An idempotent replay can return the original positive receipt without a new debit. Do not sum response captures blindly. Resolve replay identity using the recorded replay context and account request records before constructing this ledger. See request receipt inspection. If a timeout or missing record leaves the charge unknown, keep it unknown until reconciled.

Reproduce four accounting scenarios

Download these four files together:

• Cost summarizer module and command
• Fifteen runnable tests
• Synthetic operation ledger
• Payload checksums

Use Node 22.23.2 or a compatible Node 22 release. No package installation, API key, payment or network access is needed. Run:

shasum -a 256 -c SHA256SUMS
node --test summarize-cost.test.mjs
node summarize-cost.mjs operations.json

The built-in Node test runner executes fifteen tests. The example contains 27 invented operations across four independent cohorts, each with six distinct workload pages and six accepted workload results. Three additional operations are pre-warms in the third cohort.

These numbers are assigned fixture inputs, not measured Scraipe charges, a pricing offer or a prediction. The table illustrates two accounting mistakes: dropping setup makes pre-warming look free, and dropping network charges understates the last cohort. It does not establish that three layouts always cost three times as much.

The command reports integer thousandths of a credit, named Milli. Thus totalMilli: 1300 means 1.300 credits. Its average is an unrounded JavaScript ratio; the table rounds credit averages to four decimals. The tool makes no currency conversion. Current pack currencies, discounts and account transactions require separate reconciliation; a credit total is not a cash invoice.

Adapt a reconciled ledger

This is a local analysis format, not a Scraipe API response or an automatic account export. Normalize your validated receipt amounts into integer thousandths and preserve the original receipt privately. The receipt checker can help validate the decimal breakdown before normalization. Do not use arbitrary floating-point rounding to hide unsupported precision.

{
  "kind": "operation",
  "replay": false,
  "id": "study-a-original-001",
  "cohort": "one-layout",
  "layout": "product-v1",
  "page": "page-001",
  "phase": "workload",
  "outcome": "success",
  "accepted": true,
  "compileMilli": 1000,
  "egressMilli": 50,
  "totalMilli": 1050
}

All identities are caller-assigned, short lowercase labels. replay: false is a caller assertion after reconciliation, not something the program can discover. It rejects explicitly marked replays, duplicate operation IDs and non-operation summaries. Renaming the same original operation twice can still defeat that guard; the tool cannot prove ledger completeness or distinct billing events.

accepted means the result passed your independently defined consumer checks. It is not inferred from HTTP 200 or a valid JSON shape. Only a successful workload operation can be accepted. Warmups are never accepted workload results. Failure and rejected-success costs remain in the total because they are part of the work required to obtain useful outputs.

Use all three charge fields as null for an unresolved numeric breakdown. The tool reports the known subtotal but makes the cohort's total and average null, with status: "incomplete". Missing fields, mixed null/numeric fields, negative or fractional thousandths, unsafe integers and inconsistent sums are errors. This deliberately distinguishes unknown charges from measured zero charges.

Test the conclusions with negative controls

Copy operations.json and replace the three charge fields of one row with null. The command exits 1 and reports an incomplete cohort; it does not divide a partial subtotal by the full result count. Set one successful workload row's accepted to false and rerun: its charge stays in the numerator while the accepted-result denominator falls. Mark a row as a replay or duplicate its operation ID and the command exits 2 instead of silently counting it.

The tests also cover parent-summary rejection, repeat visits, arithmetic mismatch and integer overflow. Repeated visits with different operation IDs increase operations and accepted results, while distinctWorkloadPages remains deduplicated. Decide whether those repeated outputs are genuinely useful to your consumer. If your target is unique accepted entities or accepted records, calculate that separate denominator from validated outputs; this tool reports accepted operation results, not records or distinct accepted pages.

Exit 0 means the provided ledger has complete numeric charges and passed these checks. It does not mean the service is inexpensive, every result was useful or the account balance was verified. Zero accepted results produces a null average, even when total credits are zero. The CLI limits files to 4 MiB and the analyzer to 10,000 operations; larger studies need an intentionally designed ingestion and reconciliation process.

Report the result with its limits

For each real cohort, report the observation window, schema and capture policy, declared layouts, distinct pages, operations, accepted outputs, failures, warmup cost, compile cost, network cost and unresolved receipts. Include all runs that met the predeclared rules. Do not discard expensive repairs or failed outputs merely because they weaken a cost-saving story.

The useful ratio is the full observed cost divided by the useful result count, provided the numerator is complete and the denominator matches the reader's task. Also retain absolute totals and acceptance rates: a low average from a tiny easy sample is weak evidence for a larger deployment. Repeat across representative windows before extrapolating, and distinguish future assumptions from observed records.

Current product behavior can reuse an extractor across changed page structure, recompile after failure, or return a cached response. None of those possibilities is a universal per-layout tariff. Start a permitted pilot in the playground, reconcile its receipts, and test its outputs using held-out evaluation. This guide supplies the accounting method and local arithmetic evidence; it makes no claim about authenticated production savings, throughput or extraction accuracy.