Estimate an extraction workload before running it

You have 120 product pages to extract and four credits available for the run. What would make the job fit? Start with a quote for a representative page in each group. Then count how often the group will need to learn a layout and how often it will use a paid network route. Those two costs recur at different rates.

In the worked example below, 80 catalog pages cost one credit under the stated assumptions. Forty listings add 3.5 credits. Reducing the listings to 20 brings the estimate down to 3.75 credits, even though the smaller run still needs the same two listing layouts learned. The downloadable planner lets you replace each assumption and see how much headroom you have before issuing any scrapes.

Count the operations you intend to run

Make a list of the URLs for one pass. Group pages that you expect to share a host, field set, layout and network policy. A catalog detail page and a seller listing can have different layouts on the same site. A different country or field set can also change which quote is representative. If you cannot justify grouping two pages together, quote them separately.

This planner covers one single-page extraction per listed operation. Count a planned second visit again. For a request that follows pagination, resolve its page scope first or budget it separately. If discovery will choose the URLs at runtime, establish a crawl depth and page budget before using a fixed-page estimate.

Quote a representative request

Use an API key with the read scope. Supply the URL, fields or schema, egress policy and country you intend to use. Here is a shell example for a public catalog. Set SCRAIPE_API_KEY in your shell and replace the illustrative URL with your permitted target.

curl --fail-with-body -sS \
  https://scraipe-fc66a552921e.herokuapp.com/v1/quote \
  -H "Authorization: Bearer ${SCRAIPE_API_KEY}" \
  -H 'Content-Type: application/json' \
  --data '{"url":"https://catalog.example/item/42","fields":["title"],"egress":"direct"}'

The quote endpoint looks up the extractor and network policy without fetching the target or charging for extraction. Omitting the field set gives a network-only quote with warm: null. That leaves the learning cost unresolved. Keep the quote with its request parameters and capture time so you can tell which batch it describes. Scraipe API reference

Read estimatedCredits.compile and estimatedCredits.egress as estimates for the sample request. A cold sample may show one compile credit. A compatible extractor can then serve many pages. Conversely, a warm sample can have a different layout from the next page. The quote’s max is calculated for that request under its current assumptions. It neither reserves credits nor enforces a spending limit across a batch.

Check requiresPurchase and refused before using a rate. An unresolved route or missing field-specific quote should leave the group unpriced. Treating that group as free would make the total look affordable. If the quote request itself fails, inspect the error and obtain a usable quote before continuing.

Turn the quote into a workload estimate

For each group, multiply the number of assumed billable compilations by the compile rate, then add assumed successful paid fetches times the quoted egress rate. The planner takes compileCredits from public pricing and the per-fetch rate from estimatedCredits.egress, which is already expressed in credits. The separate residential rate table in the pricing API uses millicredits, with 1,000 millicredits per credit. Pricing API

The two sample quotes came from Scraipe’s local HTTP route tests using invented hosts, a test account and simulated routing. Both samples were cold. Catalog egress was quoted at zero and listing egress at 0.05 credits. The page counts, two listing layouts and ten listing operations assumed to incur no network charge are planning inputs. No hosted residential requests or customer workloads were measured.

If you know little about the host, change those assumptions before deciding. If all 40 listings incur the same 0.05-credit network charge, the total reaches 5 credits. If some pages need a differently priced route, put them in their own group with an appropriate quote. A cached sample alone tells you little about the next uncached URL.

Run the workload estimate

Save these three files in one directory. The planner runs offline in Node 22 and requires no packages, API key or credits.

• Workload planner
• Example plan and two sample quotes
• Planner tests

node --test estimate.test.mjs
node estimate.mjs plan.json

The tests pass. The second command prints a JSON report with 120 operations, 4.5 estimated credits, a budget of 4, headroom of −0.5 and the decision over-scenario-budget. It deliberately exits 1 so a calling script can hold that plan for revision.

Make a copy of plan.json. In the listings group, change pages from 40 to 20 and paidFetches from 30 to 15. Leave compiles at 2 because the smaller selection still includes both assumed layouts. Run the planner on the copy. It reports 100 operations, 3.75 estimated credits and 0.25 credits of headroom, then exits 0. Changing requiresPurchase to true instead makes the plan incomplete. The tool exits 2 and emits no total. None of these outcomes initiates a scrape.

Check the assumptions on a pilot

Pick a small permitted sample that includes the different layouts and routes you counted. If you run it, keep each request’s actual compile, egress and total credits and record whether its extracted data was usable. Compare those results with the planned group. Requote and revise the remaining inventory if you encounter a new layout, a more expensive network class or a different mix of cache hits. Include credits already consumed by the pilot when comparing the revised plan with your original budget. Request receipt guide

The planner covers the learning events and successful paid fetches you entered. Retries, repairs, paid setup outside the batch, and future visits require their own assumptions. A positive balance in the report is headroom in that scenario. To measure what the completed workload actually cost, include rejected results and setup operations in your layout and page cost comparison.