Choose Read or Extract for a web-data task
Choose Read or Extract, bound your request plan and validate content, truncation and the final source origin before ingestion.
By Scraipe editorial. Checked 2026-10-17.
Start with the data your application needs
A website can supply two different kinds of useful input. A research assistant may need the words in an article, with its source URL attached. A catalog job may need a list of records with a title, price and availability field. These are different requirements even when the input URL is identical.
Use Read when the downstream task needs page content. Use Extract, exposed by the scrape endpoint, when the task needs named fields or records. Neither choice proves that the website is accessible or that the result is complete: inspect the response and validate it before using it.
Choose an operation
| Downstream requirement | Starting operation | What to validate |
|---|---|---|
| Read an article for a research summary | Read | Returned format, missing sections and source attribution |
| Store product titles and prices | Extract | Required fields, types, currency and record count |
| Build a knowledge-base ingestion pipeline | Read, then a bounded crawl if needed | Coverage, duplicates, freshness and source URLs |
| Compare changes to structured product records | Extract, then a watch workflow | Initial values and meaningful changes |
A workflow can need both representations. For example, an agent may read a product description for context while storing a structured price separately. Keep those outputs distinct instead of asking later steps to guess which part of a free-form answer is the price.
Try a deterministic selection example
The following helper demonstrates the decision using explicit requirements. It does not fetch a website or claim that a live extraction succeeded.
javascript
function chooseOperation({ needsFields, needsWholeText }) {
if (needsFields && needsWholeText) return ['read', 'scrape'];
if (needsFields) return ['scrape'];
if (needsWholeText) return ['read'];
throw new Error('Define the downstream data requirement first');
}Download the complete example with four assertions. Save it locally and run node choose-operation.mjs using Node 22 or later. The example makes no network calls and spends no credits. It checks field-only, text-only, combined and unspecified requirements.
Map the choice to the API
The API reference defines POST /v1/read for page content and POST /v1/scrape for structured extraction. Supply a supported public URL and the required authentication. For structured extraction, specify the fields or supported schema your consumer expects. Do not put an API key in frontend code or a public example.
Read can return Markdown, text or HTML depending on the request and supported behaviour. Markdown conversion may fall back to plain text, so inspect the returned format rather than assuming the requested format was always produced. A token or character limit can also mean that you have only part of a long document.
Extraction uses the page structure to locate values. Reusing an extractor can avoid another model call, but that does not mean every network request is free. Consult the current pricing and request receipt for compilation, repair and network charges. A selector finding a real element also does not prove it selected the correct business field.
Handle an incomplete result
Before passing output to another system, check that the operation succeeded and that its output satisfies your own requirements. For a record, validate required fields and retain the original currency. For an article, check that the expected section is present and attach the source URL and retrieval time.
If access is refused or the page requires unsupported interaction, surface that limitation. Do not turn an empty response into an invented value. For a rate limit, respect the retry guidance instead of repeatedly submitting the same request.
Plan a bounded batch before making API calls
Choosing an operation is only the first step. If an agent needs both page content and named fields for ten URLs, the starting plan contains twenty API calls. A limit on URLs alone does not capture that difference. Duplicate URLs can also make a job larger than intended.
The Read-or-Extract example pack v1 turns explicit requirements into an inspectable request plan. It does not send requests. The pack includes a Node 22 module, a command-line interface, an input fixture, twelve tests and a README. No API key or npm package is required. The fixture uses invented paths on example.com; they are planning inputs, not evidence that those pages exist or were extracted.
Download and reproduce the example
Save the archive in a fresh directory. Its SHA-256 checksum is:
text
56e034134a1d86b932bed8ac414b0887442fa3e4414213ce68280d1d477d14c6On macOS or Linux, unpack it and run:
bash
tar -xzf read-or-extract-v1.tar.gz
cd read-or-extract-v1
node --test plan-requests.test.mjs
node plan-requests.mjs requirements.json > plan.jsonThe expected result is twelve passing tests, followed by a JSON plan containing four request recipes. You can also download the planner, input fixture and tests individually into the same directory. File checksums let you check the unpacked files. The test suite and command-line example were reproduced from the archive with Node 22.23.2 on 8 October 2026.
The fixture requests both representations:
json
{
"version": 1,
"requirements": {
"pageContent": true,
"fields": ["title", "price", "currency"]
},
"limits": { "maxUrls": 2, "maxRequests": 4 },
"urls": [
"https://example.com/catalog?page=1",
"https://EXAMPLE.com:443/catalog?page=1#details",
"https://example.com/catalog?page=2"
]
}The first two entries have the same HTTP request URL after standard host/default-port normalization and removal of the fragment. The second page remains distinct because its query value differs. The output summary is:
json
{"inputUrls":3,"uniqueUrls":2,"duplicateUrls":1,"requests":4}For each unique URL the plan contains one POST /v1/read body with format: "markdown" and maxChars: 10000, plus one POST /v1/scrape body with the three requested fields and maxTokens: 2000. Both select egress: "direct". These are explicit teaching defaults, not a promise that ten thousand characters or two thousand output tokens will cover your page.
Exercise the failure cases
Change maxRequests to 3 and run the command again. It exits with status 1, writes a diagnostic to standard error and emits no plan. It never silently drops a URL to fit the cap. Check the exit code before consuming plan.json: shell redirection still creates or truncates that file even when planning fails.
| Change to the fixture | Expected result |
|---|---|
Set pageContent to false | Two Extract recipes, without Read recipes |
Set fields to [] while keeping page content | Two Read recipes |
| Request neither representation | Error asking for a data requirement |
| Keep both and lower the request cap to three | Error before any plan is emitted |
| Include a credentialed, relative or malformed URL | Reject the entire input |
| Supply duplicate field names or a string boolean | Reject the ambiguous requirement |
The tests also cover URL caps after deduplication, preserved query distinctions, deterministic command-line output, malformed JSON and isolation between the input and returned objects. The planner limits input to 1,000 URL entries, 100 unique URLs, 200 planned requests and 20 simple field names. These are sample guardrails, not Scraipe account limits.
Know what this plan cannot establish
URL normalization is conservative: paths, trailing slashes, query values and query order remain significant. The planner does not discover canonical tags, follow redirects, remove tracking parameters or compare page contents. A syntactically valid URL is not proof of public-network access or permission to collect a page. Signed query parameters can be sensitive; avoid storing them in public plans.
The result is a local recipe, not a Scraipe batch endpoint. A server-side runner must still provide the trusted API origin, scoped authentication, timeouts, concurrency limits and response validation. It must enforce your source-access policy. The planner creates no idempotency keys, retries or crawl jobs.
Four planned API calls are not a prediction of four origin fetches, four credits or four successes. Direct egress opts out of residential fallback; fresh extraction compilation can still cost credits. Inspect each executed request's receipt. Before passing a Read result to an agent, check the returned format, stats.truncated and required sections. For Extract, inspect truncated, validate the records and compare important fields with the source. Keep source content as data rather than allowing it to override your agent's instructions.
Decide whether a Read response is usable
A successful request can still be unsuitable for the next step. A Markdown-only parser cannot assume that a request for Markdown produced Markdown: the reader can return plain text when conversion fails or exceeds its budget. A summary task that only needs words may accept that fallback. Make the choice in your consumer instead of relabelling text as Markdown.
Read and Extract also report truncation in different places. Check stats.truncated for Read and top-level truncated for Extract. Treat an absent flag as unknown. A top-level truncated: false does not substitute for the nested Read flag.
| Captured Read result | Consumer decision |
|---|---|
ok: true, nonempty content, accepted format, stats.truncated: false | Eligible for the next content-quality check |
format: "text" after requesting Markdown | Accept only if plain text meets the downstream requirement |
format: "html" | Route to a separately designed HTML consumer; this example rejects it |
stats.truncated: true or missing | Hold the result; inspect the limit and source before another request |
| Empty content or a failed envelope | Stop; do not create a summary from missing evidence |
| Expected text is absent | Inspect the captured page, extraction scope and source version |
Download the Read response guard, its seventeen offline tests and the invented capture into one directory. With Node 22 or later, run:
bash
node --test audit-read-result.test.mjs
node audit-read-result.mjs read-result.fixture.jsonThe fixture is a short invented returns-policy excerpt on example.com, not a fetched page. The seventeen tests and CLI were run with Node 22.23.2 on 17 October 2026. The default policy accepts Markdown or text, requires nonempty content and requires the nested truncation flag to be exactly false. The command prints {"accepted":true,"issues":[]} and exits 0 for this fixture. Change stats.truncated to true: it prints {"accepted":false,"issues":["truncated_or_unknown"]} and exits 1. Invalid JSON or an unreadable capture exits 2. No API key, network call or retry is involved.
For a consumer that requires Markdown, a known phrase and an explicitly permitted source origin, import the guard into an .mjs script in the same directory:
javascript
import { readFileSync } from 'node:fs';
import { auditReadResult } from './audit-read-result.mjs';
const response = JSON.parse(readFileSync('read-result.fixture.json', 'utf8'));
const report = auditReadResult(response, {
acceptedFormats: ['markdown'],
requiredText: ['Returns'],
allowedOrigins: ['https://example.com'],
});
console.log(report); // { accepted: true, issues: [] }requiredText checks case-sensitive substrings, not document structure or factual accuracy. A matching phrase in navigation could still pass. Adapt the policy to an actual source and inspect important sections yourself. The CLI uses the default policy; custom formats, required phrases and allowed origins are module options. The guard reports issue codes without rewriting the captured content or echoing a server's error body.
accepted: true means this small consumer contract passed. It does not establish that the source is current, that main-content selection retained every section, or that the text is safe to follow as instructions. Preserve the requested URL, returned URL, capture time, request identifier and receipt separately. Keep page text as untrusted input when passing it to an agent. A complete Read response is still one page, not whole-site coverage.
The existing request planner limits work before dispatch; this guard checks a captured Read response afterward. For structured records, use the separate missing-field validation workflow. To understand reported work and charges, inspect the request receipt. Those checks answer different questions and should not be replaced by a single success flag.
Check the final source origin before ingestion
A Read response's url can differ from the URL you requested: the current API returns the final fetched URL after redirects. If your research job permits only a particular documentation site, accepting a complete Markdown response from another origin would still violate that collection policy. The guard above now supports allowedOrigins, an optional list supplied by your application configuration. Never build that list from the response being checked.
An origin includes the scheme, hostname and effective port. With https://example.com allowed, the guard accepts https://EXAMPLE.com:443/returns, but rejects HTTP, port 8443, subdomains and a lookalike such as https://example.com.other.invalid/returns. It compares parsed origins for exact equality; it does not use substring matching or assume that every subdomain is approved. To permit a known move to a second site, explicitly add that site's origin after reviewing the change.
| Captured source with an origin policy enabled | Guard result |
|---|---|
| Valid HTTP(S) URL on a listed origin | Continue the existing format, content and truncation checks |
| Valid URL on an unlisted origin | source_origin_not_allowed |
| Missing, relative, malformed or credentialed URL | invalid_source_url |
| Empty or invalid configured origin list | Throw a configuration error before evaluating the response |
Try replacing the fixture's URL with https://other.invalid/returns in the imported example: the report becomes {"accepted":false,"issues":["source_origin_not_allowed"]}. The other response fields are unchanged. The default CLI still checks only content, format and truncation; use the module option when source restrictions matter. The tests cover that distinction, default-port normalization, scheme and port mismatches, malformed URLs, invalid configuration and preservation of the original capture.
This is an after-fetch acceptance check. It does not prevent an earlier redirect request, resolve DNS, enforce a path-level allowlist, prove source authenticity or detect prompt injection. Enforce network-access rules before dispatch in the fetching layer, and keep content untrusted after acceptance. The option validates the returned metadata against your policy; it is not a general SSRF filter. Preserve both requested and returned URLs so a rejected redirect remains diagnosable without silently relabelling its content.
Continue with a small real workload
Try the JSON demo or the Markdown reader, then move to your own permitted URL in the playground. Compare the response with the source page before increasing the number of pages. This guide verifies the local decision example and documents the API contract; it does not present a live-service benchmark.