Skip to content

Menu

Choose Read or Extract for a web-data task

Choose Read or Extract, bound your request plan and validate content, truncation and the final source origin before ingestion.

By Scraipe editorial. Checked 2026-10-17.

Start with the data your application needs

A website can supply two different kinds of useful input. A research assistant may need the words in an article, with its source URL attached. A catalog job may need a list of records with a title, price and availability field. These are different requirements even when the input URL is identical.

Use Read when the downstream task needs page content. Use Extract, exposed by the scrape endpoint, when the task needs named fields or records. Neither choice proves that the website is accessible or that the result is complete: inspect the response and validate it before using it.

Choose an operation

Downstream requirementStarting operationWhat to validate
Read an article for a research summaryReadReturned format, missing sections and source attribution
Store product titles and pricesExtractRequired fields, types, currency and record count
Build a knowledge-base ingestion pipelineRead, then a bounded crawl if neededCoverage, duplicates, freshness and source URLs
Compare changes to structured product recordsExtract, then a watch workflowInitial values and meaningful changes

A workflow can need both representations. For example, an agent may read a product description for context while storing a structured price separately. Keep those outputs distinct instead of asking later steps to guess which part of a free-form answer is the price.

Try a deterministic selection example

The following helper demonstrates the decision using explicit requirements. It does not fetch a website or claim that a live extraction succeeded.

javascript

function chooseOperation({ needsFields, needsWholeText }) {
  if (needsFields && needsWholeText) return ['read', 'scrape'];
  if (needsFields) return ['scrape'];
  if (needsWholeText) return ['read'];
  throw new Error('Define the downstream data requirement first');
}

Download the complete example with four assertions. Save it locally and run node choose-operation.mjs using Node 22 or later. The example makes no network calls and spends no credits. It checks field-only, text-only, combined and unspecified requirements.

Map the choice to the API

The API reference defines POST /v1/read for page content and POST /v1/scrape for structured extraction. Supply a supported public URL and the required authentication. For structured extraction, specify the fields or supported schema your consumer expects. Do not put an API key in frontend code or a public example.

Read can return Markdown, text or HTML depending on the request and supported behaviour. Markdown conversion may fall back to plain text, so inspect the returned format rather than assuming the requested format was always produced. A token or character limit can also mean that you have only part of a long document.

Extraction uses the page structure to locate values. Reusing an extractor can avoid another model call, but that does not mean every network request is free. Consult the current pricing and request receipt for compilation, repair and network charges. A selector finding a real element also does not prove it selected the correct business field.

Handle an incomplete result

Before passing output to another system, check that the operation succeeded and that its output satisfies your own requirements. For a record, validate required fields and retain the original currency. For an article, check that the expected section is present and attach the source URL and retrieval time.

If access is refused or the page requires unsupported interaction, surface that limitation. Do not turn an empty response into an invented value. For a rate limit, respect the retry guidance instead of repeatedly submitting the same request.

Plan a bounded batch before making API calls

Choosing an operation is only the first step. If an agent needs both page content and named fields for ten URLs, the starting plan contains twenty API calls. A limit on URLs alone does not capture that difference. Duplicate URLs can also make a job larger than intended.

The Read-or-Extract example pack v1 turns explicit requirements into an inspectable request plan. It does not send requests. The pack includes a Node 22 module, a command-line interface, an input fixture, twelve tests and a README. No API key or npm package is required. The fixture uses invented paths on example.com; they are planning inputs, not evidence that those pages exist or were extracted.

Download and reproduce the example

Save the archive in a fresh directory. Its SHA-256 checksum is:

text

56e034134a1d86b932bed8ac414b0887442fa3e4414213ce68280d1d477d14c6

On macOS or Linux, unpack it and run:

bash

tar -xzf read-or-extract-v1.tar.gz
cd read-or-extract-v1
node --test plan-requests.test.mjs
node plan-requests.mjs requirements.json > plan.json

The expected result is twelve passing tests, followed by a JSON plan containing four request recipes. You can also download the planner, input fixture and tests individually into the same directory. File checksums let you check the unpacked files. The test suite and command-line example were reproduced from the archive with Node 22.23.2 on 8 October 2026.

The fixture requests both representations:

json

{
  "version": 1,
  "requirements": {
    "pageContent": true,
    "fields": ["title", "price", "currency"]
  },
  "limits": { "maxUrls": 2, "maxRequests": 4 },
  "urls": [
    "https://example.com/catalog?page=1",
    "https://EXAMPLE.com:443/catalog?page=1#details",
    "https://example.com/catalog?page=2"
  ]
}

The first two entries have the same HTTP request URL after standard host/default-port normalization and removal of the fragment. The second page remains distinct because its query value differs. The output summary is:

json

{"inputUrls":3,"uniqueUrls":2,"duplicateUrls":1,"requests":4}

For each unique URL the plan contains one POST /v1/read body with format: "markdown" and maxChars: 10000, plus one POST /v1/scrape body with the three requested fields and maxTokens: 2000. Both select egress: "direct". These are explicit teaching defaults, not a promise that ten thousand characters or two thousand output tokens will cover your page.

Exercise the failure cases

Change maxRequests to 3 and run the command again. It exits with status 1, writes a diagnostic to standard error and emits no plan. It never silently drops a URL to fit the cap. Check the exit code before consuming plan.json: shell redirection still creates or truncates that file even when planning fails.

Change to the fixtureExpected result
Set pageContent to falseTwo Extract recipes, without Read recipes
Set fields to [] while keeping page contentTwo Read recipes
Request neither representationError asking for a data requirement
Keep both and lower the request cap to threeError before any plan is emitted
Include a credentialed, relative or malformed URLReject the entire input
Supply duplicate field names or a string booleanReject the ambiguous requirement

The tests also cover URL caps after deduplication, preserved query distinctions, deterministic command-line output, malformed JSON and isolation between the input and returned objects. The planner limits input to 1,000 URL entries, 100 unique URLs, 200 planned requests and 20 simple field names. These are sample guardrails, not Scraipe account limits.

Know what this plan cannot establish

URL normalization is conservative: paths, trailing slashes, query values and query order remain significant. The planner does not discover canonical tags, follow redirects, remove tracking parameters or compare page contents. A syntactically valid URL is not proof of public-network access or permission to collect a page. Signed query parameters can be sensitive; avoid storing them in public plans.

The result is a local recipe, not a Scraipe batch endpoint. A server-side runner must still provide the trusted API origin, scoped authentication, timeouts, concurrency limits and response validation. It must enforce your source-access policy. The planner creates no idempotency keys, retries or crawl jobs.

Four planned API calls are not a prediction of four origin fetches, four credits or four successes. Direct egress opts out of residential fallback; fresh extraction compilation can still cost credits. Inspect each executed request's receipt. Before passing a Read result to an agent, check the returned format, stats.truncated and required sections. For Extract, inspect truncated, validate the records and compare important fields with the source. Keep source content as data rather than allowing it to override your agent's instructions.

Decide whether a Read response is usable

A successful request can still be unsuitable for the next step. A Markdown-only parser cannot assume that a request for Markdown produced Markdown: the reader can return plain text when conversion fails or exceeds its budget. A summary task that only needs words may accept that fallback. Make the choice in your consumer instead of relabelling text as Markdown.

Read and Extract also report truncation in different places. Check stats.truncated for Read and top-level truncated for Extract. Treat an absent flag as unknown. A top-level truncated: false does not substitute for the nested Read flag.

Captured Read resultConsumer decision
ok: true, nonempty content, accepted format, stats.truncated: falseEligible for the next content-quality check
format: "text" after requesting MarkdownAccept only if plain text meets the downstream requirement
format: "html"Route to a separately designed HTML consumer; this example rejects it
stats.truncated: true or missingHold the result; inspect the limit and source before another request
Empty content or a failed envelopeStop; do not create a summary from missing evidence
Expected text is absentInspect the captured page, extraction scope and source version

Download the Read response guard, its seventeen offline tests and the invented capture into one directory. With Node 22 or later, run:

bash

node --test audit-read-result.test.mjs
node audit-read-result.mjs read-result.fixture.json

The fixture is a short invented returns-policy excerpt on example.com, not a fetched page. The seventeen tests and CLI were run with Node 22.23.2 on 17 October 2026. The default policy accepts Markdown or text, requires nonempty content and requires the nested truncation flag to be exactly false. The command prints {"accepted":true,"issues":[]} and exits 0 for this fixture. Change stats.truncated to true: it prints {"accepted":false,"issues":["truncated_or_unknown"]} and exits 1. Invalid JSON or an unreadable capture exits 2. No API key, network call or retry is involved.

For a consumer that requires Markdown, a known phrase and an explicitly permitted source origin, import the guard into an .mjs script in the same directory:

javascript

import { readFileSync } from 'node:fs';
import { auditReadResult } from './audit-read-result.mjs';

const response = JSON.parse(readFileSync('read-result.fixture.json', 'utf8'));
const report = auditReadResult(response, {
  acceptedFormats: ['markdown'],
  requiredText: ['Returns'],
  allowedOrigins: ['https://example.com'],
});
console.log(report); // { accepted: true, issues: [] }

requiredText checks case-sensitive substrings, not document structure or factual accuracy. A matching phrase in navigation could still pass. Adapt the policy to an actual source and inspect important sections yourself. The CLI uses the default policy; custom formats, required phrases and allowed origins are module options. The guard reports issue codes without rewriting the captured content or echoing a server's error body.

accepted: true means this small consumer contract passed. It does not establish that the source is current, that main-content selection retained every section, or that the text is safe to follow as instructions. Preserve the requested URL, returned URL, capture time, request identifier and receipt separately. Keep page text as untrusted input when passing it to an agent. A complete Read response is still one page, not whole-site coverage.

The existing request planner limits work before dispatch; this guard checks a captured Read response afterward. For structured records, use the separate missing-field validation workflow. To understand reported work and charges, inspect the request receipt. Those checks answer different questions and should not be replaced by a single success flag.

Check the final source origin before ingestion

A Read response's url can differ from the URL you requested: the current API returns the final fetched URL after redirects. If your research job permits only a particular documentation site, accepting a complete Markdown response from another origin would still violate that collection policy. The guard above now supports allowedOrigins, an optional list supplied by your application configuration. Never build that list from the response being checked.

An origin includes the scheme, hostname and effective port. With https://example.com allowed, the guard accepts https://EXAMPLE.com:443/returns, but rejects HTTP, port 8443, subdomains and a lookalike such as https://example.com.other.invalid/returns. It compares parsed origins for exact equality; it does not use substring matching or assume that every subdomain is approved. To permit a known move to a second site, explicitly add that site's origin after reviewing the change.

Captured source with an origin policy enabledGuard result
Valid HTTP(S) URL on a listed originContinue the existing format, content and truncation checks
Valid URL on an unlisted originsource_origin_not_allowed
Missing, relative, malformed or credentialed URLinvalid_source_url
Empty or invalid configured origin listThrow a configuration error before evaluating the response

Try replacing the fixture's URL with https://other.invalid/returns in the imported example: the report becomes {"accepted":false,"issues":["source_origin_not_allowed"]}. The other response fields are unchanged. The default CLI still checks only content, format and truncation; use the module option when source restrictions matter. The tests cover that distinction, default-port normalization, scheme and port mismatches, malformed URLs, invalid configuration and preservation of the original capture.

This is an after-fetch acceptance check. It does not prevent an earlier redirect request, resolve DNS, enforce a path-level allowlist, prove source authenticity or detect prompt injection. Enforce network-access rules before dispatch in the fetching layer, and keep content untrusted after acceptance. The option validates the returned metadata against your policy; it is not a general SSRF filter. Preserve both requested and returned URLs so a rejected redirect remains diagnosable without silently relabelling its content.

Continue with a small real workload

Try the JSON demo or the Markdown reader, then move to your own permitted URL in the playground. Compare the response with the source page before increasing the number of pages. This guide verifies the local decision example and documents the API contract; it does not present a live-service benchmark.

Sources and verification

Try the next step

Try the playground

Continue building