Skip to content

Menu

Extract website data with Python

Extract product titles and prices with a Python client, validate JSON, retain currency and handle rate limits, truncation and retries.

By Scraipe editorial. Checked 2026-10-17.

Extract records you can validate

To extract website data with Python, send a permitted page URL and the fields you need to Scraipe's POST /v1/scrape endpoint, then validate the returned records before saving them. This guide requests product titles and prices using Python's standard library. It preserves the source URL, currency text and request receipt so you can inspect what happened later.

If your application needs an article's text rather than product records, start with Choose Read or Extract. Here the reader task is narrower: turn one product-list page into a checked JSON result. The example deliberately stops when output is incomplete and sends no automatic retries.

Before you start

You need Python 3.10 or later, a working Scraipe API deployment and an API key with the scrape scope. The downloadable example and its fixtures were retested with Python 3.14.6 on 17 October 2026. Keep your key in your server environment or secret manager, outside source control and browser code.

Use a public page you are permitted to process. The command below uses Books to Scrape, a demonstration catalogue. It requests only that page: it does not discover every product in the catalogue or follow next-page links. Check the source page and your intended use before expanding to other sites.

Download the Python client and its local HTTP tests into the same folder. No pip packages are required.

Run the example without an API key first

bash

python3 test-extract-website-data.py

This starts a temporary server on your own computer, sends fixture requests to it and shuts it down afterward. It does not contact Scraipe or Books to Scrape and spends no credits. The expected result is Ran 24 tests followed by OK.

The tests cover the outgoing headers and request body, retained currency text, empty results, invalid records, response truncation, a rate limit, a failure envelope, non-JSON gateway responses, redirects, an insecure remote API origin, and a simulated network failure. They also check that manually repeating an operation preserves its key and body. That last check proves client behaviour; it does not simulate the backend's deduplication guarantee. Additional cases exercise duplicate JSON names, non-finite numbers, exponent overflow, Unicode, the local byte limit, invalid encoding and interrupted HTTP bodies. Four further cases check count consistency, strict integer types, explicit null cursors and complete multi-record output.

Make one authenticated extraction

Configure SCRAIPE_API_URL with the HTTPS origin of your Scraipe API deployment, without /v1 or a trailing endpoint path. The client appends /v1/scrape. Configure SCRAIPE_API_KEY securely in the process environment. An API URL is separate from the website you want to extract.

Create a new operation ID for this logical request, retain it with your job record, then run:

bash

export REQUEST_ID="$(python3 -c 'import uuid; print(uuid.uuid4())')"
python3 extract-website-data.py \
  'https://books.toscrape.com/' \
  --request-id "$REQUEST_ID"

The program prints JSON to standard output on success and returns a nonzero exit code on failure. Inspect a successful response before piping it into your storage system. If you retry this exact operation, reuse the saved ID; do not rerun the ID-generation command first. Generate a new ID when you intentionally request a fresh operation.

The request body is:

json

{
  "url": "https://books.toscrape.com/",
  "fields": ["title", "price"],
  "maxTokens": 2000,
  "egress": "direct"
}

It also sends Authorization: Bearer …, JSON content headers and Idempotency-Key. The API reference documents field hints and the alternative JSON Schema input if your application needs another contract. This client is a small product-list example, not a general SDK: changing the requested fields also requires changing its validation rules.

Understand the output

A successful result contains sourceUrl, a UTC retrievedAt timestamp, the operation key, the server's requestId, rows and the API meta object under receipt. The timestamp records when this client accepted the response; it is not the website's publication time or proof of a fresh origin fetch.

The fixture's row looks like this:

json

{
  "title": "Fixture book",
  "price": "£12.50"
}

This is invented test input, not an observed book listing. Real titles, prices, record counts and receipts depend on the source and request. The tests establish local client behaviour against the documented API contract; they are not a live-service extraction result, availability check or speed benchmark.

The validator requires ok: true, truncated: false, a list of records, and nonempty string values for every title and price. It preserves the pound sign rather than coercing a price to a floating-point number with an ambiguous currency. If you need arithmetic, perform a separate, explicitly tested currency and decimal conversion.

An empty list stays empty. A missing field stops the whole result instead of silently dropping a row or inventing a value. Passing these checks proves the data has the required shape; compare a sample with the website to confirm that the title and price refer to the intended product.

Reject ambiguous JSON before validating records

A JSON decoder returning a dictionary is not enough to accept a response. Python's default decoder keeps the last value when an object repeats a member name. For example, {"truncated":true,"truncated":false} becomes a dictionary whose truncation flag is False. Checking that dictionary afterward loses the evidence that the response was ambiguous. A repeated title can likewise silently replace an earlier value.

The official Python JSON documentation also explains that its decoder accepts NaN, Infinity and -Infinity by default. These are not JSON number literals. A different case, 1e400, is a valid JSON number expression but overflows the client's floating-point representation. It must not quietly become an infinite value in a retained receipt.

The refreshed client applies an explicit consumer policy before checking the response envelope:

Input conditionClient behaviour
Duplicate member names at any object depthReject the complete response; never choose the last value
NaN, Infinity or -InfinityReject nonstandard numeric constants
A floating-point number outside the finite supported rangeReject it instead of preserving infinity
Invalid JSON or invalid encodingReport a safe parsing failure; accept no rows
Valid Unicode title or currency textPreserve the decoded string
Response larger than 1 MiBStop before record validation

decode_response uses object_pairs_hook to check names before creating each dictionary, parse_constant to reject nonstandard constants and parse_float to check that a converted float is finite. This strictness is the example's consumer contract. It does not imply every JSON parser rejects duplicate names, or that binary floats are suitable for accounting. Price text remains text; use a separate currency-aware decimal contract when you need arithmetic.

Run the twenty-four tests after downloading both files together. The tests send the malformed bytes from a local HTTP server, so they exercise the actual response-reading and decoding path. These are deliberately constructed failure fixtures, not a claim that the production API emitted any of them. Retest your ingestion code when you change fields or number handling.

Cross-check counts before accepting a complete response

truncated: false is necessary for this example, but it is not the entire acceptance rule. The client sends one request with neither a continuation cursor nor website-following options. For that first, complete response, returned and total must both equal the number of records in data, and cursor must be explicitly null.

The earlier client checked the truncation flag and row shape without checking those metadata fields. A deliberately inconsistent local fixture with one valid row, returned: 1, total: 2 and truncated: false could therefore enter the dataset. The revised example rejects that contradiction and accepts no rows. This is a reproduced client-validation gap; it is not evidence that the live API emitted such a response.

Captured responseDecision for this one-request client
One valid row, returned: 1, total: 1, truncated: false, cursor: nullAccept the checked row
Empty data, both counts zero, no truncation and null cursorAccept an empty list without inventing a product
returned differs from the data length, or total differs from returnedReject inconsistent count metadata
A count is missing, negative, a string, a float or a BooleanReject invalid count metadata
Cursor is missing or is anything other than nullHold the result for inspection; do not guess a continuation

The Python check uses type(value) is int, then requires a nonnegative value and agreement with len(rows). Python's bool is a subclass of int, so an isinstance(value, int) check alone would admit True as a count of one. A decoded 1.0 is also rejected by this strict integer contract even though it compares equal to 1. These are deliberate consumer rules for the documented response shape.

To reproduce the correction, run the downloadable tests. They send contradictory counts, malformed count types and non-null or missing cursors over the local HTTP connection, assert rejection, and check that complete one-row, multi-row and empty fixtures still work. No continuation or automatic retry is sent. When investigating an unknown result, retain its operation key and inspect the response in the playground before importing or repeating it.

This equality rule is scoped to a request starting at the beginning of a result. On a later continuation, total describes the full extraction while returned describes that response window, so they need not be equal. A paginating client needs a separate accumulation contract that checks each window, bounds requests and detects repeated cursors. Do not transplant this one-request rule into a continuation loop unchanged. Even internally consistent counts cannot prove that every product on a website was selected correctly; compare the source and retain its provenance.

Treat incomplete output separately from website pagination

maxTokens bounds the response, not the number of products on a website or the total cost of an extraction. If the API reports truncated: true, this client rejects the partial dataset. Inspect the response and cursor in the playground before adding a continuation loop to your application.

A response cursor continues extraction output from the requested page. Following a website's next-page links is a separate operation, documented as follow in the API reference. Neither a complete response nor a nonempty list proves that you collected the entire catalogue.

When extending this example, bound the number of continuation requests and fetched pages, and track repeated cursors. A cursor changes the request body, so give that continuation its own operation key. Keep each page's source and receipt rather than merging records without their origin.

Handle errors without creating a retry loop

The program reports a safe error reason, request ID and retry delay when available. It does not dump the response body or the API key into an error message. Common decisions are:

ResponseNext step
401Check the key and its permissions before another request.
402Inspect the reason and account/network requirement; repeating the request does not fix a missing prerequisite.
409Distinguish an operation still running from a key reused with a different body.
429Respect retryAfter or Retry-After, then apply a bounded retry policy.
HTML, invalid JSON or a network timeoutTreat the outcome as unknown; preserve the operation key and inspect the request before retrying.
Truncated or invalid recordsFix the collection or validation requirement before importing data.

The current backend scopes idempotency keys to the caller and retains them for ten minutes in process memory. Within that window, the same key and body can replay a stored success without another charge. Failed requests are not stored. Restarts can discard the window, so idempotency is not a durable job ledger or an unlimited guarantee against repeated work.

An interrupted HTTP body is another failure mode. Python can raise an http.client.HTTPException, including IncompleteRead, while reading a chunked response. The example catches these transport exceptions and reports an unknown outcome without a traceback or an automatic retry. If the server already returned an HTTP error status but its body is incomplete or ambiguous, the client retains that status and a safe Retry-After header, while reporting unavailable body metadata as unknown. It does not guess a request identifier or quote a partial error body.

The client sets a 60-second timeout for blocking network operations. Python's timeout is not an end-to-end job deadline, and an upstream host can stop waiting earlier. It accepts at most 1 MiB of response bytes, reading one extra byte to detect overflow. This is a byte cap, not a character count; Unicode characters can occupy several bytes. The same bound and strict decoding apply to error bodies before their metadata is used. A timeout or size rejection does not establish that the server cancelled its work.

Redirects from the API origin are rejected so the client cannot forward the bearer key to another host. Use the final trusted API origin directly. Remote API connections require HTTPS; HTTP is accepted only on loopback for local tests.

Check the receipt before increasing volume

The example selects egress: direct, so it does not opt into a paid residential fallback. A blocked site can therefore fail. Direct fetching does not make extractor compilation free: the API describes compiled-fresh, healed and llm as billable tiers. Inspect receipt.credits.compile, receipt.credits.egress and receipt.credits.total for the successful request rather than inferring price from the number of output rows.

Start with one page in the playground, verify a few records against the source and read the receipt. Then decide whether you need pagination, another schema or the Markdown reader. Keeping this first extraction small makes failures and unexpected data easier to diagnose.

Sources and verification

Try the next step

Try the playground

Continue building