Preserve source URLs in a reader pipeline

Keep the source attached to the content

A reader pipeline should store the URL it requested, the URL returned with the content and the content itself in one record. If a page redirects, those URLs can differ. Saving only a Markdown string loses the information needed to cite the source, inspect a changed page or decide whether two captures belong together.

Scraipe's Read response exposes its returned page URL as url. The implemented route uses the fetcher's final URL when available and falls back to the requested URL. It also supplies content, format, metadata and a request receipt. Preserve your original request separately; do not assume the response can reconstruct the entire redirect chain.

This guide builds a provenance record from an already captured response. Its example is offline and invented. It does not fetch a website, authenticate a key or claim to have observed a real redirect.

Separate the fields that answer different questions

observedAt is a client observation time. It is not the page's publication date, the origin's last-modified value or a server-certified retrieval time. Record it when capturing the response; do not invent it later from the filename. Likewise, a content hash supports comparison, not proof that the publisher is trustworthy or that a page was fetched at a particular time.

Reproduce a source record locally

Download these files into the same directory:

• Source-record builder
• Thirteen tests
• Invented capture and policy
• Read acceptance helper
• Bounded capture loader

Use Node 22 or later. No npm installation, API key or network access is needed:

node --test preserve-read-source.test.mjs
node preserve-read-source.mjs read-source.fixture.json > source-record.json

The test suite expects thirteen passing tests. The second command writes a JSON record to stdout; shell redirection creates your chosen local file. Choose a new output path because shell redirection can replace an existing file. This command does not modify the input capture.

The fixture models a request to one address and a response from another approved origin:

{
  "requestedUrl": "https://example.com/old-guide#install",
  "returnedUrl": "https://www.example.com/guide?lang=en&edition=2#install",
  "sourceUrl": "https://www.example.com/guide?lang=en&edition=2",
  "requestId": null
}

This excerpt omits the content and hash fields. The full output preserves the content string and adds two SHA-256 identifiers. requestId is null because the fixture contains no real receipt. An unrelated canonical-like field in the fixture is deliberately ignored; the builder does not let an arbitrary page hint replace the returned source.

Make origin acceptance explicit

The input contains requestedUrl, response, observedAt and allowedOrigins. For the example's cross-origin scenario, the policy explicitly allows both https://example.com and https://www.example.com. If the final origin is missing from that list, the builder refuses the record. It also checks the requested origin.

For your own job, set the policy before ingesting the response. Do not build an allowlist by copying whatever host the response happened to contain. A deliberate cross-origin redirect may be acceptable, but a change to an unrelated sign-in or error page is a reason to inspect the result.

The builder accepts absolute HTTP(S) URLs without embedded credentials, control characters, whitespace or backslashes. The existing Read acceptance checks additionally require successful, nonempty Markdown or text with explicit stats.truncated: false. Missing completeness evidence is refused. The bounded loader requires valid UTF-8 JSON of at most 1 MiB.

These are consumer checks. They do not verify a redirect chain, DNS ownership, a site's access policy or the truth of its content. A valid source URL attached to the wrong page text is still an incorrect capture.

Preserve queries and anchors deliberately

The example keeps the raw requested and returned strings, then uses Node's WHATWG URL implementation to produce sourceUrl. Serialization can normalize host case and default ports. The builder removes the fragment only from that identity address; the original anchors remain in the raw fields for later citation or debugging.

It does not remove query parameters or sort them. A language, edition, search query or repeated parameter can identify different content. A universal “remove everything after the question mark” cleanup would merge unrelated documents. Query URLs may also contain private values, so retain them with the same access controls as the captured content and redact deliberately when publishing citations.

Node documents separate hash, search and searchParams properties, including serialization differences when mutating query parameters. The URL reference is useful when adapting this policy. Do not mistake a convenient normalization rule for proof that two URLs are equivalent on the origin server.

Keep identity separate from deduplication

The builder hashes the exact content bytes into contentSha256. It then hashes a JSON array containing the normalized source URL, returned format and content hash into documentId. A changed source, format or content changes the document identity. A different observation time or fragment alone does not.

This is one explicit versioning policy, not Scraipe's server-side identity contract. The same text at two source URLs has the same content hash but different document IDs. You can use that relationship to investigate duplicates while retaining each source's provenance. Do not discard one source automatically just because the text matches.

When you split content for retrieval, carry documentId and a reference to the source record on every chunk. Give chunks their own IDs and keep enough information to recover the corresponding text span. If a later step rewrites or summarizes the text, treat that as a derived artifact and retain its relationship to the original capture rather than reusing the original content hash.

Apply the record to a hosted Read response

Use the Markdown conversion workflow to capture a permitted page through POST /v1/read. At capture time, store the original request URL and a UTC timestamp such as new Date().toISOString(). Place the response object and your preselected origin policy in the input envelope, then run the builder.

A hosted request needs a key with read scope and remains subject to the API contract. Direct-network reading has no conversion credit charge under that contract; residential networking is separately metered. This offline metadata step adds no API call or credit charge. Preserve the actual receipt when investigating costs; the builder copies meta.requestId when supplied and never invents one.

If a capture fails validation, keep it out of the accepted index and inspect the reason. Do not silently substitute the requested URL for a missing returned URL, accept an unexpected origin or fill in a plausible timestamp. Once a valid record is stored, its source and version remain available even after the content leaves the original request process.