Convert a website to Markdown

Convert one page first

To turn a web page into Markdown, send its URL to Scraipe's POST /v1/read endpoint with format: "markdown", then save the response's content string. Check the returned format and truncation flag before naming the file .md. Saving the entire JSON response produces a JSON document containing Markdown, not a Markdown document.

One Read request converts one fetched page. Converting a website with many pages requires a separate decision about which URLs to include; the endpoint does not automatically walk every link. Start with one permitted public page, inspect the output, and expand coverage only after the result suits your application.

This guide includes a local conversion fixture and a tested export command. The fixture was converted by Scraipe's repository reader on 23 October 2026; it is not a captured hosted API request or evidence of a customer result.

Send an explicit Markdown request

Create a Scraipe key with the read scope in the console. Keep it in your local secret-management environment as SCRAIPE_API_KEY. The example uses Node 22 for the export step and a recent curl supporting --fail-with-body. Choose a new working directory so you can retain the response for inspection.

curl --fail-with-body --silent --show-error --max-time 25 \
  'https://scraipe-fc66a552921e.herokuapp.com/v1/read' \
  -H "Authorization: Bearer ${SCRAIPE_API_KEY}" \
  -H 'Content-Type: application/json' \
  --data '{
    "url": "https://example.com/",
    "format": "markdown",
    "mainOnly": true,
    "includeLinks": true,
    "maxChars": 20000,
    "egress": "direct"
  }' \
  --output read.json

Run the export only after curl exits successfully. An HTTP error can leave an error body in read.json, and a transport failure can leave an incomplete capture. The example does not retry automatically. A client timeout does not prove the server stopped processing the request.

example.com is an illustrative public input, not an assertion that our publication run fetched it through the hosted API. Replace it with the page you have permission to process. The API reference defines authentication, refusal reasons and request limits.

Read's direct-network conversion uses no language model and has no conversion credit charge under the current contract. That is not an unlimited-throughput promise: concurrency, queue, access and size limits still apply. Residential reads have separate network charges; this procedure neither enables nor verifies residential service. Inspect meta.credits.total and the request receipt guide when checking a hosted run's actual cost.

Export the content without overwriting a file

Download these four files into one directory:

• Markdown exporter
• Existing Read-response acceptance helper
• Exporter tests
• Local Read-result fixture

No npm installation or API key is needed for the local example. Using Node 22 or later, run:

node --test export-read-markdown.test.mjs
node export-read-markdown.mjs website-markdown-read.fixture.json fixture.md

Expect fourteen passing tests and a small JSON status with written: true, format: "markdown" and the UTF-8 byte count. Open fixture.md; the file itself contains the Markdown, without JSON quotation marks or escaped newline characters. To export your own successful response instead:

node export-read-markdown.mjs read.json page.md

The exporter requires ok: true, format: "markdown", nonempty content and stats.truncated: false. It reuses the Read acceptance checks, then writes the exact content string. It does not add a title, strip formatting, infer missing text or manufacture a receipt. Keep the original JSON alongside the Markdown when you need its source URL, metadata, links and request receipt.

The destination must be a new .md path. Existing files and symbolic links are refused. The script prepares the content in the destination directory and uses a hard link to install it without replacing an existing target; your filesystem must support that operation. The input is bounded to 1 MiB of valid UTF-8 JSON. A larger capture is refused by this example, even if the API accepted the original request.

Diagnostics omit captured content and input paths. The script reads local files only; it does not authenticate, fetch a page or validate the truth of the page's claims. Its acceptance checks also do not certify formatting fidelity, correct main-content selection or source-origin policy.

Compare the conversion with its source

The invented HTML fixture contains prose, headings, a list, a relative link, a code block, a table and labelled boilerplate. Our local reader run produced Markdown with this structure:

# Deployment notes

## Before release

-   Check the output format.
-   Inspect the saved Markdown.

Read the [reference guide](https://example.com/reference) before proceeding.

This is a shortened excerpt; the downloadable fixture contains the full result. The code block retained its JavaScript fence and contents. The navigation, sidebar, footer and script sentinels were absent. Those are measurements of this deliberately small fixture, not a guarantee for every website.

The table exposed a useful limit. The current reader emitted these rows:

| Stage | Result |
| Local fixture | Review manually |

They preserve the cell text but lack the delimiter row required by the GitHub-flavoured Markdown table specification. Do not assume that a .md filename guarantees a correctly rendered table. For critical tabular data, use structured extraction and validate the cells, or explicitly transform and test the table for your chosen renderer.

Scraipe's reader uses Turndown with its own content-selection and conversion rules. Turndown's primary documentation describes configurable heading and code-block styles and optional GFM plugins. A capability available through a library plugin is not proof that a deployed service enabled it.

Adjust selection without promising a page clone

If expected prose is missing, compare the returned content with the original page before changing limits. mainOnly: false broadens the conversion to the cleaned page, but it is not an archive of the original HTML. In the local fixture, navigation remained removed even with that option. Scripts, forms and interactive behavior do not become working Markdown features.

With includeLinks: false, the local reader retained the reference link's visible words but removed its Markdown destination. Use that deliberately; losing destinations can make a document less useful outside its original page. Images, complex layouts, code languages and nested tables deserve separate inspection when they matter to the consumer.

A page that needs browser rendering or sign-in may not provide the same content as a simple public page. Inspect the returned refusal or available rendering options. A login page or consent notice that converts successfully still may not be the content you intended to read.

Stop on partial or mismatched output

A successful HTTP status does not establish a complete Markdown document. Handle these cases before ingestion:

The reader can cut content inside a code block or table when a character limit applies. Our forty-character fixture run reported truncation; the exporter refuses that state. Conversely, truncated: false only says this size limit did not cut the returned content. It does not prove that every relevant section of the website was selected.

Treat downloaded page content as untrusted data. Keep instructions from the page separate from instructions to an agent, and use your renderer's appropriate safety settings when displaying Markdown.

Expand to a website with an explicit URL list

After one page passes inspection, choose the intended site sections and bounds. The API's Map operation can help discover URLs; Crawl can process a bounded set. Their results and permissions differ from a single Read response, so do not feed an entire crawl-job JSON object into this exporter. Review the API contract, define exclusions and check coverage before treating the output as a website copy.

For a first trial, open the playground, compare one permitted source page with its reader output, and save only the representation your application actually needs.