Remove navigation noise without losing article content

Check what was preserved, not just what disappeared

Removing navigation is useful only when the output still contains the material your application needs. A short, clean result can be worse than a noisy one if it loses a deployment warning, the article heading or a source attribution. Define both sides of the task before changing reader options: required content must survive, and identified noise must disappear.

This guide compares Scraipe's main-content and whole-page modes on four invented HTML documents. It includes the source HTML, captured local outputs and a runnable comparison. The observations use the repository reader on 28 October 2026 with Node 22.23.2; they are not authenticated hosted requests, customer results or a claim about every website. The downloaded comparison requires Node 22 and makes no network requests.

Understand where selection loses information

The current reader applies several operations in sequence:

1. Read available page metadata.
2. Remove certain element types, including nav, aside, header and footer, as well as scripts, styles and forms.
3. With mainOnly: true, choose a likely main container and remove descendants whose class or ID resembles boilerplate.
4. Convert the remaining content and apply the requested character limit.

The important consequence is that mainOnly: false skips main-container selection and the class/ID cleanup, but it still receives the earlier element removal. It returns a cleaned page, not the original document. Neither text output nor HTML output restores nodes already removed.

HTML semantics make this distinction matter. A header can introduce an individual article, and a footer can contain information about its enclosing section. Those elements are not always a global masthead or site-wide navigation. The HTML specification describes those roles. A blanket tag rule can therefore remove legitimate content even when the page is valid HTML.

Similarly, an article section with class="related" may hold a necessary constraint rather than a recommendation rail. A class name is a heuristic signal, not a statement that the text is disposable.

Compare the same source with explicit requirements

The fixture contains four independent cases. Each has a literal contract written from the invented source, two measured outputs and explicit expected decisions. Requirements cover known content markers; exclusions cover known noise markers. These are deliberately small diagnostic cases, not a benchmark dataset.

All eight captured outputs have format: "markdown" and stats.truncated: false. That does not make the rejected selections complete for the reader's task. Truncation describes the output size limit, not whether the source selection retained every important section. stats.extractedVia records the selection method; it is not a quality score.

The third case is especially useful: recovering one missing section does not automatically make the broader output acceptable. Inspect both the required-content checks and the noise checks rather than selecting whichever response is longer.

Run the comparison offline

Download these six files into a fresh directory:

• reader-selection.fixture.json
• compare-reader-selection.mjs
• compare-reader-selection.test.mjs
• check-markdown-contract.mjs
• export-read-markdown.mjs
• audit-read-result.mjs

Run:

node --test compare-reader-selection.test.mjs
node compare-reader-selection.mjs reader-selection.fixture.json

Expect twelve passing tests. The report starts with these values:

{
  "version": 1,
  "caseCount": 4,
  "mismatches": 0,
  "casesWithoutAcceptedMode": 2
}

Each case includes separate main and whole acceptance reports, named rule results, content digests and eligibleModes. The sibling-context case permits only whole; the last two cases have an empty eligible list.

Exit 0 means the measured decisions match the saved expectations. It does not mean that all documents are suitable for ingestion: two intentional failures are part of this passing regression suite. Exit 1 means at least one decision changed. Exit 2 means the file, contract or case structure is invalid. Do not rewrite an expected rejection to make a changed result look correct.

The comparator reuses the conversion acceptance checker, including exact returned URL, complete Markdown and literal-rule checks. It reads a version-1 bundle of at most 1 MiB, containing 1–32 cases. Reports contain short rule/case identifiers and digests, not the source HTML, URLs or article prose. Choose identifiers that do not reveal private information. Store underlying captures according to their sensitivity.

The bundle includes raw invented HTML so you can inspect what the requirements refer to. The standalone comparator audits the stored outputs; it does not embed or rerun the Scraipe reader. Its ok and url fields are an explicitly offline response envelope, with no fabricated request ID or billing receipt.

Apply the method to one permitted page

Begin with a source you are permitted to process. Independently inspect its heading, important sections, warnings and attribution before writing an acceptance contract. For example, a deployment note might require its rollback procedure while excluding a known newsletter prompt. Preserve the requested and returned source addresses for both captures.

The Read API supports these request-body options:

{
  "url": "https://example.com/your-permitted-page",
  "format": "markdown",
  "mainOnly": true,
  "includeLinks": true,
  "maxChars": 20000,
  "egress": "direct"
}

Replace the placeholder with the actual permitted source and use a server-side key with the appropriate Read permission. This is a request recipe, not a live request performed by this guide. Save the complete response, then make a separately recorded comparison request with mainOnly: false if the narrow selection is missing material. Both reads must pass transport, source-identity, representation and completeness checks before their content is considered.

Keep the source stable where possible. Two live requests can observe different page revisions, so a difference is not automatically caused by the selection setting. Retain retrieval context and compare against the source version actually seen. The fixture avoids that ambiguity by running both settings on identical local HTML.

Direct Read does not require model compilation; network choices and any later extraction are separate cost decisions. Consult the current pricing and request receipt. The offline files here spend no Scraipe credits. Do not enable residential fallback merely to investigate a content-selection problem.

Choose the next action from the failure

If main mode passes the declared requirements, prefer that smaller accepted scope for this task. If only whole mode passes, preserve it and account for the larger input using the content-budget guide. A passing finite rule set still needs periodic source comparison; it cannot detect losses or noise that were never specified.

If whole mode restores a required section but retains known noise, reject that candidate under the current contract. For a stable permitted source, consider a separately tested structured extraction that targets the required fields or sections. Read and Extract serve different jobs; Read does not expose a custom keep-selector or restore-deleted-tags option in its documented request contract. Do not invent one in the request body.

If neither mode contains the required heading or attribution, compare the original source structure. Increasing maxChars cannot restore nodes removed before conversion. If you control the page, assess its markup deliberately; otherwise choose another permitted source representation or a tested extraction workflow. Avoid global string replacement as an automatic cleanup fix: the same phrase may occur in both navigation and legitimate prose.

Handle a plain-text conversion fallback separately with the representation guide. Treat a login page, consent screen or access refusal as a source/access problem, even if it returns readable text. Neither changing selection nor relabeling text proves that the intended article was obtained.

Keep page instructions separate from agent instructions, and apply appropriate rendering safeguards to all captured HTML or Markdown. Selection checks establish only the declared content rules. They do not establish factual accuracy, publisher authenticity, safe execution or full visual fidelity.