Bound the size of web content for an agent

Budget the document payload before calling the model

A web reader's output limit and an agent's model context limit measure different things. Set both. Use the reader limit to bound conversion output, then measure the exact text you intend to send with the tokenizer appropriate to your model. Leave room for instructions, history, tool definitions, framing and the answer.

This guide provides an offline whole-document packer. It preserves selected documents unchanged, includes their IDs and URLs in the measured JSON payload, and reports every document it skips. It makes no model or Scraipe API calls. The small budget and documents are invented fixtures, not a recommendation for any model's context window.

Distinguish characters, bytes and tokens

Scraipe's current Read implementation measures content.length in JavaScript UTF-16 code units. Its maxChars truncation uses slice(0, maxChars), and stats.estimatedTokens uses Math.ceil(content.length / 4). That estimate is useful as a rough signal; it is not an exact tokenizer result or a guarantee that a request fits.

These locally measured strings illustrate the difference. Counts use tiktoken@1.0.16, the explicit cl100k_base encoding, and ordinary-text treatment of special-token-looking strings:

These are exact observations for these strings and this encoding, not a language-wide ratio or performance benchmark. cl100k_base is a reproducible example choice; check the actual encoding and request accounting for your chosen model. Do not transfer these counts to another tokenizer.

A character slice can also break a Markdown fence, table or Unicode surrogate pair. If Read reports stats.truncated: true, do not quietly relabel the result complete. Select a smaller source region, recapture within a suitable bound, or deliberately create a derived chunk with its own provenance and completeness policy.

Reserve space before selecting documents

The example uses this accounting:

available document tokens = contextTokens
  - reservedOtherInputTokens
  - reservedOutputTokens
  - safetyTokens

reservedOtherInputTokens must cover everything outside this document payload: system instructions, conversation history, tool schemas and provider framing. reservedOutputTokens is your intended output allowance. safetyTokens provides an explicit margin for uncertainty. These values are caller policy; the packer cannot infer a provider's limits or verify that your reservations are adequate.

In the fixture, 256 minus 64 minus 64 minus 16 leaves 112 tokens for the document envelope. Those intentionally small numbers make refusal easy to reproduce. They do not configure a model or enforce a provider's output setting. Separately set and validate those settings in your real request.

Run the whole-document packer

Download all five files into a new directory:

• Pinned package manifest
• Registry-only dependency lock
• Token-budget packer
• Fourteen tests
• Invented document fixture

With Node 22 or later:

npm ci --ignore-scripts
npm test
node pack.mjs fixture.json

Installation downloads the pinned tokenizer package from npm. Once installed, the test suite and packer run locally without credentials or network requests. The package contains the encoding data and WebAssembly tokenizer. The example frees the encoder after each packing operation as described in the package's primary documentation.

The fourteen tests pass on Node 22.23.2. They exercise exact budget boundaries, metadata overhead, skipping oversized documents, an empty selection, Unicode, ordinary handling of token markers, invalid inputs, bounded file loading and standalone CLI behavior.

The fixture produces this summary:

{
  "encoding": "cl100k_base",
  "availableTokens": 112,
  "usedTokens": 66,
  "selectedIds": ["intro", "limit"],
  "skipped": [
    {"id": "long", "reason": "whole-document-does-not-fit"}
  ]
}

The full output also contains payloadText: the exact JSON string that was counted. It includes each selected document's id, url, format and content. The summary printed around that string is diagnostic output; do not send the entire diagnostic result and assume it is still 66 tokens. Adding labels, changing whitespace or serializing it differently changes what needs to be counted.

Understand the selection policy

Input order is priority order. The packer tries each document by serializing the complete candidate envelope and counting that candidate. If it fits, the document is retained in full. If it does not fit, the document is skipped and the next one is tried. That is why the oversized middle fixture does not prevent the smaller final document from being selected.

Counting the whole candidate matters: tokens at text boundaries and JSON punctuation do not necessarily add up like independent content-only counts. The measured payload includes source metadata, quotation escaping and envelope syntax. The result guarantees only that this exact payloadText fits the supplied document budget under this encoding.

This algorithm is deterministic and easy to inspect, but it does not rank relevance or maximize the amount of useful evidence. Choose priority order before packing. A high-value oversized document may deserve a separately prepared section rather than being skipped. The example limits input to 100 documents and 1 MiB of UTF-8 JSON; it repeatedly tokenizes candidates and is not a large-corpus ingestion engine.

Reject ambiguous captures and expose omissions

Each document needs a unique nonempty ID, an absolute HTTP(S) URL without embedded credentials, Markdown or text content, and explicit truncated: false. Blank content, unpaired Unicode surrogates, invalid budgets and duplicate IDs are refused. The CLI rejects malformed UTF-8/JSON and oversized input with fixed messages rather than echoing the capture. It leaves the input file unchanged.

The completeness flag is an assertion supplied by your capture process, not proof that the origin page was fully or correctly read. Before adapting a hosted response to this schema, apply the Read acceptance checks and your own source policy. Carry the document ID and URL from a source-preserving record. The packer validates URL syntax but does not approve an origin, fetch a URL or verify the text's truth.

An empty selection is a valid result if the empty JSON envelope fits. Treat it as “no usable evidence selected”; do not ask the agent to invent an answer. If even the empty envelope cannot fit, the packer fails. Record skipped IDs and reasons so a citation or coverage gap can be investigated.

Check the final request, not just this component

For production use, adapt the encoding to the target model and count the fully assembled request using the provider's documented accounting. The local count does not include chat-message wrappers, tool-call framing or hidden provider behavior. A safety margin cannot turn the wrong tokenizer into the right one.

Keep retrieved content in a clearly identified evidence field. Treat special-token-looking strings as ordinary text, as this example does, but do not confuse token handling with prompt-injection protection. Source content remains untrusted; tool permissions and instruction boundaries need separate controls.

Start with a permitted page through the Markdown conversion workflow, accept and retain its source record, then prepare the model payload. A hosted Read request requires appropriate credentials and follows the API contract; the offline packer adds no API call or credit charge. Use its exact output, omissions and measured limits to make the next agent step explicit and inspectable.