Map a site before crawling

Discover the URL set you intend to visit

Map a site when your first question is which URLs might belong in a job. Review that inventory before asking a crawler to fetch page content. A sitemap can expose pages that are absent from the starting page's navigation. A link fallback can reveal a useful starting set when a sitemap is unavailable.

Start with a permitted site and a concrete scope, such as public documentation under one origin's /docs/ path. Write down excluded areas, how you will treat query parameters, and how many pages you will inspect in the first run. A count of discovered URLs alone does not tell you whether the pages are relevant, accessible or complete.

This guide includes an offline planner with an invented Map response. It selects two candidate URLs and records a decision for every input row. The hosted request examples show the current API contract. The reproducible tests use local fixtures.

Request a bounded map

Scraipe's POST /v1/map requires a key with read scope. Its limit defaults to 500 and supports up to 5,000 returned URLs. Begin with a modest positive integer. Replace the domain below with your permitted target, set SCRAIPE_API_KEY, and save the response to a new local file.

curl --fail-with-body --silent --show-error \
  https://scraipe-fc66a552921e.herokuapp.com/v1/map \
  -H "Authorization: Bearer $SCRAIPE_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"url":"https://example.com/","limit":50}' \
  -o map-response.json

Keep the request settings beside the response. Check the HTTP result and ok before processing urls. An error body saved by curl needs investigation. It cannot be used as a successful inventory.

The current Map implementation first tries sitemap locations declared in robots.txt, or conventional sitemap paths when none are declared. It uses the first candidate that yields URLs. If that produces no URLs, it tries links on the requested page. This may involve several requests, particularly for sitemap indexes or unsuccessful attempts.

Map is free under the current Scraipe contract and uses direct networking. Reading a sitemap still consumes origin requests and time. Refer to the API reference for current scope, response and pricing details, and retain actual request receipts for later jobs.

Interpret a small result carefully

A response with truncated: false does not establish complete site coverage. The current discovery code bounds candidate collection before filtering. It can stop at the first nonempty sitemap, and link fallback examines a bounded set of links from the starting page.

A local reproduction used a five-URL sitemap with four news URLs followed by one /docs/ URL. Mapping with limit: 2 and search: "/docs/" returned zero matches and truncated: false. Repeating against the same fixture with limit: 5 returned the documentation URL. A second experiment declared two sitemap locations and found that a nonempty first sitemap prevented inspection of the second. These results came from the repository's discovery code with local XML and transport stubs.

If a narrow search returns too little, compare a broader map, inspect the site's published sitemap structure and check a known relevant page. The current search option performs a case-insensitive literal match on a URL or available link text. It does not traverse the site to discover every page matching that string. No Map pagination cursor is exposed.

The Sitemap protocol supports indexes and optional modification metadata. Inspect those records when investigating gaps, and preserve the publisher's values alongside your own observation time. A sitemap entry is not a fresh measurement of page availability. Sitemap protocol

Review a saved map locally

Download these five files into one directory.

• Offline planner
• Ten tests
• Invented Map response and policy
• Bounded JSON loader
• Loader dependency

Use Node 22. There are no package installation steps, network calls or API charges for these commands.

node --test plan-map-crawl.test.mjs
node plan-map-crawl.mjs map-crawl.fixture.json

Expect ten passing tests. The planner's output includes this selected list.

{
  "coverage": "unproven",
  "selected": [
    "https://example.com/docs/intro",
    "https://example.com/docs/setup"
  ]
}

The full report also contains the input count, reported total, truncation flag and ten row decisions. Each row's zero-based index points back to the saved input. Fragment variants collapse into one candidate. Further eligible URLs are deferred after the policy's two-URL cap. Query variants and encoded paths are held for review. Other origins, paths outside /docs/, and malformed URLs receive explicit dispositions.

For your own saved response, copy the fixture's envelope and replace request, response and policy. Keep request.limit equal to the value actually submitted. Choose policy.origin, a simple slash-terminated pathPrefix, and maxUrls before examining untrusted candidates. The example accepts a maximum of 100 selected URLs and a 1 MiB input file.

Try changing response.total from 10 to 9 while leaving all ten rows. The planner exits with status 1 because the envelope is inconsistent. Exit 0 means a planning report was produced, including when the selected set is empty. Read the dispositions before taking action.

Preserve URL meaning while narrowing scope

Exact origins include the scheme, host and port. https://docs.example.com, http://example.com and https://example.com:444 are different from https://example.com. Map can include subdomains when requested. The local planner applies its own exact-origin policy, so that broader discovery does not silently expand the intended job.

The example keeps query URLs for manual review rather than dropping their parameters and merging them. Language, edition, page number and filter parameters can select different content. It also preserves trailing-slash variants as separate candidates. Scraipe's current crawler normalizes trailing slashes during traversal, so inspect important alias cases before relying on its deduplication.

The URL parser normalizes addresses, and the helper removes fragments for its candidate identity. Those choices need to fit the target site. Keep the saved raw inventory so a later investigation can recover the original strings. For the corresponding content records, use the source-URL preservation workflow.

Turn the review into a small crawl

Inspect the selected pages first. Confirm that they contain the expected material, identify layout differences and define acceptance checks for the resulting content. The sample-selection guide helps choose pages that exercise different conditions.

A separate POST /v1/crawl requires scrape scope, even in read mode. It accepts one starting URL and URL globs. For a permitted documentation site, a deliberately small request body could be:

{
  "url": "https://example.com/docs/intro",
  "mode": "read",
  "format": "markdown",
  "limit": 5,
  "maxDepth": 1,
  "include": ["https://example.com/docs/*"],
  "exclude": ["*?*", "https://example.com/docs/print*"],
  "egress": "direct"
}

Review the globs against the target's actual addresses. The starting URL must satisfy them too. This crawl performs its own sitemap seeding and link discovery, so it can find URLs beyond the saved Map response. The returned-page limit is not an exact prediction of origin request count, particularly when pages fail. If your workflow requires visiting only an explicit saved list, process that list with individually bounded Read requests instead.

Direct-network read mode has no conversion credit charge under the current contract. Extraction and residential networking have their own charging rules. Keep the pilot's receipts and useful-output count before estimating a larger workload.

A successful submission returns a job ID and polling path. Poll GET /v1/jobs/:id through a terminal status, then inspect the page results and collected errors. A queued acknowledgement has no extracted content to validate. Handle credential errors, rate limits and site opt-outs according to the returned reason. Keep failed or incomplete pages out of the accepted corpus until reviewed.

Save the request, discovery source, local policy, selected candidates and crawl outcome together. This lets the next run explain whether a difference came from discovery, scope selection or the pages themselves.