Include and exclude URL patterns

Suppose an agent needs documentation under /docs, including the section home, but should skip print copies and search results. An include pattern ending in /docs* admits /docs-old too. Changing it to /docs/* fixes that mistake and introduces another: the section home no longer matches. Use two include rules, then test both a page you want and its nearest unwanted neighbour before starting a job.

Write the policy as examples first

Pick a site you have permission to crawl. Write down a few real addresses and decide which you want in the corpus before writing any globs. Start with the seed, one useful child, a deeper child, a sibling section with a similar name, a print view and a query variant. Use the Map workflow for candidates and add negative cases even when they are missing from the map. A sample containing only pages you want cannot reveal an overly broad rule.

For the example policy below, /docs/printing explains how to print; keep it. The separate /docs/print section contains duplicate print views; drop the section home and its descendants. Treat every query as out of scope for this pilot. On a site that serves languages or editions through parameters, that last decision would lose useful pages.

Match the URL the crawler actually sees

Scraipe parses an absolute URL, strips its fragment and removes one trailing slash from a non-root pathname before testing the filters. The same-origin check comes first. Of eligible same-origin URLs, any matching exclusion wins. When an include list is present, at least one include must then match. This applies to the seed, sitemap candidates and links discovered later.

A pattern matches the entire normalized URL, including scheme, host, path and query. Matching is case-insensitive. The only wildcard is *: it can consume zero characters, slashes and query text. A question mark means a literal question mark. Square brackets are literal too; these are neither regular expressions nor shell filename patterns. An exact pattern contains no wildcard. Omit a filter you do not use. In the current implementation an explicitly empty exclude array can exclude every candidate.

Use a concrete origin in rules you intend to review or reuse. A subdomain, a different scheme or a nondefault port changes the origin. The URL Standard describes parsing and serialization; Scraipe adds its own fragment and trailing-slash treatment. Rules match the serialized string, not the resource's meaning. If a server treats two spellings as aliases, test both. In particular, a percent-encoded path segment may stay encoded. The fixture deliberately exposes such a case: treat it as a gap to review if your site resolves it to an excluded resource.

Run the policy against a labelled matrix

Download the Node checker and 12-case URL matrix into one directory. The fixture uses invented addresses. Run it on Node 22; no account, installed packages, network access or credits are required.

node check-crawl-patterns.mjs crawl-pattern-cases.json

The fixture returns pass: true with six eligible rows and six rejected rows. Each row lists its normalized URL, expected decision, actual reason and the first relevant matching pattern, where applicable. A pass means the observed decisions match the labels. It says nothing about whether those labels are the right policy for another site.

For a negative control, copy the JSON and replace the two include patterns with ["https://example.com/docs*"]. Leave the labels unchanged. The checker exits 2 and marks the /docs-old row as a mismatch. Correct the rule before changing any expectation. Exit 1 means the input itself is invalid, for example an empty filter or no positive seed case. Distinguishing those exits prevents a broken fixture from looking like a policy regression.

The checker is a local preflight for absolute HTTP(S) examples and printable ASCII patterns. Its fixture was compared against the repository's normalization and matching functions under five filter configurations. It does not discover new URLs, evaluate redirects, fetch pages or inspect any server's routing. The matrix is evidence about these cases, not an access-control boundary.

Submit a small read pilot

Save the following body with your permitted target in place of the example host. Keep the same seed and filter strings you tested. Submit it once to POST /v1/crawl with a key that has the scrape scope, then poll the job using the API reference. The example here is a request template; the fixture test does not start a hosted job.

{
  "url": "https://example.com/docs",
  "include": ["https://example.com/docs", "https://example.com/docs/*"],
  "exclude": ["https://example.com/docs/print", "https://example.com/docs/print/*", "*?*"],
  "mode": "read",
  "egress": "direct",
  "limit": 5,
  "maxDepth": 1
}

The page cap and depth bound address different questions from the filters. A matching page may never be discovered, may fail to load or may be left out when the result limit is reached. A filtered intermediate page also cannot contribute links through its body, although the sitemap may supply its descendants separately. See Limit crawl depth and page count before increasing either bound.

Diagnose a missing or unexpected page

If the job returns no pages, check the normalized seed first. Test for an empty exclusion list, a missing exact section-home rule and a host or port mismatch. Next look at the job's errors and discovery results. Making the pattern broader cannot fix a refused fetch.

If a page slips in, add its exact returned spelling to the matrix with the decision you wanted. Inspect which rule admitted it, then add a nearby page you must keep before tightening the rule. For the print example, replacing the two exclusions with */print* would also discard the useful printing guide. Keep the failing matrix alongside the fix so the next policy change cannot silently reintroduce the problem.

Direct Read pages do not incur compilation or residential credits under the current contract. Scope still matters: unwanted pages consume time and downstream indexing work. Compare the finished URLs with your reviewed policy before importing their content. Retain the request and job together; if an exact list is required, review that list and process its entries individually instead of inferring exact membership from a crawl.