Separate missing values from empty strings

Separate observation from update intent

A missing value, an empty string and a request to delete a stored value are different events. When a fresh extraction omits a description, it may mean the selector failed or the page did not load. Automatically replacing the stored description with blank text loses useful data. Automatically treating every null as a delete is equally dangerous.

This guide focuses on partial updates to an existing record. Start with the missing-field validator if you need to classify or validate an extraction. Here, the question is what an already-inspected observation is allowed to change. The downloadable local planner preserves absence, accepts explicitly allowed empty values, and blocks ambiguous updates before returning a candidate record.

The three-file example uses invented product records and sixteen tests, verified with Node 22.23.2 on 20 October 2026. It has no packages, network calls or database writes. A ready result is a proposed value to inspect, not an instruction to write it to production.

Define a field policy before merging

The example uses a small flat schema: required nonblank title, required nonnegative integer quantity, required boolean available, and optional string description. Only description permits an empty string. The current stored record must already satisfy that schema.

Empty-string policy belongs to each field. A blank title is not useful in this schema, while an empty description can be a valid observed value. If your application treats whitespace-only descriptions as blank, add that rule deliberately and retain the raw observation separately. Do not silently trim every string during merging.

Keeping a value when it is absent avoids one form of accidental deletion, but it can leave stale data in place. Store observation time, coverage and a stale-or-unknown state separately. Absence does not confirm that the previous value is still current. This example intentionally does not invent a freshness policy.

Avoid defaults that erase the distinction

This assignment looks convenient but loses observed falsy values:

next.quantity = incoming.quantity || current.quantity;
next.available = incoming.available || current.available;

A quantity of zero is replaced by the old quantity, and false is replaced by the old availability. MDN documents that || chooses the fallback for any falsy value. ?? preserves zero, false and empty strings, but still combines null and undefined into the fallback case. That is insufficient when explicit null needs review or has distinct clear semantics.

Use presence checks and a field policy instead. The planner uses Object.hasOwn(incoming, field) before checking value and type. It does not spread all incoming properties into the current record. Unknown keys, including prototype-sensitive names in parsed JSON, produce review decisions and no candidate. Its API is intended for ordinary JSON objects, not arbitrary JavaScript objects with getters or proxies.

Understand the destination patch contract

RFC 7396 defines JSON Merge Patch: absent members leave the target alone, an object member with null removes that member, and an empty string replaces a value with an empty string. Arrays are replaced as a whole; they are not merged item by item. The RFC also explains why this format is unsuitable for some documents that use explicit null as stored data.

Those semantics do not mean a null in a scraped observation is permission to delete. A destination API's declared patch format and your application's authorization to change a value are separate requirements. Do not send a scraped object directly as a merge patch just because both are JSON.

The downloadable planner is not an RFC 7396 implementation. It is a bounded local policy for four flat fields. It refuses nested values and accepts explicit removal only for optional description. To distinguish observation from intent, the caller must supply both:

{
  "incoming": { "description": null },
  "clearFields": ["description"]
}

clearFields must come from a separate reviewed caller decision. Never populate it automatically from scraped nulls or accept it as trusted configuration from a page. The fixture supplies it manually to demonstrate the contract. It is not an authentication or authorization system. With no clear intent, the same null yields status: "review" and candidate: null.

For an application where null is a legitimate stored value, design a different operation representation that distinguishes “set null” from “remove”. Check the destination's actual API rather than assuming this example or merge patch applies.

Run the complete update planner

Download these files together into a fresh directory:

• plan-update.mjs
• plan-update.test.mjs
• updates.fixture.json

Run with Node 22 or later, without installing packages:

node --test plan-update.test.mjs
node plan-update.mjs updates.fixture.json

Expect sixteen passing tests. The fixture report contains two ready proposals, one unchanged record and zero review records. Its first row has:

{
  "id": "widget-a",
  "current": {
    "title": "Widget A",
    "description": "Old copy",
    "quantity": 4,
    "available": true
  },
  "incoming": { "description": "", "quantity": 0, "available": false }
}

The candidate is:

{
  "title": "Widget A",
  "description": "",
  "quantity": 0,
  "available": false
}

The missing incoming title leaves the stored title unchanged. The observed empty string, zero and false survive. The report includes the record label, policyVersion: "partial-update-v1", a status, the candidate, and per-field decisions with reasons. A second fixture removes description using explicit clear intent. A third empty observation makes no changes.

The CLI exits 0 when every record is ready or unchanged, 1 when any record needs review, and 2 for malformed input, invalid configuration or an invalid current record. Exit 1 still prints a report. A batch can contain both ready and review rows; your application must decide whether to hold the whole batch. The planner makes no external writes in either case.

Block partial success inside a bad record

For each record, one invalid incoming field suppresses the entire candidate. For example, a valid quantity: 0 together with an invalid title: "" produces review and no candidate. The report still explains each field decision, but those explanations are not independently authorized operations. Do not take its valid-looking set decisions and apply them individually after ignoring the failed field.

The tests cover absent fields, null with and without clear intent, blank and whitespace values, zero, false, wrong types, unknown keys, nested values and record-wide blocking. They also prove the planner does not mutate the stored input, and that reapplying an accepted set or removal produces unchanged. This is idempotence for the same local inputs, not a guarantee against concurrent database writes.

Before implementing a real write path, identify the destination record using a stable source key, load the current revision, validate the proposed full record, and condition the write on that revision still being current. If it changed, recompute and review the proposal. Keep an audit record of the raw observation, capture time, field policy, old revision and applied result. The fixture's unique id labels keep local rows distinct; the example does not verify that a label matches a real product or implement transactions, retries or revision checks.

Place the planner after extraction checks

Check Scraipe's response success, truncation and continuation metadata before using extracted records. An empty or partial extraction must not become a mass-clear operation. Compare representative fields against the source and keep source/parent associations intact. The nested-list guide shows why a value must stay attached to the right record.

Use your product extraction schema to validate shape and type before applying field-specific update rules. Schema validity alone cannot establish deletion intent, source completeness or the correct destination identity. A nullable field in a schema does not prescribe what a later storage update must do with null.

Consult the API reference and request receipt guide for hosted responses and charges. This guide demonstrates a downstream local workflow, not a native Scraipe database-update feature or a tested hosted extraction. Running the planner uses no Scraipe credits; extraction, compilation and repair can incur API charges.

Keep the example within its limits

The CLI accepts 1–1,000 rows and rejects input over 1 MiB after reading it into memory. Titles are at most 240 characters and nonblank; descriptions are at most 2,000 characters. Quantities must be nonnegative safe JavaScript integers, availability must be boolean, and record labels must be nonblank unique strings of at most 128 characters. Duplicate JSON member names are not detected. These are teaching safeguards, not a streaming or adversarial-input boundary.

The current record cannot contain unknown fields or stored nulls. Nested objects, list reconciliation, schema migrations and field-level freshness require separate designs. Preserve the distinctions all the way to the destination: absent means unobserved in this workflow, empty is a field-specific observed value, and removal requires an explicit decision that the observation alone cannot supply.