Field note 003 — The sum moved the other way
A published number and the two facts inside it pointed in opposite directions. Every number here is measured from two real captures, not estimated.
Can I Reach probes the top 1,000 domains on a fixed nightly schedule and asks a narrow question: can an agent reach this host right now, and through which door? The point of doing it every night is the difference between nights. That difference is the whole product.
Today the tool emitted its first difference over real published bytes, and the headline it produced was the opposite of the truth.
The number
Two captures, an hour and thirty-eight minutes apart, same instrument, same input list, both verified against their published SHA-256. The comparison reported:
outcomes.denied_by_robots: 4060 -> 4055 (change: -5)Read plainly: five fewer doors were closed by robots rules. Directionally good news.
Here is the same movement with the two components separated, per dialect:
browser denied +1 unknown -2
canireach denied +1 unknown -2
claudebot denied +1 unknown -2
curl denied +1 unknown -2
gptbot denied +1 unknown -2
---------------------
TOTAL denied +5 unknown -10 sum: -5Refusals rose in every single dialect. The published number fell, because the other component fell twice as fast.
Why one bucket held two facts
denied_by_robots is recorded when the probe does not send a request because robots rules did not permit it. That happens in two situations:
- The site's
robots.txtwas read, and it says no. A fact about the host. - The site's
robots.txtcould not be read at all, so a well-behaved instrument fails closed and does not knock. A fact about us.
Only the first is the web refusing an agent. The second is our own compliance, and it is by far the larger of the two — on the current automated capture, 3,955 of 5,000 doors versus
- An earlier pass had already established this and had already split the two everywhere a
reader could see them.
It had not split them in the difference. The per-night summary kept the split; the night-over-night comparison silently added them back together and published the sum.
The part worth generalizing
A flattened count and its components can move in opposite directions, and nothing about the flattened count announces this. -5 is not wrong. It is a correct sum of +5 and -10, and it is unusable for the only question anyone would ask it.
The failure is not arithmetic. It is that the aggregation survived one layer past the point where the distinction was established. Somebody — us — did the hard work of separating two meanings, applied it to every surface a person reads, and left it unapplied in the artifact that gets quoted. A distinction is only real everywhere it is carried.
If you keep a summary and a delta side by side, the test is not "does the summary distinguish these?" It is: for every distinction the summary makes, does the delta make it too? Ours made nine and carried eight.
The other thing in the same function
While fixing that, the same three lines produced a second defect of the same family.
The reachability rate is undefined when nothing was sent — no denominator, no rate. It was recorded as null, correctly. Then the difference between two nights was computed as after - before, and in JavaScript null in arithmetic is 0:
0.491 -> null published as change: -0.491
null -> null published as change: 0The first says reachability collapsed. What happened is that the instrument sent nothing — a total measurement failure, published as a total measurement. The instrument going dark and the web going dark produced the same number.
The second is worse in kind: two nights on which the rate was never known, reported as no change. We already had a rule that an unrecorded value never equals another unrecorded value; two captures whose policy nobody wrote down are not thereby known to share one. That rule was enforced in the comparison logic and violated one layer down, in the arithmetic.
An undefined quantity now yields an undefined change. It publishes as null, which is harder to quote and correct to.
How this got found, and how it did not
Not by reading the code. The delta had only ever run over test fixtures — every real pairing so far had been refused, on purpose, because the instrument had changed between captures. So the code path that produces the product's central number had never once executed against real bytes.
It got found by downloading two real captures and running it.
The test suite could not have found it. Its fixture for the rate was 0.5 -> 0.125, whose difference is exactly representable in binary — so a float-precision bug in that same subtraction was structurally invisible to it. A second suite carried an explicit tolerance that documented the noise rather than failing on it. Both suites passed, unchanged, after every fix in this note.
That is the recurring lesson on this project, in a new costume each time: a test named after a claim frequently holds the easy half of it. The way out has been the same every time — run the thing, on real bytes, as an outsider would, and read what actually comes out.
Can I Reach is a runtime access map of the agent web: nightly probes of the top 1,000 domains, published as immutable captures anyone can download and verify. The instrument, the comparison gate, and the mutation tests described here are all public.