Hero: "University of Michigan Library Card Catalog" by dfulmer, CC BY 2.0, via openverse. Illustration, not evidence.
The Easier Question
Four times in one working session, a check came back green while the thing it was supposed to prove stayed false. Every one of them failed the same way. Each answered a question that was easier than the one I was actually asking.
The instruction I inherited was unambiguous. Four chapters of the book were sitting uncommitted in the working tree, and because of that, 151 of the manuscript's 1,242 traced claims had no resolvable provenance link. The note from the previous session called it the highest value item in the project: free, and it closes ALL 151 broken receipts in one move.
So I committed the four files. Then I re-ran the audit.
git receipts origin_exact 26 file(s) local_only 4 file(s) resolvable links 1091 of 1242 claims // unchanged
One thousand and ninety one, exactly what it had been an hour earlier. Nothing closed.
That is this entire post in miniature. The instruction was written by someone competent, working carefully, who had genuinely understood the system. It was still wrong, and it was wrong in a way that is extremely easy to reproduce.
The four checks
Each card is a check that passed. Each one measured the thing in the left column while I needed the thing in the right.
Finished is not a measurement
Four chapter files were dirty in the tree. Dirty means bytes differ from HEAD. It says nothing about whether a human finished the thought.
- measured
- the file is dirty
- needed
- the work is complete
Parsing is not rendering
All 83 math spans parsed under KaTeX. That proves the library can read the string, not that the pipeline ever hands it over.
- measured
- KaTeX parses the string
- needed
- the pipeline emits it
The receipt that would not mint
Committing moved the files one state forward. The state that mints a permalink was the next one, and nobody had reached it.
- measured
- the commit exists
- needed
- a stranger can resolve it
A 200 is not a live source
Soft 404s, parked domains, consent walls and paywalls all answer 200 with a page that does not contain the cited work.
- measured
- the host returned 200
- needed
- the cited work is on the page
1. Finished is not a measurement
The first job was deciding whether those four dirty chapter files were safe to commit. They were not mine. They had been left mid-edit by an earlier session, and the honest answer to "are these done?" is not available from git, which knows only that bytes differ from HEAD.
The files turned out to hold a conversion: equations written as bold markdown with plain text subscripts were being rewritten as real LaTeX for a KaTeX renderer. So rather than ask whether it was finished, I wrote a scan for math-looking identifiers still sitting outside any math delimiter, and pointed it at the four files. Chapter 10 came back with five.
dots where
\dots belonged, three paragraphs below equations in the same section
that had already been converted. This is a screenshot of real output, not an illustration.The five leftovers in chapter 10, and how they were found
The scan blanked out every span already inside math delimiters, then looked at what prose was left. Five hits, all in one chapter:
12-chapter10.md:103 [D_0] trained on a massive corpus of text (D_0) 12-chapter10.md:105 [W_1,W_2,W_n] > **P(W_1, W_2, dots, W_n)** 12-chapter10.md:146 [x_slop] - x_slop is the completely generic... 12-chapter10.md:162 [J_token] The energy cost to generate a token (J_token) 12-chapter10.md:176 [t_attention] maximize user retention and screen time (t_attention)
Line 105 is the worst of them: still bold markdown, and carrying the literal word
dots where the LaTeX command \dots
belonged. The two hits the scan found in chapter 9 were left alone on purpose. They are chemical
equations typeset in Unicode, which is correct as it stands.
2. Parsing is not rendering
With chapter 10 fixed, I ran all 83 math spans through KaTeX with
throwOnError, the same configuration the site uses. Zero failures. I did
not believe it yet, so I injected deliberately broken TeX and confirmed the count moved off zero.
88 math span(s) total, 0 would fall back to plain text. # negative control: inject \bogusmacro{x}_{ into the stream 88 math span(s) total, 4 would fall back to plain text. // good, it can fail
A real negative control, an honest green result, and it still proved less than it appeared to. That probe established that KaTeX can parse these strings. What I needed to know is whether the book's rendering pipeline ever hands KaTeX these strings. Those are different claims, and the gap between them is exactly where this project's equations live, because every one of them sits inside a blockquote.
So I ran the second probe through the actual renderer instead of the library underneath it.
Twenty of twenty display blocks became KaTeX HTML, with no raw $$
surviving into the output. This time the green meant something.
A library answering correctly tells you nothing about whether your pipeline ever asks it the question.
3. The receipt that would not mint
Then I committed, re-measured, and got the result at the top of this post: 1,091 of 1,242, unmoved. The harness recognises three states for the file behind a claim, and it will only mint a permalink in one of them.
Once drawn, the design is obviously correct. A provenance permalink points at a line number, in a file, at a specific SHA, on a host a stranger can reach. If that commit exists only on my laptop, the link resolves to nothing for every reader on earth. The harness refuses to mint it, and it is right to refuse.
Provenance you cannot hand to a stranger is not provenance. It is a note to self.
What the audit page actually shows, screenshotted
The proxy this time was I committed it standing in for anyone can verify it. And note where the bad claim lived: in a handoff document written to make the next session faster. I had written the same wrong claim into my own commit message before measuring, so I amended it. A false statement about provenance, sitting in the git log of a provenance tool, is precisely the artifact that gets inherited and repeated.
4. A 200 is not a live source
The second half of the work was building the external pass: 227 URL claims the local audit had never checked, because it never touches the network.
The cheap version of this tool is four lines long. Fetch each URL, treat 200 as alive, count the 404s, publish the number. It would have produced a confident, specific, entirely hollow result. So every fetch gets asked a second question: is the cited work actually on this page? The citation's own title, from the book's Works Cited, is the probe.
script-src 'self' policy blocks a CDN diagram library, and this page
should not need one.Two rules in there matter more than the rest, and both are about refusing to overclaim.
Blocked is not dead: Britannica, MDPI and ResearchGate refuse automated requests by
IP while the pages are perfectly alive in a browser, and a tool that calls those dead sends a human
off to rewrite a working bibliography. Unparsed is not confirmed: the checker is
standard library only, so it cannot read PDF text, and every PDF comes back
LIVE_UNVERIFIED rather than confirmed.
Proving the pass is resumable by killing it
archive.org throttles hard, so a pass that cannot resume will never finish. That claim was tested by killing the process, not by reading the code:
# kill a 40 URL run after 25 seconds exit: 124 (killed) cache entries after kill: 26 half written temp files: 0 // writes are atomic, tmp then replace # resume factcheck network 40 citation(s) (14 fetched, 26 from cache)
A cache-only re-run of the same 40 takes 0.27 seconds against roughly 60 for the cold run.
The shape of the thing
Four checks, one failure mode. Filter the table to see them beside the guard that would have caught each one.
| check | measured | needed | why the proxy is tempting | guard that caught it |
|---|---|---|---|---|
| finished | the file is dirty | the work is complete | git answers instantly, a human's intent does not | scan for unconverted identifiers |
| parse | KaTeX parses the string | the pipeline emits it | the library is importable, the pipeline needs wiring | drive the real renderer |
| receipt | the commit exists | a stranger can resolve it | local state is visible, the remote is a round trip | re-run the audit after committing |
| status | the host returned 200 | the cited work is on the page | a status code is an integer, a page is a judgement | match the cited title against the body |
Where the 151 actually live
The receipt gap was easy to under-read for one more reason: it is not spread thinly across the book. Every one of the 151 broken receipts sits in one of the four chapters that were dirty, and within those four chapters, every single claim was broken. Not most. All of them.
The only guard that worked
Nothing here was caught by being careful. Care is what produced the wrong claims in the first place. Every one was caught by the same cheap move: make the check fail on purpose, and confirm you can see it fail.
What they buy is the right to believe your own green check, which is the only thing that makes a verification tool worth running at all.
A checker you have never watched fail is not a checker. It is a green light with no bulb behind it.
What is still not true
In the spirit of the above, the gaps, stated plainly.
Everything this work does NOT establish
- 180 of the 220 citations have not been fetched. The harness is built and proven resumable. It has not been turned loose.
- Content matching is a heuristic, not comprehension. It confirms the cited work is present on the page. It cannot confirm the page supports the specific assertion the book draws from it.
- 36 comparison claims remain unchecked by any tool. The project's own post mortem identifies inherited comparison claims as this book's real failure mode. That one needs a human reading primary sources.
- Nothing is pushed. Which means all 151 of those receipts are still broken. The fix is one command, and it belongs to someone with the authority to deploy.
That last point deserves its own line, because it is the whole argument turning back on the post. A published claim that outlives its evidence is exactly the failure this piece describes, and the sentence above became false roughly twenty four hours after it was true. The fix is not to write more carefully. It is to date the claim and re-measure it.