The Reference Check

What the sweep found

The public investigation, by the advisory firm Article One and reported by News24 and MyBroadband, identified six fabricated citations. The sweep flagged all six. It also flagged a seventh that hand-verification confirmed as a fabrication nobody had reported, and nine more that turned out to be something else entirely.

The seventh fabrication

Reference 33 of the gazetted bibliography reads:

Karr, V., & Smith, L. (2023). “Digital Rights and AI Governance in Africa: A Focus on Ethical and Legal Challenges.” Journal of African Law, 67(1), 95-108.

The Journal of African Law is real. Volume 67, issue 1 of February 2023 is real. We checked its table of contents on the publisher's own pages: no such article appears in the issue, no such author pair appears anywhere in it, and the cited page range falls across two entirely different articles, a Kenyan presidential-veto study ending at page 96 and a Tanzanian sentencing study beginning at 97. The citation is built the way the other six are built: a plausible title, plausible authors, a real journal, and pagination that does not survive contact with the actual issue.

This was not found by cleverness. It was found by the same routine sweep that anyone could have run on this document before it was gazetted, and that nobody did.

Why the verdict has four values

An earlier version of this sweep reported one outcome called indexed, meaning a record had been found. Laurent Poupet read the results and named the fault precisely: the checker had been asked for resemblance and allowed to answer with a boolean. A record that exists but disagrees with the claim is a louder signal than no record at all, and it was being filed under the same word as a clean pass.

Splitting confirmed from contradicted fixed that and immediately exposed a second conflation. Seven references turned out to be real sources, live at the addresses they cite, that the indexes had matched to an entirely different work. Calling those contradicted blames the author for somebody else's filing error. They are their own verdict, mismatched, and the reason to separate them is that the repair differs: a contradiction goes back to whoever wrote the claim, a mismatch goes back to the resolver.

The fourth value falls out of the same logic. When a record is retrieved using the identifier the citation itself supplied, identity is settled and only the description can differ, so a venue named two ways is metadata variance rather than a challenge to anything. Which makes the identifier a route rather than a check: a resolving DOI sends you to field comparison, and its absence sends you to a search that has to come back with a positive statement, the contents of the issue were read and the article is not among them, instead of a silence.

Absence keeps its own verdict, not found, because a hunt and a comparison are different work and they route to different people. A single bucket marked suspicious can be sent to nobody.

Both ways a boolean lies

Three of the six known fabrications had come back marked as found. Fuzzy title-matching had paired each invented paper with a real one sharing a few words of its title: an AI regulation paper matched to a study of postgraduate supervision, a paper on AI and social justice matched to a 2014 handbook chapter on transitional justice written before its subject existed. A checker that stops at “a similar title exists” would have cleared half of this document's fakes.

The seven mismatches are the same weakness pointing the other way. The OECD AI Principles, the Presidential Commission's Diagnostic Report, the Harvard Business Review article, a Wits working paper: all real, all live at the addresses cited, and every one of them vouched for by a record that was not about them. The same weak comparison was clearing fabrications and certifying honest work on false evidence, and one word reported both as a pass.

Sixteen references came out needing a person to look, which is nine more than the six that made the news.

The overrule record

Every adjudication is written down, with who made it and why, because an overruled flag is only evidence of a real human step if the overrule is itself recorded. Without that file the review is a green tick that takes longer. The record is what makes a false-positive rate computable per verdict, and one sweep produces this:

One document is not a sample, and these rates describe this sweep rather than the method. They are kept so that after thirty of them they will describe something. The bucket to watch is not found, because that is where absence gets mistaken for invention.

Everything the machine flagged

Sixteen references, the machine verdict, and what a person made of each one. Nothing here is a verdict the sweep reached on its own.

The thirty-four unflagged, unindexed entries are public-comment submissions to the department itself, a class of document no scholarly index holds and no index check can condemn. Reporting those as failures would be its own kind of fabrication. It is worth saying, though, that these entries do not have to be unverifiable. Public-participation practice has an old and unglamorous answer: a register of submissions, published as an annex, so that anyone can confirm who contributed what. Departments running comment processes keep such registers as a matter of course. Publishing this one alongside the policy would have cost a page and made half the bibliography checkable.

Reference 44, or why a flag is never a verdict

The NACI recommendations report returned nothing from any index, the same result as four of the seven fabrications. It is real. Government advisory reports, evaluations, terms of reference, most scholarship published outside the North: entire classes of real work are invisible to every index there is. A checker that shouts “fake” at every miss would defame more honest documents than it catches dishonest ones. The judgment call between fabricated and real but unfindable cannot be automated, and it is the difference between a useful verification record and an incident.

Reference 38 makes the same point from the other end. The sweep called it contradicted because the citation names a Proceedings volume and the index names the IS4SI 2019 Summit. Those are the same event described two ways, so the flag is noise, and only a person reading both records can say so.

We hold ourselves to the same standard we sell. The method behind this sweep runs continuously against our own published sources, self-caught errors included, at Sixteen Days.

What the Reference Check is

A pre-filing, pre-publication sweep of a document's complete reference list. Every citation checked against the scholarly indexes, every DOI resolved, every URL tested, and every index match compared field by field against what the document actually claims. Flags are verified by a human being before they are called anything. The deliverable is a signed verification record: what was checked, against what, what was confirmed, what could not be, and what that means.

For a legal practice, that record is the demonstrable discharge of a duty South African courts have now twice said cannot be delegated to the tool that created the risk. For a publisher or an institution, it is the difference between this page existing about somebody else's document and existing about yours.