Always/Never Audit · worked example

The evaluation standards, audited

The gold-standard evaluation canon, read the way a funder's results desk or an external evaluation reads a finished report: for whether a reported result can be traced back to a preserved source record. Every quote below is checked against the source, character for character.

The finding

Three gold-standard evaluation frameworks. Not one of them requires that a reported result be regenerable from a preserved source record.

The OECD-DAC criteria, the UNEG Norms and Standards, and the DAC Quality Standards are the closest thing development evaluation has to a constitution. A funder’s results desk treats them as the floor; an evaluation office cites them the way counsel cites statute. Read them for what a competent evaluator should intend and they are humane and unusually clear. Read them the way an auditor reads a claim, following the number backward until either a source record appears or the trail goes cold, and the same pattern surfaces in all three: the evidence is described and called sufficient, and then, at the precise point where an outsider would need to re-derive it, the record that would let them is nowhere required.

This is a worked example of the Evidence Chain Read, run on the standards themselves rather than on a report that answers to them.

The two reads

One canon, two readings

Read for good faith

A profession that documents because that is the job

Credible, valid, sufficient, triangulated. Written for a careful evaluator who keeps their own working papers and could be asked, in person, to defend them.

Read for a cold check

A trail that ends before the record

Described. Sufficient. Explained. Publicly accessible. Each word stops one step short of a record an outsider could take and re-run without re-contacting a soul.

Method

Six hostile readers at one table

The standards were put to the six seats a results claim actually faces. The council supplies breadth; it does not supply verified quotes, so every quotation here was checked back against the sourced text by hand. The garbles the room produced, including a few citations to clauses that do not exist, were struck.

The Funder

the results desk

Reads the result for whether it survives the verification desk, and whether the next tranche can safely rest on it.

The Evaluator

the claim it cannot verify

Reads for the claim the report asserts and the evidence it could not actually produce if the desk asked to see it.

The Respondent

whose data it is

Reads one line: when the finding quotes them, can the answer still be traced back, and did they ever agree to that?

The Methodologist

the inference chain

Reads the inference chain for the join where a correlation quietly became a cause.

The Auditor

can the number be traced

Reads the one question that matters here: can this number be regenerated from a record someone actually kept?

Counsel

consent and exposure

Reads for the consent the data outran, and the exposure a traceable quote would create.

Audit summary

The grade grid

Four reads, because the audit keeps surfacing the same four questions and they do not collapse into one. Every cell is graded on a single test: could someone outside the evaluation team regenerate the headline result from a record the standard requires the team to keep?

The read	OECD-DAC 2019Evaluation Criteria	UNEG 2016Norms & Standards	OECD-DAC 2010Quality Standards
Source traceabilityclaim back to a preserved record	weak The six criteria grade merit, worth or significance; none asks where the evidence behind the grade is kept, or in what form.	weak Norm 6 requires data that “cannot be traced to its source”; Standard 5.3 asks only for “sufficient and appropriate” sources (¶10, ¶86).	partial §3.9 wants sources “described … in sufficient detail” and “complete lists of interviewees,” the closest in the set, inside a document that is “not mandatory.”
Independent verification	weak Silent on who checks. The criteria “should be applied thoughtfully,” by whoever is applying them.	partial Norm 4 independence is genuine, and yet Standard 5.1 lets assurance run on “internal peer review or external review” (¶84).	partial §3.2 independence is real; §1.7 accepts an “internal and/or external mechanism,” so internal alone clears it.
Attribution discipline	weak Impact reaches for “higher-level effects,” and the note declares “valid attribution is embedded in all the criteria” (Box 6).	weak The definition names “causality,” then leaves it to a methodology that “should” triangulate (¶1, ¶63).	partial §3.10 asks that “attribution and/or contribution to results are explained.” The and/or is the exit; “explained” is the ceiling.
Definitional stability	weak The 2019 revision rewrote all five inherited definitions and added a sixth; a “relevance” score shifts meaning across the change.	weak The 2025 addendum reissues “Norm 11” and “Standard 4.8,” numbers already held by different items in the 2016 text.	partial Built on the pre-2019 five criteria the revision later changed beneath it; §2.8 still promises terms “defined in unambiguous terms.”

Read the top row. No document earns a strong grade anywhere, and all three bottom out on the same read: the preserved record that would let an outsider re-derive a result is the one thing none of them requires.

strong a preserved record an outsider could regenerate it frompartial described, listed, or internally checked, short of preservedweak assumed, discretionary, or actively severed

This grid is one moment. The discipline has a history: see how it moved from 1991 to today, and how the clinical evidence standards went the other way, in the evidence chain, over thirty years.

The teardowns

Three documents, read to the seam

OECD-DAC 2019 · Revised Evaluation Criteria

The criteria that judge without a ledger

Six lenses for merit, none for where the evidence is kept

The six criteria decide merit, worth or significance. Every one describes what to weigh; not one asks where the evidence behind the weighing is kept, or in what form, or for how long. The stated purpose is to “support accountability, including the provision of information to the public,” which means telling the public what was concluded. It does not mean handing anyone the record to check the conclusion against. Impact is where the seam opens widest: it hunts for higher-level effects, the most attribution-hungry move an evaluation makes, and then declines to set any evidentiary threshold for the causal claim.

“valid attribution is embedded in all the criteria”
Revised Criteria, Impact / Box 6 · verified. Embedded by assertion, where a design would have to earn it.

“Data availability, resource constraints, timing, and methodological considerations may also influence how (and whether) a particular criterion is covered”
Principle Two · verified. Whether a criterion is examined at all turns on what was convenient to gather.

UNEG 2016 · Norms and Standards for Evaluation

The norm that cuts the wire it needs

The one framework serious about independence, and the only one that severs the trail

UNEG is the most rigorous of the three on independence and credibility, and it is also the only document that requires the evidence trail be cut. Norm 6, written to protect people who speak to evaluators in confidence, a real and necessary good, instructs that sensitive data cannot be traced to its source. The confidentiality the ethics demand is the exact inverse of an auditable chain, and nothing elsewhere restores the link for the reviewer who is entitled to see it. UNEG also gives the cleanest example of transparency standing in for traceability: Norm 7 makes the finished products “publicly accessible,” which is access to the report, never to the record behind it.

“sensitive data is protected and that it cannot be traced to its source”
Norm 6, ¶10 · verified. Written for confidentiality, and the exact opposite of an auditable chain.

“the mechanism can be operated with internal peer review or external review”
Standard 5.1, ¶84 · verified. The room that made the finding is allowed to be the room that checks it.

OECD-DAC 2010 · Quality Standards for Development Evaluation

The chain it almost draws, then marks optional

The nearest thing to custody in the canon, inside the one document that binds no one

The Quality Standards get closer to a chain of custody than anything else in the evaluation canon. Standard 3.9 asks the report to describe its sources in enough detail to judge their adequacy, to cross-validate them, and to carry complete lists of who was consulted. That is genuine, and genuinely useful. It is also description rather than preservation: a list of interviewees tells a reader the evaluation reached the right people, and hands them nothing they could re-run to get the same number. And the document disarms its own standards in the opening line, where it says, before anything else, that none of it binds.

“The Standards are not mandatory, but provide a guide to good practice”
Introduction · verified. The strongest traceability language in the triad sits in the one framework that says up front it compels nothing.

“Complete lists of interviewees and other information sources consulted are included in the report”
§3.9 · verified. A list of who was consulted is not a record an outsider could re-run.

How the grades are reached

Each cell, traced to source

Source traceability

Could an outside reviewer take what the standard requires an evaluator to keep, and regenerate the headline result from it?

OECD-DAC 2019

weak

The criteria exist to “support accountability, including the provision of information to the public” (¶12). Accountability here is disclosure of the conclusion, with no record required behind it.

UNEG 2016

weak

Norm 6 requires that sensitive data “cannot be traced to its source” (¶10); Standard 5.3 checks only that “data was collected from sufficient and appropriate sources to ensure credibility” (¶86).

OECD-DAC 2010

partial

§3.9 wants sources described “in sufficient detail so that the adequacy of the information can be assessed,” with “complete lists of interviewees.” Description of the sources, not a preserved record of them.

Definitional stability

Does a criterion or indicator mean the same thing from one reporting cycle to the next?

OECD-DAC 2019

weak

The 2019 revision refined all five inherited definitions and added coherence as a sixth criterion. A “relevance” or “impact” score is measured against a definition that moved.

UNEG 2016

weak

In the 2016 text, Norm 11 is “Enabling environment” and Standard 4.8 is “Selection and composition of evaluation teams.” The 2025 addendum reassigns both numbers to Environmental and Social Impacts. A citation to “UNEG Norm 11” now means nothing without a year.

OECD-DAC 2010

partial

Built on the pre-2019 five criteria that the revision later changed underneath it, while §2.8 promises that “all criteria applied are defined in unambiguous terms,” for whichever cycle you happen to hold.

The shared absence

The one hard floor none of the three contains

All three are built from a single move: name the evidentiary virtue, credible, valid, sufficient, described, and then never require the artifact that would let anyone outside the team check it. Each standard was written for a careful human evaluator, working in good faith, who keeps their own working papers because that is what professionals do. The standard is only ever as strong as that assumption, and the assumption is written nowhere into the standard.

Disclosure is abundant. UNEG even mandates public access to the products. A document can disclose everything and preserve nothing, and the report will still read as evidence. Proceeding on a result is never made conditional on a preserved, re-checkable chain from claim to source record, verifiable by someone who was not in the room.

A field can hold a gold standard for quality and still leave its evidence chain resting on trust. These three are the proof.

The next reader is not human

What happens when AI moves into the workflow

Every soft clause here was drafted for a human evaluator whose judgement could be questioned, whose working papers could be pulled, whose memory could be cross-examined. Move that judgement into a system that drafts, codes, translates, aggregates, or synthesises, and the softness does not disappear. It hardens into a default, applied at scale, and it produces its own clean paper trail.

“in sufficient detail” · “sufficient and appropriate sources”DAC 2010 §3.9 · UNEG 5.3

Sufficiency becomes a threshold the model applies in silence. The file records the sources as adequate because the model classed them adequate, and no reviewer sees the cutoff, or the citations that were generated rather than consulted.

“attribution and/or contribution … explained”DAC 2010 §3.10

Attribution becomes a writing task. Ask a model to explain how the intervention produced the outcome and it will, fluently, from correlation and priors, having touched no counterfactual. “Explained” was always the low bar, and it is now trivially cleared.

“it cannot be traced to its source”UNEG Norm 6 ¶10

The severing that ethics asked for, once anonymisation and summarisation run automatically, becomes the severing the auditor hits. The model is very good at a clean finding with no way back to the record.

“internal peer review or external review”UNEG Standard 5.1 ¶84

Peer review collapses into the model checking its own output for consistency, a review it will always pass. The wording is satisfied and the scrutiny is gone.

These standards were drafted for human evaluators, and their soft words are the seams where human judgement was meant to enter. Automate the seams and the discretion the standard was always carrying becomes invisible, instant, and impossible to appeal, and it produces its own paper trail. The report will still say a person weighed the evidence. Somewhere in the workflow, that will quietly stop being true.

Try it live

Convene the council on one claim

The grid above is the method run on the standards. Point it at your own work by starting with a single line. Type one results or evidence claim a MEL document might make, and a six-seat council of its hardest readers will try to break it in real time.

Type a results or evidence claim

Six hostile readers. One claim. About fifteen seconds.

Read your own document

This is what the Evidence Chain Read does to your document.

The grid above took the gold-standard documents and found where each one lets a result stand without a record anyone could re-run. Your results framework, evaluation report, evidence synthesis, or donor report carries the same seams, and a funder’s results desk or an external evaluation will read for them whether or not you have. Better that the first cold read is one you commissioned.

The Evidence Chain Read →The Always/Never Audit method →

Provenance

OECD-DAC Revised Evaluation Criteria: “Better Criteria for Better Evaluation,” approved by EvalNet 20 November 2019, adopted by the DAC 10 December 2019. Current. Sourced from oecd.org.
OECD-DAC Quality Standards for Development Evaluation: ISBN 978-92-64-08390-5, approved by EvalNet 8 January 2010, endorsed by the DAC 1 February 2010. Not mandatory by their own terms. Current. Sourced from oecd.org.
UNEG Norms and Standards for Evaluation: consolidated 2016 edition, adopted April 2016, reprinted 2017. In this text, Norm 11 is “Enabling environment” and Standard 4.8 is “Selection and composition of evaluation teams.” Sourced from unevaluation.org.
UNEG Norm 11 & Standard 4.8 on Environmental and Social Impacts: approved at the 2025 UNEG Annual General Meeting as an addendum; its numbering collides with the 2016 text above. Sourced from unevaluation.org.
Method: a six-role adversarial council on the operative text, then every carried quote verified against the source PDFs, character for character.

Documents read as published, retrieved 4 July 2026. This is an audit of the text, not a claim about any institution’s or evaluator’s conduct, and it is not evaluation or legal advice.

Back to the Always/Never Audit or all advisory offerings.