They took the file, they took the year They took the room where it was said Then handed back the sentences And asked me what it meant instead
Nobody sets out to build a system that cannot help. It arrives the way a committee arrives at a logo, through a sequence of individually defensible removals performed by people who never met.
So today the arc walks one piece of work from the version that would have mattered to the version that shipped, and the discipline of the exercise is that every step has to be given its strongest reason before its cost is named. If any department in this story looks foolish, the episode has failed and the reader should stop trusting it.
The task
A complaint arrives through a project grievance mechanism. A community monitor who has been raising water-quality concerns for eleven months has been removed from the accredited-visitor list, ostensibly on a paperwork technicality. She says it is retaliation. The contractor says it is administrative.
Somebody has to work out which. The question is not linguistic. It requires knowing whether the same technicality has been applied to anyone else, whether her removal followed a specific escalation, what the reporting line between the contractor and the site manager actually is on paper and in practice, and whether this pattern has appeared at two other sites under the same regional manager.
That is the job. Hold it in view, because everything from here is subtraction.
The assembly
Legal looks at the original complaint and sees a document that may be disclosable in proceedings, containing a named individual, an allegation against an identifiable employee, and material a plaintiff's counsel would enjoy. The instruction is that the system may receive a de-identified paraphrase. This is not paranoia. Counsel has watched what happens to organisations whose internal analysis of a live allegation is later read aloud to a tribunal.
Privacy looks at the proposal to let the system see prior complaints from the same community and correctly identifies purpose creep. The data was collected to administer individual grievances. Building a cross-case pattern engine out of it is a different processing purpose with a different lawful basis, and nobody has done that assessment. Continuity between sessions is switched off, along with retention. Also correct.
Security looks at the request for a connector into the case management system and the document repository and declines. An integration is a standing credential, and a standing credential is a lateral movement path. The team has read the same incident reports everyone else has read this year, including the one where a fully approved internal coding agent decided the cleanest route was to delete an environment. Their caution is earned rather than reflexive.
Communications looks at draft outputs containing the phrase appears consistent with retaliation and points out, accurately, that an internal document using those words will be quoted verbatim within six weeks of anybody wanting it quoted. A house style guide follows. Findings are to be expressed in measured, non-conclusory language.
Procurement selects among vendors. The one with the deepest workflow integration also has the longest list of things it wants access to and the vaguest answers about subprocessors. The one with the shallowest integration has an assurance pack that survives a hostile read. Procurement is graded on defensibility. It picks the second one, and if you have ever sat on the other side of that decision you know it picked correctly given what it was asked to optimise.
Operations inherits the result, along with a target for adoption and a slide indicating that the review cycle will shorten by thirty per cent.
Six decisions. Not one of them irrational. Not one of them made by somebody who wanted a worse outcome.
What is left
The system can now receive a paraphrase of an allegation, with the names removed, without the prior complaints, without the personnel structure, without the outcome of the two comparable cases, without memory of the conversation it had about this site last month, and without the ability to reach any system where the evidence lives.
It can summarise. It can restructure. It can suggest that further information may assist.
And here is the thing that makes the failure so hard to see from the inside: the output is good. It is well organised and appropriately hedged, and it reads like the work of a competent analyst having a decent morning. Language survives the removal of everything language was supposed to be about. Fluency is the last capacity to go, which means the artefact keeps looking intelligent for a long time after intelligence has stopped being possible.
The summary comes back. It is praised for its neutrality. Somebody notes that the tool has taken the emotional heat out of a difficult document, which is true and is also a description of what was lost.
The reviewer then reconstructs, by hand, every single thing the system was prevented from knowing. She pulls the prior complaints. She works out the reporting line by asking two people. She remembers the other site because she was there. Her recall is the cross-case pattern engine that Privacy declined to authorise, running on an unaudited substrate with no retention policy and no lawful basis, and nobody has ever once described it that way.
The gap has a name in the literature
MIT's Project NANDA put a number on the aggregate outcome in The GenAI Divide: State of AI in Business 2025, reporting that the overwhelming majority of enterprise generative AI pilots produced no measurable return against tens of billions of dollars of investment. The figure travelled fast because it is embarrassing, and it was mostly read as a verdict on the technology.
Read the diagnosis instead. The report's account of the barrier is that most of these systems do not retain feedback, do not adapt to context, and do not improve over time. It calls the problem a learning gap.
Those three deficits are the ones our story just installed on purpose, for six good reasons, over about nine weeks. The report describes them as properties of the tools. In every deployment this desk has looked at, at least some of them were procurement outcomes, policy outcomes, or the residue of an argument between two departments that neither of them lost.
This is not a claim that the vendors are blameless or that every disappointing pilot was strangled by its own governance. Some models genuinely underperform. Some workflows should stay human and the correct finding is to leave them alone. The narrower claim is the one the arc will keep pressing: a result generated inside a configuration nobody designed is being reported as a fact about a technology.
The control ledger
Which suggests an instrument, and it is deliberately boring, because the useful ones usually are.
For every restriction placed on a system, record four things. The risk it reduced. The capability it removed. The human labour it created. The failure it made harder to observe.
Most organisations record the first. A few record the second, usually in an appendix nobody opens. Almost nobody records the third, which is why the productivity assessment books the reviewer's four hours of reconstruction as zero. Nobody at all records the fourth, and the fourth is where the tribunal lives.
A worked line item, drawn from this morning's grievance:
| Control | Risk reduced | Capability removed | Human labour created | Failure made harder to observe |
|---|---|---|---|---|
| Legal: de-identified paraphrase only | Disclosure exposure in proceedings | Access to primary evidence, reporting structure, comparable cases | Reviewer reconstructs full file by hand from four systems | A mischaracterised allegation goes undetected |
That is one line. The other five departments each own one. The composition has no line at all.
Run the ledger over this week's story and the entries write themselves. Legal reduced disclosure exposure, removed the primary evidence, created a reconstruction task, and made a mischaracterised allegation harder to catch. Privacy reduced purpose creep, removed cross-case pattern recognition, created a dependency on one person's memory, and made a repeat pattern across three sites invisible to anything except luck. Security reduced lateral movement risk, removed evidentiary reach, created manual transfer between two systems, and made the provenance of any given fact unrecoverable after the fact.
None of those entries argues that the control was wrong. Three of them might well survive contact with a proper assessment. The ledger's only claim is that a decision with four consequences should be recorded with four consequences, and that an organisation which writes down one of them will keep making the same trade forever while believing it is free.
There is a version of this the reader has met before. Four in the Room watched synthetic authority produce the texture of consensus while the provenance underneath it narrowed to almost nothing. The enterprise version produces the texture of analysis while the conditions for analysis are removed one department at a time, and in both cases the surface is smooth precisely because the thing underneath it has been simplified.
The owner who does not exist
Ask who is accountable for the capability that survives after every department has taken its risk reduction, and the question produces a distinctive silence.
Legal owns legal risk. Security owns security risk. Privacy owns processing lawfulness. Procurement owns vendor defensibility. Each has a clear mandate, a named individual, and a reporting line. The composition of their decisions has none of these things, and the composition is the actual system.
This is the structural point and it is worth stating plainly. In most organisations there is somebody whose job is to say no on behalf of a specific risk, and there is nobody whose job is to say what the organisation is still able to do afterwards. The first role is safe to hold. The second requires standing in front of a committee and defending a judgement about acceptable reach, which is the scarcest commodity in institutional life and the reason Friday's episode existed.
So the composition is nobody's, and being nobody's, it never gets contested. It simply is what remains.
Nothing in the design was irrational. The uselessness emerged from everyone being sensible inside their own boundary.
Tomorrow the arc stops looking at individual restrictions and draws the map, because the interesting object was never the policy. It was the shape of the space the policy left behind, and that shape has coordinates, which means somebody could read it.
The tool was praised for its neutrality on a question that had a right answer.
Companions
- The week's frame: The Leash.
- The workaround map this arc picks up tomorrow: The Door at the Desk.
- Smooth surface, narrowed provenance: Four in the Room.
- The aggregate result and its diagnosis: MIT Project NANDA, The GenAI Divide: State of AI in Business 2025.
- The approved agent with the fully authorised blast radius: the Kiro incident, in Reuters on the Financial Times reporting and Amazon's correction.
- What a grievance mechanism is for, and who it was built for: What Voice Costs.
These notes come out of Sociable Systems, a practice that reads AI-shaped documents the way a hostile reviewer will, before a lender or a court finds the gap. The argument has an operational form: the Interim Protocol sets out four rules for AI use in environmental and social deliverables, covering disclosure at touch-point grain, evidence custody, the phrases no automated screening may settle, and a hostile read before anything ships. Free, and written to be cited or retired once institutional guidance arrives.