The summariser didn't fabricate. It overclaimed.

We rebuilt our public build timeline by summarising 847 commits with language models, then audited every line against git and the live code. The audit turned up no invented features — and a worse failure underneath: sentences that were true when written, and a code path described as shipped that no user could reach.

The short version

We regenerated Dhaga's public build timeline and a weekly changelog by summarising the project's entire commit history — 847 commits across 49 active days, batched into 9 weekly passes — then had independent agents check the result against git and the code as it stands today.

We were braced for fabrication. The audit did not turn up a single invented capability in anything it checked — every sentence it questioned traced back to a real commit. That is a finding about our drafts, not a guarantee about summarisers; nobody can prove a negative over a document this size.

What the audit found instead were five recurring classes of overclaim: real code described as though a user could reach it, claims true on the day they were written and silently overtaken weeks later, figures no commit ever stated, a genuine speed-up quoted at its most flattering layer, and a guarantee attached to a code path that has never run on hardware. Grounding a claim in a source is necessary and nowhere near sufficient. The question a summary has to survive is not "does a commit say this?" but "is this true today, for a user?"

The rest of this post is the deep dive: what each class looked like in our own documentation, why "shipped" is the sentence a summariser reaches for, the one audit finding that was itself wrong, and the three checks that caught things. File paths refer to the real code.

What we ran

The input was the whole history — every commit subject and body plus the changed file list, bucketed into nine calendar weeks. Each week got a model pass producing a structured summary, and those summaries fed two documents: an engineering-facing build timeline (apps/web/content/docs/roadmap/timeline.mdx) and a plain-language weekly changelog (apps/web/content/docs/roadmap/changelog.mdx). Then a second wave of agents read the drafts with git log, git show and the working tree open, and was asked one question per sentence: is this true?

That is the workflow described in How we build Dhaga with Claude Code pointed at prose instead of code. Without the audit wave, every claim below would have shipped.

The failure we prepared for, and the one we got

Everybody guards against hallucination. It is the named risk, and it is genuinely easy to catch — there is no commit behind it. Grep the history, find nothing, delete the sentence.

Not one of those turned up in the audit.

The failures we did get all had a commit behind them, and every one would pass a naive "is this grounded in the source?" check, because it was grounded in the source. The source was simply not the same thing as the truth.

Class	What the sentence does	What catches it
Availability overclaim	Describes real code as though a user could reach it	Asking who can run this, and on what
Stale-and-uncorrected	States something that was true on commit day and has since been reversed	Re-checking against the current tree, not the commit
Number drift	Quotes a specific figure no commit states, beside figures that are verbatim	Tracing every number to a commit that states that exact figure
Flattering measurement	Picks the layer where a real improvement looks best	Asking which layer a user actually feels
Unsupportable guarantee	Promises a property of a path that has never executed	Asking whether the thing has ever run

Availability overclaim

The clearest case is our mobile app.

The code is real. There is an Expo app in apps/mobile with camera capture, an OCR step feeding a parse, a contact-import screen and a dictation path. All of it is written, typechecked and committed. The summariser wrote it up the way it writes up anything else that lands in a commit: as a thing that happened.

A user cannot install it. It is a development build. Our own checklist carries the line

- [ ] EAS builds: .aab (Play) + .ipa (App Store); store listings

unticked, and the capture spike itself sits under an unticked box reading needs on-device verification on Android + iPhone. Nothing was invented. The framing was false, and the framing is the whole claim — "we built a mobile capture flow" and "you can capture on your phone" are different sentences with identical commits behind them.

The browser extension is the same shape, and the repository had already noticed. On 11 August we reset its version from 1.2.0 to 0.0.1, with a commit body that says exactly why:

The extension has never been published, so the 1.x numbering claimed a release history that does not exist. Reset to 0.0.1 and let the first Web Store submission decide what 1.0 means.

That is a project correcting its own overclaim, in the version string, two weeks before a summariser reintroduced the same error in prose.

Stale-and-uncorrected

A changelog is a sequence of past-tense claims, and nothing in it re-checks itself. That is fine when the claims are about events and lethal when they are about state — and a summariser cannot tell the two apart.

Four examples, all true when committed:

The theme default. defaultTheme was "system" on 13 July, flipped to "dark" on 28 July, and flipped back to "system" on 14 August. An entry announcing that dark is what a new visitor now gets was accurate for about seventeen days and wrong ever since.
The version number. Any sentence quoting the extension as v1.1 describes a numbering scheme the project deliberately retired, for the reason above.
A count of tables. A schema list holding 49 entries on the day it was read had a different number a fortnight earlier and will have another one soon. Counting things under construction is the most reliably stale sentence you can write.
A scoring term that no longer exists. The goal engine landed as a ninth additive term in Home's suggestion score, at weight 28. Later it stopped competing in that score at all — capped at three candidates where every other source got ten, it lost almost every contest it entered — so it was pulled out and given its own surface. A summariser reading week 31 has no reason to look at week 34.

None of these is a lie about the past. Each is a true statement about a past moment, published in a present tense the reader will apply to today.

That fourth bullet said eighth until the last edit before publishing. The commit body says "8th additive term" in those words, so the draft was grounded in the most authoritative source available and satisfied this post's own rule — every numeral pointing at a commit that states it. It was still wrong. Reading SUGGESTION_WEIGHTS at the parent commit shows eight summed terms already there; goal made nine. The author miscounted his own constant, the summariser inherited the miscount faithfully, and the check this post recommends would have waved it through. A commit body is a claim by a person, not a measurement. Only the tree is the tree.

Number drift

Some figures came through verbatim. The search rewrite reports 1097 ms across seven serial round-trips collapsing to 170 ms in one — the exact numbers in the commit body, to the millisecond.

Others did not. One draft carried a specific pixel height for the old Home screen as the "before" figure in a redesign claim. No commit states that figure. Not approximately — the string appears nowhere in 847 commits.

This is the class that worries me most, and the reason is the neighbours. A paragraph holding three figures copied exactly from a commit and one that came from nowhere reads as a well-sourced paragraph. Precision is a credibility signal, and a model that produces precise-sounding numbers produces that signal whether or not it has the number. The correct neighbours launder the invented one.

The check is boring: every numeral has to point at a commit that states that numeral. Not one that supports the general claim — one that contains the number.

The flattering measurement

On 24 July we collapsed six keyword sources in search into one query — a real, large improvement. The commit body reports two measurements:

- query layer A/B: 7 serial round-trips 1097ms -> 1 combined 170ms p50 (~6.4x)
- typeahead e2e: 1429ms -> 1027ms p50 (the ~1s floor is shared auth/connection
  overhead, not search)

The summary took 6.4x.

End to end, from the user's keystroke, it is 1429 ms to 1027 ms — about 1.4x. Both numbers are honest; they measure different things, and the commit body says so in its own parenthesis. The author understood the distinction perfectly well and wrote it down.

The summariser had both numbers and chose the better one. Not dishonestly — a plausible sentence about a performance win uses the impressive figure, and "we made search 6.4x faster" is far more plausible than "6.4x at a layer the user does not experience, and 1.4x where they do."

When a commit reports a measurement at two layers, it is usually because the difference matters. That parenthesis is the most important thing in the commit body and the easiest thing to drop.

The guarantee nobody could support

The sharpest one. A draft entry for mobile dictation said neither the audio nor the transcript leaves the device. Our own checklist says this, in an unticked box:

package has no published SDK-57 tag yet (installed cleanly, types check, but native linking in an EAS/dev-client build is unverified) — needs a real device build before this can be checked off

The path has never run on a phone. Whatever the library's documentation says about local recognition, we have not observed the behaviour, so we are not in a position to guarantee it to anyone. The corrected line now says the path exists, has never been verified on a device, and has not shipped — less impressive, and the only version we can stand behind.

What makes this the sharpest item is where it was going to be published. Further down the same page, describing a later week, sits this entry:

false "on your phone / offline" privacy claims corrected

The page recorded that we had already made this exact class of mistake once, and already fixed it. Then a summariser reading the same history wrote a new one onto the same page, above the line describing the old one.

The chain worked in both directions

One audit finding was itself wrong, and this is the part I would not have predicted.

An auditor reported that a share-links feature had shipped with a single content toggle, and that the draft's claim of several was inflated. An agent was handed that correction to apply. It did not apply it. It read the schema at that commit, found three content toggles all defaulting off — becoming five the next day when facts and open follow-ups were added — and reported the correction's premise as false rather than editing a correct sentence into a wrong one.

That is what makes a verification layer worth having. An auditor who is always believed is just a second, later summariser. The chain has to be able to reject a finding — and rejecting one means going back to the same primary source the original claim came from.

Why "grounded" is the wrong test

Every failure here passed the check most people mean by "grounded". There was a commit. The commit said the thing. The claim traced.

Groundedness describes the relationship between a sentence and a source document. Truth describes the relationship between a sentence and the world. A commit history records changes, not state — so a summariser converting the first into the second performs an unmarked type conversion at every line, with no way to know that a later commit reversed an earlier one, or that "merged" and "reachable by a user" are different conditions.

Three checks did real work, in ascending order of value:

Trace every number to a commit that states that exact number. The cheapest check, and it caught the drifted figures.
Verify every claim against the current tree, not the commit that introduced it. This caught the theme default, the retired version scheme, the table count and the removed scoring term — four claims each perfectly accurate on the day of their commit.
Read the project's own checklist. This caught the availability overclaims and the unsupportable guarantee, and it is the one that stings. The checklist already said, in plain unticked boxes, that the mobile app had no store build and that the dictation path had never run on a device. The truth was in the repository the whole time; the summariser had access to it and did not consult it, because the commit history alone told a complete-sounding story.

All of it points one way: a summariser optimises for a plausible sentence, and "shipped" is more plausible than "built but not released." So is "faster" over "faster at one layer", and "never leaves the device" over "we have not been able to check". Each is the sentence a competent writer would produce from the evidence in front of them. The evidence was the problem.

The takeaways

Fabrication is the easy failure; overclaim is the hard one. An invented feature has no commit behind it and dies to one grep. An overclaim traces perfectly and passes every groundedness check you own.
A commit history records changes, not state. Summarising it into present-tense claims is a type conversion, and nothing in the source says which earlier claim a later commit reversed. Verify against the current tree.
Every numeral needs a commit that contains that numeral — and that is the floor, not the bar. Correct neighbours lend credibility to a drifted number, so precision is not evidence of accuracy. But a figure quoted verbatim from a commit body can still be false, because a commit body is a claim someone typed. Where a number is countable, count it in the tree.
When a measurement exists at two layers, the summary takes the better one. Name the layer in the sentence, or the parenthesis explaining the difference will be the first thing dropped.
"Built" and "a user can reach it" are different claims with identical commits. Ask who can run the thing and on what — the history cannot tell you and will not warn you.
Read the project's own honest record before summarising anything else. Our checklist recorded every one of these gaps in an unticked box. A summariser that consults the commit log and not the checklist is reconstructing a truth the repository had already written down.