The model that could not read a card

The rest of this post is the deep dive: why a hallucinated field is worse than a failed one, the controlled experiment that disproved the pixel hypothesis, what upscaling actually bought (more tokens, no accuracy), and the resolution ladder that shows where pixels genuinely do start to matter. File paths refer to the real code.

A failure with no signature

Dhaga scans business cards. You photograph a card, a vision model extracts the fields, and a review screen shows what it got before anything is saved.

Two real cards came back wrong. Not blank, not malformed, not an error — wrong. The printed domain came back as a different domain, spelled differently, still perfectly well-formed. A landline came back with its middle digits changed. One card came back under a different person's name.

Here is the shape of it on an invented card — every sample contact in this codebase is invented, and those two weren't ours to print:

// Printed on the card.
Priya Raghunath — Orbaquill Systems — priya@orbaquill.com

// Returned by the scan.
Priya Raghunath — Orbaquill Systems — priya@ooraquill.com

Look at what a checker can do with that. It's a valid email address. It passes a regex, it passes a Zod schema, it parses, it renders, it saves. No downstream check anywhere in the system can tell it from a correct read, because nothing about it is malformed — it is simply not what the card says.

That is the thing worth internalising about vision models: handed pixels it cannot resolve, a model does not return null. It returns a plausible substitute. An empty result is a failure you can catch or retry. A confident invention propagates into your database, your search index, and eventually into an email the user sends to nobody.

The review screen was showing a guess and a reading in the same typeface.

The obvious hypothesis, and why we liked it

Both failing photos were 408px on the long edge — WhatsApp-compressed forwards of handheld shots, rotated 90 degrees, of cards printed in something like 6pt type. Card A was 408×666. Card B was 410×632.

So the hypothesis wrote itself: not enough pixels. The model can't read 6pt type through JPEG compression at 408px, and no model could. The fix is therefore in the image pipeline — raise the upload cap, upscale before sending, sharpen, stop throwing away detail.

It's a good hypothesis, and the kind an experienced engineer reaches for first: mechanical, cheap to implement, and it doesn't require admitting the model choice was wrong.

It was wrong twice over.

The controlled experiment

The experiment is embarrassingly simple, which is the point: hold the pixels fixed and change one variable. We re-ran both cards through the extraction-tier model and the reasoning-tier model on the identical images — same file, same 408px, no preprocessing difference at all.

The extraction-tier model got every field of both cards wrong: a name, a company, a domain and a mobile number each replaced by a confident, well-formed invention. The reasoning-tier model read both cards perfectly — every field, street address included — from those same pixels.

No resolution argument survives that. The information was in the file. One model could extract it and one could not.

Upscaling: the non-fix that costs money

We didn't stop there, because "the small model just needs help" is a seductive next hypothesis. So the small model got every advantage: the hardened prompt, and the images upscaled ×2 and ×3.8 (lanczos plus sharpen, to 1568px).

It read neither card. At ×2 it was, if anything, worse. At ×3.8 it finally got one name right while still inventing that card's domain, email and mobile number.

Then the arithmetic, which settles it for good:

• The small model on a ×3.8 upscale sends ~2,970 input tokens.
• The strong model reading the untouched 408px original sends ~390.

Per-token, the strong model is a flat 3.0× the cheap one — a fixed ratio between two rate cards, not something a measurement can move. The token volumes differ by 7.6×, which more than wipes that out: on input, the cheap model reading the upscale costs about 2.5× what the strong model costs reading the untouched original.

Be precise about which side of the ledger that is. Input is the only side upscaling moves, and there the cheap path loses outright. Output tokens carry the same flat 3.0× ratio and the two paths emit comparable amounts of it, so every output token added closes the gap — enough, at the output volumes a real scan produces, that a whole scan can land the other way round. Which is the honest summary: on cost the two configurations are in the same neighbourhood, and the cheap one is wrong.

That's the general trap. Upscaling cannot add information the JPEG never had. It renders interpolated pixels, you pay for those pixels by the token, and the model gets a larger picture of exactly the same uncertainty. Every preprocessing knob shares this ceiling: it can only help the model use information already in the file.

Where pixels do matter

Having established the cap wasn't the bug, the tempting next move was to revert it. The raise from 1024 to 1568px at quality 0.9 had shipped on a token-budget argument, and halving the image would halve the bill on the now-pricier tier.

So we measured that too, stepping one of the same cards down through a ladder on the model that can read it. The finding wasn't the one we expected: the fields do not fail together.

The ladder ran 500 / 400 / 320 / 256 / 200px of card height. Two of its rungs were written down field by field:

The other three rungs were stepped through but not recorded per field, so I am not going to tabulate them. What the record does say is the shape, and the shape is the finding: the phone digits go first, and they go quietly. Two digits in the middle of a mobile number swap for two others; step down again and a different pair moves. Every intermediate result is a valid ten-digit mobile number.

The reason is redundancy. A name and an email address are linguistically redundant — a model that resolves two thirds of the glyphs can reconstruct the rest, and be right. A digit string has no redundancy at all. Nothing about one digit constrains the digit beside it. Which is precisely why a phone number is the field a model must never be asked to guess, and the one that fails first and most silently when you ask it to.

That puts the onset of degradation between 400 and 500px of card — of card, not of frame. A card filling half the frame gets ~512px at a 1024 cap, sitting exactly on that boundary, and ~784px at 1568, clear of it by ~1.6×. The headroom is the point, so 1568 stays. It is also the long edge Anthropic's vision models resize to internally — simultaneously the most detail the model can use and the point past which extra bytes are uploaded only to be discarded.

The comment on the constant in apps/web/src/utils/constants/app/contacts.ts says the thing we most needed a future reader to know:

Do not read this number as the reason scans got better.

The tension with cost, stated honestly

We've argued elsewhere that inference is the P&L — the model bill dwarfs the infrastructure bill, and the discipline that matters is never call a bigger model where a smaller one passes evals. That rule stands. This post is its fine print.

The move cost real money. A scan is still one credit to the user, but the dollars behind that credit went up 2.2× on a small forwarded photo and ~5.2× on a full-size camera capture — the second multiplier being the image cap, not the model. It cost latency too: ~5.2s and ~5.9s of model time on the small photos against ~2.3s for the old cheap scan, with a 9.7s worst case. Both reads run in parallel, so the scan pays the slower of them rather than their sum; that's the only reason it isn't roughly double.

We paid all of it, because the alternative was a feature that silently writes wrong phone numbers into your address book.

The rule isn't wrong. The word doing all the work in it is passes. The small model passed on flat, square-on photos of clean cards at generous resolution — which is what you scan when you are testing a scanner, and not what arrives from a real user, who shoots a card handheld at a conference in bad light and forwards it through a messenger that recompresses it to 408px. Our eval was measuring the wrong distribution.

Being precise about what moved

Both of the scan's model calls run on the reasoning tier now:

// Before. The scan's fields came off the cheap extraction tier.
tier: "extract",

// After. Both calls in lib/ai/card-scan/reads.ts.
tier: "reason",

One path deliberately did not move. apps/web/src/lib/ai/card-transcription.ts is the post-save recovery transcription — the fallback that fills in a receipt note's text a few seconds after saving, when the scan's own second read failed. It still runs tier: "extract". That's a considered choice, not an oversight: nothing on that path decides a contact field. It makes an already-saved note more searchable, and a garbled fallback note is a worse note, not a wrong contact. If it ever starts feeding fields, it moves.

The second half of the fix is a net under the first. The card is now read twice in parallel — once for fields, once for a line-by-line transcription — and unverifiedCardAtoms in packages/core/src/parse/card-verify.ts reports every email, phone, link, name or company the two reads disagree on, for the user to check against the photo already on screen. Nothing is dropped: two disagreeing reads say one of them invented the value, never which one.

Pairing the strong model's fields with a cheap transcript was the obvious economy, and it was measured and rejected. The cheap read contradicted every one of the correct fields, so the panel fired on all of them. A checker that flags correct output is worse than no checker — it trains the user to dismiss the panel before the one scan where it's right.

The takeaways

1. A vision model handed unreadable pixels returns a plausible substitute, not nothing. Well-formed and wrong is the default failure mode, and no schema, regex or format check downstream can detect it. Budget for it in the UI, not in validation.
2. "The cheapest model that passes your eval" is only as good as the word passes. The eval has to be the real input at the real resolution from the real capture path — the compressed, rotated, handheld photo, not the flat shot of a clean card.
3. Change one variable. Same file, different model settled in one run what a week of image-pipeline tuning would not have. If the obvious hypothesis is about your preprocessing, disprove it before you build on it.
4. Upscaling cannot add information the file never had. It only spends tokens rendering interpolated pixels — enough of them that on input the cheap model on a ×3.8 upscale cost about 2.5× the strong model on the original, and was still wrong. There was no cheap-and-correct configuration to find.
5. Fields degrade at different rates, and the least redundant one goes first. Names and emails can be reconstructed from partial glyphs; a digit string cannot. Set quality floors against the phone number.
6. A cross-check that fires on correct output is worse than none. The checker has to be at least as capable as the thing it checks, or you are teaching users to click past a warning.