LLM Extraction: Go Big or Go Small?

My workhorse for extraction is GPT-4o-mini. When I need a pile of messy text turned into clean typed fields, that is what I reach for, and on most projects the bill never comes up.

One of my side projects broke that habit. It had a few hundred thousand records to get through, each one a blob of prose that had to become a small structured object. I wired up 4o-mini, got the extraction working, and then did the math on running it across the whole set. The number came back in the hundreds of dollars. On a side project, that is the line between a thing I keep running and a thing I quietly turn off.

So I did the opposite of the usual instinct. Instead of reaching for a bigger, smarter model to feel safe, I went looking for the smallest one that could still get it right.

Why Extraction Gives a Small Model a Chance

Most of the reasoning we ask big models to do is open-ended. Write the thing, plan the thing, argue both sides. The output space is enormous and the ceiling is basically the model's intelligence. That is where the flagship models earn their price.

Extraction is bounded on both ends. You know the schema before you start: these six fields, these types, these enums. The model is filling in a shape it was handed. The hard part is reading a specific value out of noisy prose and putting it in the right slot without inventing one that was never there.

I enforce that shape with a Zod schema encoded into the prompt and validated on the way out, so a malformed answer fails a check instead of slipping downstream. How to do that well is a topic on its own, and it is a companion piece to this one. What matters for model choice here is the consequence: once the task is bounded this tightly, a small model has far less room to be worse, which is what makes walking down the ladder worth trying.

The Descent

4o-mini worked. The cost was the whole problem, and the cost scales with the volume. So I started walking down the ladder.

Mistral Small was next, partly because the free tier let me trial it without committing. It handled the extraction and it was cheaper. At full volume the numbers were still higher than I wanted. Better, not done.

Then the real question: could a small model I run myself do this at all? I did not shop for a hosted option first. I pulled Qwen3 4B down and ran it locally on my own laptop, just to see whether something that small could hold the schema and read the fields. I expected it to fall apart on careful reading. It did not. Qwen3 4B proved very reasonable, getting most fields right on most records.

It was still dropping some details, and the pattern was the tell: it slipped on the records where there was a lot to read at once. The capability was there; it was running out of focus. The fix for that is to give the model less to read at once. A few chunking approaches later, breaking each record into smaller and more focused slices, the details it had been losing came back.

With the accuracy sorted, I went looking for a cheap way to run it at full volume, and hit a wrinkle. The 4B I had just proven was not on OpenRouter or HuggingFace's hosted inference the way the big names are, so there was no API key to point at it. What OpenRouter did have were the larger Qwen3 variants, a 32B and a 14B, so I tried those, expecting the easy answer to be a bigger model someone else runs for me.

That was the real surprise of the exercise. Both were more expensive than my little 4B, which I half expected from bigger hosted models. What I did not expect was that both did the extraction worse, and I had run them with reasoning turned off, so this was not a big model wasting tokens overthinking a simple task. Same bounded job, reasoning disabled, and the 4B still read the fields more reliably than its larger siblings. I do not have a tidy story for why, and I am not going to invent one. What I have is the measurement, and it was not close. So I went back to the 4B and hosted it myself on RunPod, cheap enough that a self-run small model undercuts the hosted options instead of costing more than them.

The Harness That Made It a Fair Fight

Every one of those comparisons only meant something because the inputs never moved. Stepbook records a typed pipeline and replays every step from cache, so I could hold the record set completely fixed, swap the model or the chunking, and re-run just that step on identical data. Same records, same schema, every run.

That is how I could see that Qwen3 was dropping details specifically on the dense records, and watch those details return once I chunked, instead of guessing from a handful of examples I happened to eyeball. Reproducibility is the difference between "the small model seems fine" and knowing exactly where it broke and exactly what fixed it.

If you would rather see the harness than take my word for it, I put a runnable version up as a Stepbook example. It pulls fields out of a handful of generic invoice records, and you swap the model with a one-line config change and re-run only the extract step, so every comparison lands on the same cached inputs. Point it at 4o-mini, at Mistral Small, or at a small model you host yourself, and read the cost and the fields off each run.

Here is what one record looks like through the harness. The same messy receipt, handed to a hosted 4o-mini and to a local Qwen3 4B on the identical input:

blue ridge coffee roasters
2026/02/17
thanks for stopping by. 2x house blend pour-over came to 9.00 (4.50 ea), one almond
croissant 4.25, and a bag of whole-bean espresso 16.50. paid in euros at the counter.
total EUR 29.75.

The hosted mini returned this:

{
  "vendor": "blue ridge coffee roasters",
  "date": "2026/02/17",
  "total": 29.75,
  "currency": "EUR",
  "lineItems": [
    { "description": "house blend pour-over", "qty": 2, "amount": 9 },
    { "description": "almond croissant", "qty": 1, "amount": 4.25 },
    { "description": "bag of whole-bean espresso", "qty": 1, "amount": 16.5 }
  ]
}

The local 4B returned this:

{
  "vendor": "blue ridge coffee roasters",
  "date": "2026/02/17",
  "total": 29.75,
  "currency": "EUR",
  "lineItems": [
    { "description": "2x house blend pour-over", "qty": 2, "amount": 9 },
    { "description": "one almond croissant", "qty": 1, "amount": 4.25 },
    { "description": "a bag of whole-bean espresso", "qty": 1, "amount": 16.5 }
  ]
}

Every structured field agrees: vendor, date, the total, the currency it inferred from "paid in euros," and all three line items with the right quantities and amounts. The one difference is cosmetic. The hosted model tidies the descriptions, the 4B keeps them closer to the source text. On a bounded task like this, the small model held the schema and the values. What the flagship bought me was tidier prose and a faster answer. The little model did it for free.

Where It Landed

Qwen3 4B Instruct, fed in properly sized chunks, did the job at a fraction of what the hosted models cost. The model I had the least faith in walking in, a 4-billion-parameter one on rented GPU, is the one carrying the load in production.

I'm not going to quote you a dollar figure or an accuracy number. Both depend entirely on your records, your schema, and your volume, and a number from my run would be worse than useless as a promise about yours. The point is the shape of the search, not my totals. Run it on your own data. That is what the harness is for.

The Rule Underneath It

Model choice at volume runs the opposite direction from model choice on a one-off. For a handful of documents, reach for your workhorse and never think about it again. But once the volume is high enough that the bill is a real number, the interesting question is how far down you can push. A small model on focused input can match a much larger one reading the whole document at once, and it wins on cost by a margin that actually decides whether the project ships.

Two threads run straight out of this one. The detail-loss I hit with Qwen3, and the chunking that fixed it, is its own subject: how you feed a model to trade speed, cost, and accuracy on purpose. And underneath all of it sits a question I mostly skipped here, whether a given field needs a model at all or whether a parser would do. Both are companion pieces to this one.

The habit worth breaking is treating the biggest model as the safe default and stopping there. On a few hundred thousand records, safe is expensive, and the smallest model that still gets it right is usually smaller than you would guess. You only find that floor by measuring.