Speed, Cost, Accuracy: How You Feed an LLM

You get an extraction pipeline working on one document. The demo looks great. Then you run it across the whole set, and it stops holding up. That is the moment how you feed the model starts to matter more than which model you picked.

There are three ways to feed it, and which one fits depends on the record in front of you.

Strategy One: One Pass, One Big Model

The simplest move is to hand the whole record to a capable model, maybe a reasoning one, and ask for every field at once. One call, everything back.

It is the least code and, per record, usually the cheapest: one model call, one set of tokens. What it does not buy you is a guarantee. A model in a single pass still misses fields on a messy or long record, and the more you ask for in one call, the less stable the answer is run to run, so the same record can come back a little different on the next pass. A stronger model holds that together better than a small one, so part of what a flagship buys you is consistency as much as accuracy. Drop to a smaller model in one pass and you lose ground on both at once.

This is the right move more often than people admit, and not only at low volume. A capable model reading the whole record in one pass is frequently the cheapest and the most accurate option you have, and reaching for something fancier can quietly make it worse: fragment the read and you lose the context some fields depend on. So the next two strategies are for when one pass hits a wall you can name, rather than a default you reach for. The record is too long to fit one prompt, or to hold in the model's attention past the middle: chunk it. You are pulling many fields or categories the model starts to confuse when it juggles them all at once: split the questions. There is a third wall this post does not solve, that you need every value traceable to the line it came from, and that one is a whole pipeline, not a feeding trick. Short of hitting one of these, measure before you reach, because on a record that fits, one pass usually wins.

Strategy Two: Chunk It, and Let the Prompt Navigate

Break the record into smaller slices and process each one, with a bigger prompt doing the navigation: what to pull, from where, and when to expect it. The prompt is carrying the load, telling the model how to find each field, and that holds regardless of model size. It is what lets a cheaper model keep up.

There is a real reason smaller slices read better, and it is more than token cost. Stanford's Lost in the Middle is the result to sit with: models use information at the very start and end of their context and get measurably worse at finding what is buried in the middle. Feed a long record whole and the field you need most lands in the spot the model reads least carefully. A focused slice keeps it out of that dead zone. Every chunking guide circles the same tension: small enough to stay in the model's sharp attention, big enough that no single answer straddles a seam.

Strategy Three: Split the Questions

The third lever is cognitive load. Instead of asking for twelve fields in one call, ask for three, four times over.

Each call carries less at once, so the accuracy per field goes up, especially on the fields that used to come last in a long instruction and get the thinnest slice of attention. You buy that accuracy with more calls and more wall-clock. The right amount to ask per call is the most the model can carry on your records before the answers start to drop, and the only way to find that line is to watch where it actually falls.

The Catch: Parallel, or It Chokes

Strategies two and three share a consequence you cannot ignore. You just went from one call per record to many, and if you run those one after another the pipeline crawls. A few hundred thousand records become a job that never finishes.

Parallelism is the price of chunking or splitting, not an optimization you add later. You fan the calls out concurrently, cap the concurrency to what your rate limits and your budget tolerate, and let the batch run wide. The throughput math is its own tradeoff, more concurrency lifts total throughput while any single call waits longer in the queue, but for a batch nobody is watching that is free time back. Skip it and the accuracy you bought with strategy two or three costs you a run time you cannot ship.

Measure It, Do Not Guess

Every lever here trades one thing for another, and the exchange rate is different for your records than for anyone else's. Change the chunk size, eyeball a few outputs, and you have no idea whether accuracy moved, cost moved, or you got lucky on the three examples you happened to read.

So I do not eyeball it. Stepbook holds the input set completely fixed and replays every step from cache, so I change one thing, chunk size, or fields per call, and read three numbers off each run: fields correct against a labeled set, tokens spent, wall-clock. Now "smaller chunks were more accurate" has numbers under it, or it turns out they just spent more tokens for the same result. Either way it is a call you can defend.

Choosing for the Job

Once you can see the exchange rate, the choice gets concrete. For the nightly run over a few hundred thousand records I optimize for cost: bigger chunks, several fields per call, concurrency turned up, and I reconcile the handful of fields I lose later. For a small high-stakes batch I optimize for accuracy: tight chunks, fewer questions per call, and I let it cost what it costs. Same code, a different tradeoff chosen on purpose.

Feeding a model is a set of trades, made with your eyes open and measured on your own inputs. Three companion posts sit next to this one: which model to reach for in the first place, whether the field needs a model at all instead of a plain parser, and what shape to demand back once you have asked.