One Step at a Time: Debugging LLM Pipelines Without Re-Running the Whole Thing

For the last few months I've been building a big LLM pipeline — it splits a document into twelve sections, then extracts, classifies, and verifies its way through them. Each step makes several passes over every section, so a single step is sixty-odd model calls; the full pipeline, a dozen steps deep, runs to hundreds. Satisfying work when it goes well. But I kept falling into the same loop — not a bug, a workflow one, and the kind you don't notice until it's already eaten your evening.

The Loop That Wouldn't End

Picture the tail of that pipeline. The last step, a verification pass, keeps coming back wrong. So I tweak its prompt and run it again.

And the whole thing runs again. Every step ahead of it re-fires — hundreds of calls, the exact same output as last time — while I wait to find out whether my one edit to the last step helped. Usually it hadn't, so I'd do it again. Tweak, run, wait, pay. Repeat. To iterate on the final step, I was re-running hundreds of model calls every single time, paying for all of them just to look at the one I actually cared about. Parallel or not, that's a slow, expensive way to change one prompt.

And when the output did land, "is this better than last time?" meant staring at two blobs of JSON in my terminal. For a nondeterministic step, "did this change because of my edit, or because the model just felt different today?" was a question I mostly answered by vibes.

That's a terrible way to work: it's slow, it costs real money, and it's blind.

What I Actually Wanted

Stripped down, three things:

1. Edit one step and re-run only that step — and whatever genuinely depends on it. Not the whole graph.
2. See every step's output, not just the final one, so I know where a regression crept in.
3. Measure drift — run a step ten times and know how much it moves, instead of guessing.

I went looking for the tool that did this. The good LLM observability platforms — LangSmith, Langfuse, Braintrust — are built for production: watching a deployed app, in a cloud dashboard, after the fact. That's a real job. It just isn't my job at 11pm with a prompt that won't behave. I wanted something for the dev loop, running locally, in my repo, before anything ships.

Borrowing an Idea from the Frontend

Frontend developers already solved a version of this. It's called Storybook: pull one component out of the app, see it on its own, snapshot it, tweak it — without booting the whole thing.

So I built Stepbook: that same idea, but for a pipeline of typed steps — especially the LLM kind. Think of each step as a component you can pull out and open on its own.

The piece that kills the wasted reruns is replay. Run the pipeline once and every step's output — all those calls — gets recorded. From then on each step is keyed by its input and the hash of its source file, so when I edit the last step's prompt, only that step goes stale. Everything ahead of it replays from the recording instead of hitting the model again — instant, and free. The run that used to fire hundreds of calls now fires only the ones I actually changed.

Watching the Loop Get Faster

Now every step's output is captured as something I can open and read, not just the final blob. I make my edit, run again, and — as in the clip above — the steps ahead replay in an instant while only the one I touched actually calls the model. Cost and tokens roll up into a per-run total, since each step reports them as it goes.

All of it lives in a state/ folder in the repo — snapshots, the cache, a run log. Nothing leaves my machine. No account, no cloud dashboard, no shipping my prompts somewhere just to debug them.

But Isn't This Just Testing?

If you've read this far, part of your brain is probably yelling dude, just write tests. Mine was too — steps with typed inputs and outputs is exactly the shape that screams "assert on it and move on." And for the deterministic steps, that's right: Stepbook has assertions, and you should use them.

It breaks the moment a step is a model call. A unit test asserts one output for one input — deterministic in, deterministic out, green or red. But an LLM step doesn't have an output; it has a distribution. Same input, ten runs, ten different answers. There's no assertion to write for "how consistent is this," because the answer isn't a value — it's a spread. A green test on a single run tells you nothing about the other nine, and "did my edit make this steadier, or did I just get a lucky sample?" is the question that actually decides whether a prompt is done.

So Stepbook measures the spread instead of pretending it isn't there: run a step N times, cluster the outputs by similarity, flag the outliers, and watch a stability number move as you tune. And because the steps upstream replay for free, checking one step's variance twenty times costs twenty calls — not twenty whole pipelines.

The Same Trick Works for Agents

The thing that records a straight-line pipeline records a loop just as well. An agent is really just a pipeline whose runner loops — and each turn becomes its own node with its own input, output, checks, and cost. Call the same step twenty times and you get decide#0, decide#1, decide#2, and on down — each turn inspectable on its own and diffable against the others. "What did my agent actually do on turn three, and why" stops being a black box you re-run to peek inside; you just open turn three. And because every turn is recorded, you can replay the whole loop against the frozen responses for free while you fix whatever went sideways downstream.

Where This Goes

Stepbook is alpha. It's MIT-licensed, it's on npm, and the whole thing runs locally. To be clear about the lane: it's not production observability — the tools I mentioned own that, and they're good at it. This is for the messy part before production, the tweak-run-wait-pay loop I couldn't stop falling into.

Which brings me to the actual lesson. The tokens and the latency were real, but they were symptoms. The real barrier was working blind — re-running everything, one full pass at a time, just to see one step. Once I could see every step and re-run only the one I touched, the whole loop changed shape.

If you've been stuck in that same loop, I'd genuinely love for you to try it and tell me where it breaks.

npx @stepbook/cli init

Docs at stepbook.dev, source on GitHub. There's more to come.