Hard isn't the reason to go expensive. Undefined is.
I run a personal knowledge-graph pipeline that turns everything I read into a structured artifact: claims extracted, gaps flagged, cross-referenced against everything I've read before. It had quietly been running on the most expensive model available. Default by inertia, not by decision.
I switched to a faster, cheaper model for cost reasons: ~5 minutes and ~$1.57 per item, versus ~14-15 minutes and $5-7 on the old default. Roughly 3x faster, 3-4x cheaper. With 211 items in the backlog, that's an 18-hour job instead of a 46-hour one.
I didn't just trust the switch. I spot-checked ten outputs by hand, then had a second, genuinely blind reviewer (fresh context, its own sample) verify sixteen more. No weak ones.
One catch stood out. A CNBC piece on Microsoft's disclosure change reported Azure revenue at $29.42B (33% of total revenue), with growth shifting from 43% to 42% under a narrower segment definition. The model caught something the reporter didn't: if the excluded businesses (GitHub, Security Copilot, healthcare cloud) were dragging growth down, removing them should have pushed growth up, not down a point. Nobody in the source article noticed their own numbers implied the opposite of what happened.
That's not "the cheap model did fine." That's something I'd have wanted a sharp junior analyst to catch, at a third of the cost.
Here's the generalizable part: it's not about whether the content is hard. Dense financial transcripts, technical detail, all of it held up. It's about whether the task already has a spec. This pipeline has one: a fixed extraction template, an existing graph to ground new material against. Even hard content gets tractable when the model isn't also inventing its own approach.
The expensive model earns its cost on a different axis: open-ended problems with no spec, where the model has to decide how to attack the problem before it can start solving it. Architecture decisions. Debugging with no clear root cause. Anywhere a subtle wrong turn compounds silently and nobody's there to spot-check sixteen outputs the way I just did here.
Before defaulting to the biggest model, check for spec, not scale. A task with a spec to follow gets solved cheap. A task with no spec to write still needs the expensive one.
#AI #LLMOps #ModelSelection #AgenticAI #ProductStrategy
