Stop Chasing the Best AI Model. Start Defining the Problem Better!

During an agentic coding session, my AI assistant hit an accuracy wall on a classification task and decided to swap models.

2.5 is an older version than 3.1. Why does the AI think that the older model would be stronger? So I benchmarked it: 5 models, same task, same prompt, temperature 0. The results reminded me of a lesson about model selection in multi-agent architectures.

What do the benchmarks show?

The task: categorize items into one of 20 domains. The kind of work that powers agent routing, support triage, content moderation, and intent detection.

The AI assistant was right. Gemini 2.5-flash (82%) outperformed both newer models - including 3-flash-preview (80%), which is the same tier, just a newer generation.

Google’s Gemini family runs a clean three-tier architecture: Pro for heavyweight reasoning, Flash as the balanced workhorse, Flash-Lite for high-volume cost-optimized workloads. Newer generation doesn’t automatically mean better accuracy on every task. Each generation is optimized for different strengths, and on this particular classification workload, 2.5-flash’s training mix gave it an edge over its successor.

But the ranking isn’t the only real finding. All 5 models scored within 4 percentage points of each other.

Why do all models hit the same ceiling?

If the gap between cheapest and most expensive was 30 points, the advice would be simple: pay for the best model. But 4 points?

I dug into areas where models disagreed:

• 54% of inputs: all 5 models correct. Clear-cut categories.
• 42%: models disagreed. Category boundaries overlapped.
• 4%: all 5 wrong. Genuinely ambiguous.

The disagreements weren’t random. All models struggled with the same boundary cases - inputs that legitimately fit two categories depending on how you frame them. Like routing a complaint about a delayed payment: is that “Payments” or “Customer Service”? And all 5 models made the same types of errors, regardless of architecture, price, or generation.

The accuracy ceiling isn’t set by model capability. It’s set by the problem definition. When category boundaries are fuzzy, no model can perform better than the taxonomy allows. The cheapest model and the most expensive model both hit the same wall.

This is the insight that matters for anyone selecting models for agentic AI: when all models converge, upgrading the model is marginal. Sharpening the problem definition is where the real gains live.

What does this mean for multi-agent architecture?

The benchmark tested a single task. In multi-agent systems, this pattern compounds.

Consider a three-agent pipeline: a classifier routes inputs to specialist agents, specialists process them, and a synthesizer produces the final output. If the classifier runs at 80% accuracy and each specialist at 90%, end-to-end accuracy drops to roughly 0.80 x 0.90 x 0.90 = 65%. Errors multiply across agents. A 4-point improvement at one stage doesn’t give you 4 points end-to-end; compounding through the chain, it nets you about 2-3 points.

79% of multi-agent production failures trace to specification and coordination issues, not to model capability. My benchmark saw the same pattern at the single-agent level: 42% of errors came from how the problem was specified, not from how the model processed it.

And the lesson applies beyond routing. Every time you scope an agent’s responsibilities - what it specializes in, how it decomposes tasks, how it coordinates with other agents, what quality criteria it uses for self-evaluation - you’re drawing the same kind of boundaries. Fuzzy agent scopes produce the same kind of compounding errors that fuzzy classification categories produce.

Teams I’ve seen succeed with multi-agent systems invest in harness engineering before they touch model selection. The harness is everything around the model: prompt structure, agent boundary definitions, handoff criteria, evaluation logic, and test cases for ambiguous inputs. Build that scaffolding first, pick your model last. A well-engineered harness running a cheap model will beat a poorly-defined agent running a premium one, because the harness is what turns a 78% classifier into a reliable pipeline component.

Google’s own documentation recommends using Flash-Lite as a classifier that routes queries to Flash or Pro, and the Gemini CLI uses exactly this pattern. Even Google treats the routing layer as a “good enough” problem and saves the premium model budget for the agents that need deep reasoning.

The little surprise 😊

Google recommends cheap models for routing but I wondered: how cheap can you actually go?

A 35B-parameter open model running on my homelab’s AMD Strix Halo - a mini-PC APU drawing about 54W, roughly what a light bulb uses - matched Google’s paid lite models in accuracy (78%). It came within 0.6 seconds of the fastest cloud model and had the most consistent latency of the group (1.4-1.9s range vs 3.8-12.5s for the premium tier).

Cost per inference: $0.

For enterprises with data residency requirements or teams already running GPU infrastructure, this changes the math entirely. High-volume, repetitive tasks - routing, tagging, categorization - can run locally with equivalent accuracy and better latency predictability. No data leaves your infrastructure. No per-call billing. The cloud API cost premium makes sense for frontier reasoning tasks where the latest generation genuinely outperforms.

My key takeaways

Three things I learned from running this benchmark:

Define the problem before evaluating models. When all models converge, the bottleneck is the specification. Sharpen your agent scopes, your category definitions, your handoff criteria. That’s where accuracy improvement lives, and it’s free.

Benchmark on YOUR data. Public leaderboards rank models by general capability. On your specific task, the rankings reshuffle. A one-afternoon benchmark with ground truth labels gives you actual data for your decision.

Match the model tier to the task type. Use lite/mini for routing and categorization. Reserve premium tiers for agents doing multi-step reasoning and synthesis. The 24x cost premium between tiers buys reasoning depth, and that depth is wasted on tasks with a fixed accuracy ceiling.