When Cheap Models Beat Frontier Models: Six Workloads We Tested

"We need the best model" usually means "we need the most expensive model." That conflation costs companies six figures a year on tasks where frontier capability adds nothing measurable.

Autark's eval suite covers knowledge reasoning, coding, long context, and agentic benchmarks. On many everyday workloads, Autark Deep ranks first or second against Opus, GPT, and Gemini while routing through tiers priced at $5/$10 per million tokens (input/output), not $15–$30/M flagship direct rates.

Here are six workloads where routing beats all-flagship every time.

How we evaluated

We used prompts representative of production knowledge work, not academic trivia. Quality was judged on task completion (correct classification, valid JSON, usable draft), not stylistic flourish.

Pricing comparisons use illustrative direct flagship rates (~$15/M output) vs Autark/Ultra tiers from our pricing page. Your exact savings depend on volume and routing mode.

1. Intent classification

Task: Route incoming support messages to the right queue (billing, technical, sales).

Typical prompt: 200–500 input tokens, 10–30 output tokens.

Why flagship is overkill: The output space is small and structured. Models do not need deep reasoning to label "refund request" vs "login issue."

Routing win: Autark Flash or Autark tier handles this at $2–$6/M output vs $15/M+ direct flagship. At 100K classifications/month, the difference is thousands of dollars on output tokens alone.

2. Document summarization

Task: Condense a 10-page report into five bullet points for an executive brief.

Typical prompt: 8K–20K input tokens, 200–400 output tokens.

Why routing works: Summarization is a well-studied capability. Open models score competitively on MMLU-Pro and GPQA-style knowledge tasks in our eval data. Autark Deep hits 87.5% on MMLU-Pro vs 89.1% Opus and 91.3% GPQA Diamond vs 91.3% Opus, at lower tier pricing.

Routing win: Quality parity on structured summaries; cost drops because you are not paying flagship output rates on every page of input context.

3. Email draft and rewrite

Task: Turn bullet notes into a professional client email, or tighten verbose copy.

Typical prompt: 300–800 input tokens, 150–300 output tokens.

Why routing works: Style and format tasks rarely need multi-step reasoning. Errors are obvious to human reviewers before send.

Routing win: Autark tier default. Reserve Autark Deep for high-stakes external communications if your policy requires it.

4. RAG Q&A over internal docs

Task: Answer "what is our refund policy for enterprise contracts?" using retrieved chunks.

Typical prompt: 4K–15K input tokens (retrieval + question), 100–250 output tokens.

Why routing works: Retrieval does the heavy lifting. The model synthesizes known text; it is not inventing from scratch.

Eval note: Autark Deep scores 62% on CorpusQA 1M vs 71.7% Gemini on long-context QA in our sheet, but ranks second on MRCR 1M at 83.5% vs 92.9% Opus. For typical RAG chunk sizes (not full 1M context), Pro/Ultra tiers are sufficient and cheaper than sending every query to the longest-context flagship.

5. Code explanation (not generation)

Task: Explain what a 40-line function does for a new team member.

Typical prompt: 500–2K input tokens, 200–500 output tokens.

Why routing works: Reading and explaining existing code is easier than writing correct novel code from scratch.

Eval note: Autark Deep leads on LiveCodeBench (93.5%) and Codeforces rating (3206) in our benchmarks, competitive with frontier on code tasks. Explanation workloads need less than full generation capability.

6. Structured data extraction (JSON)

Task: Extract {name, date, amount, vendor} from an invoice paragraph into valid JSON.

Typical prompt: 400–1K input tokens, 50–150 output tokens.

Why routing works: Output schema is constrained. Validation is mechanical (JSON parse, required fields).

When cheap models fail: Handwritten OCR garbage, ambiguous tables, or fields requiring cross-document inference. Escalate those to Autark Deep.

When cheap models do fail

Be honest about the boundary:

• Multi-step research across conflicting sources
• Novel agentic plans with tool chains and recovery logic
• Highest-stakes reasoning (legal strategy, complex architecture decisions)
• Lowest-error-tolerance production code generation with no human review

For those, Ultra or pinned frontier models are appropriate. The mistake is using them for all six workloads above too.

Worked example: monthly spend

Assume 500K requests/month, blended 2K input + 500 output tokens per request on classification-heavy traffic:

Input costs add on top, but the ratio holds: routing saves 50–70% on typical operational mixes, consistent with production gateway benchmarks.

The bottom line

Frontier models are not wrong. Using them for everything is wrong.

Match the model to the workload. Measure on your prompts. Route by default.

Benchmarks & ROI · Pricing