Benchmark Series #2: Autark on LiveBench
We submitted Autark to LiveBench — a public, objective benchmark with no LLM judge. 682 questions across reasoning, math, language, data analysis, and instruction following. Every answer graded against verifiable ground truth.
We sent every question to model: "autark". Under the hood, our classifier identified each task type and A-TEM routed it to the proven best model. No hardcoded maps. No manual tuning. Automatic.
The result
Autark: 85.7% global. Best single model on the leaderboard: 85.0%.
| Category | Autark | Best single model |
|---|---|---|
| Language | 94.0% | 92.0% |
| Reasoning | 93.0% | 93.0% |
| Instruction following | 90.0% | 91.4% |
| Math | 78.8% | 76.9% |
| Data analysis | 72.8% | 71.5% |
Autark matched or beat the best single model in four of five categories. That model — DeepSeek V4 Pro — is one of the models we route to. But it's not the right model for every question. When it's not, we send the question elsewhere.
And it costs less
The benchmark processed 3.8 million tokens. Here's what that would cost:
| Approach | Cost | Accuracy |
|---|---|---|
| Autark | $7.25 | 85.7% |
| DeepSeek V4 Pro (retail) | $12.75 | ≤85.0% |
| Frontier (GPT-5.5, Claude Opus) | ~$57 | — |
Autark is cheaper than buying the best single model at retail prices — and more accurate. Compared to buying a frontier model directly, it's nearly eight times cheaper for equivalent accuracy.
This is the worst case
LiveBench is an intentionally hard benchmark. Reasoning puzzles. Olympiad math. Adversarial language tasks. It's designed to stress-test models at their limits.
Real-world usage is different. Most requests aren't zebra puzzles. They're "summarize this email," "draft a follow-up," "classify this ticket." Those tasks go to cheaper, faster models — dropping your blended cost further.
The LiveBench numbers are a ceiling. Your actual costs are lower.
How it works
Every request to Autark goes through three steps:
- Classify. What kind of task is this? Reasoning? Math? Data analysis? Writing?
- Route. Which model is proven best for this specific task type? Check the evaluation matrix.
- Deliver. Send the request, return the response. Same OpenAI-compatible format.
No model selection. No provider management. No guessing which model to use. One endpoint, one API key, always the right model for the job.
Methodology: LiveBench public questions (release 2024-11-25), 682 tasks across 5 categories. Automatic classifier-driven routing via A-TEM. Objective scoring against ground truth answers. Cost basis: output tokens at list prices. Full results and per-task data available on request.