Benchmark Series #1: Autark on KWBench
We're launching a public benchmark series. No cherry-picking. Third-party datasets. Published methodology. Starting with KWBench — the hardest knowledge-work benchmark in public AI.
The benchmark
KWBench tests whether models can identify what kind of problem they're looking at before attempting to solve it. Tasks are sourced from practitioners across acquisitions, contract negotiations, fraud analysis, and strategic reasoning. Every task carries structured ground truth from domain experts.
The scoring is brutal: a three-tier rubric with a mandatory gate. Fail any core criterion, score zero. The KWBench paper evaluated 16 models. The best single model scored 27.9%. Routing across multiple models reached 50.7%.
How we tested
We selected KWBench tasks that test pure reasoning — research analysis, strategic communication, market assessment, creative work. Tasks requiring code execution or file access are excluded from this round. Same judge as the KWBench paper: Gemini 3 Flash.
For every task, we sent a single API call to Autark. Under the hood, our routing engine identified the task type and dispatched it to the model proven best for that domain.
The result
Autark routing: 54% pass rate. Nearly double the best single model at 27.9%.
| Category | Tasks | Autark | Best single model |
|---|---|---|---|
| Research & analysis | 13 | 54% | ~28% |
| Creative & content | 6 | 50% | ~28% |
| Communication | 4 | 38% | ~28% |
| Product & strategy | 5 | 30% | ~28% |
How it works
Autark sends every task to a different model — whichever is proven best for that specific type of work. Research tasks, creative tasks, strategic communication, product work — each gets routed to a model that has demonstrated an edge on that category.
No single model wins every category. That's the point.
What's next
We're expanding to cover the full KWBench dataset, including tool-assisted tasks. Same judge, same methodology, same public results.
We'll also report on additional benchmarks in the series. If there's a benchmark you'd like to see Autark run against, tell us.
Run it yourself
We built a general-purpose benchmark runner. Load any dataset, run it through Autark, compare routing decisions, export results. Same tool we used for this test.
Methodology: KWBench text-only tasks (research, creative_writing, communication, product, strategy). Judge: Gemini 3 Flash. Scoring: three-tier rubric with mandatory conjunctive gate, per the KWBench paper. Full results available on request.