The Inference Inflection: Why AI's Biggest Market Is Still Wide Open

"The inflection point of inference has arrived. It's way past training now."

That's Jensen Huang at GTC 2026 in San Jose, announcing that Nvidia expects $1 trillion in chip orders through 2027, double its prior outlook. "AI now has to think. In order to think, it has to inference. AI now has to do. In order to do it, it has to inference."

He's not exaggerating. The numbers behind inference growth are so large they're difficult to internalize. Google now processes 3.2 quadrillion tokens per month across its products, a 330-fold increase over two years. OpenRouter alone routes over 25 trillion tokens per week, up 5x in six months. The global daily token throughput is estimated at roughly 360 trillion tokens. Per day.

In early 2026, the cumulative global spend on running AI models officially surpassed training spend for the first time, what analysts now call the "Inference Flip." Inference currently accounts for roughly 80 to 90 percent of total AI compute dollars over a model's lifecycle. For every dollar spent training a model, somewhere between fifteen and twenty dollars will be spent running it. Training is a capital expense. Inference is the operating system.

This isn't a trend that might happen. It's the economic reality already underway.

Graphs that broke their own forecasts

The most revealing number isn't a token count, it's how badly the forecasts missed.

In October 2025, Dell COO Jeff Clarke said the company modeled inference reaching 1 quadrillion tokens annually by 2028. By the time he spoke, Dell had already revised that estimate upward by a factor of 57, to 57 quadrillion, and he said they were probably still wrong. He was right. Current token consumption is already roughly 135 quadrillion per year, more than double Dell's revised forecast, with years to spare.

Goldman Sachs projects token processing will hit 47 quadrillion monthly by 2028, a 70-fold increase from mid-2025 levels. Agentic AI, autonomous systems that plan, retrieve, and execute multi-step tasks, is expected to account for 84 percent of all workloads by 2030, up from near zero today. The same bank noted, with some understatement: "We weren't talking about agents a year ago. Now we are."

Consulting firm Tirias Research first forecast annual token output at 20 trillion by end of 2024. Actual usage hit 667 trillion, more than 33 times higher. They revised to 77 quadrillion for 2030. Current consumption is already nearly twice that.

The pattern is consistent: every serious forecast has been wrong, always in the same direction.

Why training was Phase 1

For the first three years of the AI boom, the narrative was about models. GPT-4. Claude. Gemini. Llama. The race to build bigger, smarter, more capable systems consumed the industry's attention and most of its capital. Training runs cost hundreds of millions of dollars, took months, and required thousands of interconnected GPUs. Nvidia built a monopoly on training hardware, and the scarcity drove prices and margins to extraordinary levels.

That phase created the foundation. It also created a misunderstanding about where the real economic value would accrue.

Training produces a model. Inference produces revenue. A model gets trained once and then used billions of times. Every ChatGPT question, every API call, every agent task, every line of code written by Claude Code or Cursor, every single one of those consumes inference compute. The model is the fixed asset. Inference is the revenue stream. And the revenue stream is scaling much faster than anyone expected, for one structural reason: the models themselves are consuming more inference per task.

OpenAI's reasoning models, the o-series, changed the math. Before o1, inference was straightforward: you sent a prompt, the model generated tokens, you got an answer. The new reasoning models think before they answer. They backtrack, self-correct, evaluate multiple paths, and consume four to ten times more inference compute per request than the generation before them. OpenAI researcher Noam Brown demonstrated that for reasoning tasks, 15 times the inference compute produces roughly the same quality improvement as 10 times the training compute. Inference is now a scaling law of its own.

And then came agents.

Anthropic estimates that multi-agent systems, where multiple AI models collaborate on complex tasks, consume up to 15 times more tokens than a standard chatbot interaction. Coding agents, which read entire repositories, plan changes, write tests, and iterate, can consume a thousand times more tokens than simple code generation. Goldman Sachs' forecast that agentic tasks will represent 84 percent of all AI workloads by 2030 isn't just a prediction about technology, it's a prediction about economics. Every percentage point that shifts from chat to agents multiplies total inference demand.

Why the inference market is up for grabs

If inference were just a scaled-up version of training, rent GPUs, run models, collect money, the hyperscalers would own it. They own the data centers. They own the power contracts. They have the capital.

But inference is structurally different from training, in ways that create openings.

First: training hardware is a monopoly. Nvidia's GPUs, interconnects, and software stack make training on anything else impractical for frontier-scale work. Inference is different. It can run on any hardware: GPUs, LPUs, TPUs, even CPUs at the low end. The chip market for inference is competitive in a way training never was. Nvidia's own GTC announcement confirmed this: the Groq acquisition and the new LPX inference chip are explicitly aimed at inference workloads where Nvidia's GPU monopoly doesn't naturally extend.

Second: training is concentrated among a handful of labs. OpenAI, Anthropic, Google, Meta, xAI, maybe five to ten companies with the resources to train frontier models. Inference is distributed. Every company using AI is an inference customer. The market isn't winner-take-all by structure, it scales horizontally across industries, geographies, and use cases.

Third, and most important, no single model wins at everything. The model field is fragmenting faster than consolidation can occur. New open-weight models launch every week. Some are better at reasoning, others at writing, others at code, others at classification. Some cost twenty times less per token and produce the same quality for specific tasks. The winners in inference won't own the best model. They'll combine them.

The hyperscalers are structurally incapable of doing this. Google, Amazon, and Microsoft are locked into exclusive partnerships with frontier labs. Their incentive is to sell one model at the highest rate they can justify. They have no path to neutrality. The market is open for companies that can sit above the models, not inside one of them.

What this means if you're running AI workloads

Most companies currently pick one model provider and send everything to it. This made sense two years ago, when the choice was effectively GPT-4 or nothing. It makes no sense now. There are roughly thirty open-weight models worth evaluating at any given time, the set turns over every few months, and the cost spread between the most and least expensive model that produces the same quality for a given task can be a factor of twenty.

The problem is that nobody at a hundred-person company owns the job of figuring out which model to use for which task. They don't have the evaluation infrastructure, they don't have the time, and they shouldn't need to. The inference market's fragmentation is an opportunity for the companies who solve it, and an invisible tax on everyone who ignores it.

The per-token price of inference has fallen roughly 280-fold since late 2022. GPT-4 class capability, which cost 30 dollars per million tokens in March 2023, is now available at around 40 cents. But as Gartner warns, these savings do not pass directly to the customer. Lower per-token costs enable more advanced capabilities, reasoning, agents, tool use, which consume disproportionately more tokens. Total inference spend continues to rise even as unit costs fall. The organizations that win will be the ones that can deliver the best answer per dollar of inference spend, not the ones that find the cheapest per-token rate.

Tokens are the new kilowatt-hours. Cost per unit is falling. Total consumption is exploding. The gap between the two is where value gets created, and it's a gap that no single model provider, and no hyperscaler, is positioned to capture.

At Autark, we measure every leading model continuously against real-world tasks, not benchmarks, but the actual work people do, to know which model performs best at what. That intelligence powers a single API. Better answers. Lower cost. No model selection required.