reading
9 min read

Tokens per watt will define 2027 and beyond

Here are the technologies that will have the biggest impact on efficiency

Feyza Haskaraman

close up of a chip

We are entering an era of power scarcity.

According to SemiAnalysis (opens in new tab), the U.S.’s public grid will run out of headroom in 2027. Once in the red on headroom, a grid’s ability to provide power becomes unreliable. And once there, it could be years until we’re out.

Over the past decade, we’ve measured the velocity of the AI infrastructure build out in FLOPS, then in GPUs, then in dollars. These metrics all assumed compute was the scarce input. Now, efficiency has become the fastest source of new compute capacity. And as AI infrastructure shifts its focus towards efficiency, tokens per watt will define 2027 and beyond. Tokens per watt measures how many tokens are generated per second for each watt of power the hardware draws. Today’s baseline, a Vera Rubin NVL72 facility, is about 59 tokens per second per watt.

What comes next? We need to reimagine how a watt becomes a token at each layer of the stack, and even how that watt is produced. Some like Crusoe (opens in new tab) will reimagine the entire stack from start to finish. But with demand for compute continuing to accelerate, there’s opportunity for efficiency at each point on the chain.

technologies' impact on tokens per watt baseline, ranked

Fortunately, a cohort of new technologies have emerged to address the shortfall. We’ve analyzed roughly how many tokens each adds to a watt of power from the current NVIDIA Vera Rubin baseline of 59 tokens per second per watt. The numbers themselves are directional, not exact, as each blend ranges from different set ups. But ranked, they give a sense of the scale of the opportunities, and what a path towards more tokens per watt could look like.

The numbers towards the bottom of the chart might seem negligible, but at scale they make a massive difference. These are also technologies that can be stacked one on top of another, while those at the top are indicative and represent a wholesale change.

We think of each of the following technologies as a lever. Flipped, each improves the amount of tokens we can extract from our current baseline.

The generational opportunity: Inference chips

The biggest opportunity is also the most complicated: memory. Up to 85% of dynamic GPU energy isn’t spent on compute (opens in new tab); it’s spent on the memory retrieving the context that the model runs on. If we can eliminate the need to move memory, we could theoretically be able to generate thousands of tokens per watt of power (opens in new tab).

To do this, we have to first rebuild the chips that our models run on. This is Fractile (opens in new tab)’s approach. The upside is as vast as the complexity involved: Fractile’s targets would not only solve the efficiency bottleneck, but make entirely new categories of work economically viable by unlocking the kind of longform agentic reasoning that’s often unreliable now.

The 3-4x opportunity: Serving software

To date, the greatest performance improvements have been to the serving software at the chip level. The adaptability of the software layer is key to the steep curve of its efficiency gains: it’s hardware agnostic and can be shipped any time. That’s why we see algorithmic price-performance improving about 3x a year (opens in new tab) while hardware price-performance has previously averaged only 1.3x annual improvements.

Annual price-performance improvements

  • 3x

    algorithmic

  • 1.3x

    hardware

Crusoe recently measured the gains (opens in new tab) you could isolate to just software. In October 2026, its inference team ran the same model on the same two nodes with byte-identical traffic, once on stock vLLM defaults and once with its production scheduling and transport settings. The hardware stayed the same. The tuned stack served 37% more tokens per watt — that’s 27% less energy per output token — while clearing the same quality benchmarks. These are percentages on top of already-shipped hardware, but we’re also seeing the same software making whole new chips usable (opens in new tab).

The 1.4-1.8x opportunity: Efficient CPUs

When you assign an agent a task, it usually has to bounce between dozens of subactions to complete it: web searches, retrieval calls, code execution, etc. Each of those actions has to run through the CPU in each chip, not the GPU. In agentic workflows, CPU tasks account for 50–90% of total task completion time. As we use more and more agents, CPU capacity is rapidly becoming a bottleneck.

We’re seeing some promising approaches to solving this. Cambridge-based Tendrils (opens in new tab) eschews the single memory, multi-core CPU that’s become industry standard for graph-based models called interaction nets (opens in new tab) that they claim are able to handle scaled workloads much more efficiently.

The 1.3-1.5x Opportunity: Dynamic load management

More efficient consumption is one way through power scarcity. Another is getting more usable watts out of existing facilities. Sites are sized for every GPU running at peak draw, but most don’t consistently run at that level. NVIDIA’s MaxLPS (opens in new tab) (short for “Maximum Land Power Shell”) uses that gap to oversubscribe power, like an airline overbooking a flight. Every few seconds, its software resets per-GPU power limits, moving unused headroom to the GPUs that need it while the site stays within its fixed consumption. According to Nvidia, this can pack 40% more GPUs within the same site power budget. When Nscale ran MaxLPS on GB300s, it went from 4.10 to 6.12 tokens per second per watt, roughly 50% more tokens from every watt. Startups like Emerald AI (opens in new tab) are taking similar approaches to improve power efficiency.

The 1.15x opportunity: New liquid cooling tech

Liquid cooling isn’t new: it accounts for a lot of the current gaps between buildings’ power use effectiveness. The cooling plates that dominate the market are limited by their inflexibility and uniformity. A new wave of tech is routing liquid cooling directly to where the transistor is hottest, via microfluid channels like those from Corintis (opens in new tab) or direct-to-chip applications like those Frore Systems (opens in new tab) recently debuted.

The 1.04-1.09x opportunity: Solid-state transformers

There are two ways to make a GPU system more powerful: connect GPUs across racks fast enough that they act as one, or pack more power-hungry GPUs into a single rack. In the near term, the first path is most viable. Optical switches, which we talk about below, let the scale-up network stretch from GPUs in a single rack to GPUs across many racks. That allows next year’s system to be more powerful than this year’s, just spanning several racks instead of one.

But that’s a bridge solution. Ultimately, the biggest gains come from denser racks, and denser racks need a new power path. A conventional AC power path loses about 18% of its power (opens in new tab) across roughly seven conversions. Solid-state transformers like those built by Heron Power (opens in new tab), DG Matrix (opens in new tab), Amperesand (opens in new tab), and Eaton (opens in new tab) all simplify the conversion stack. Manufacturing and supply chain constraints mean this shift won’t reach scale in 2027, but when it does, SSTs will be core infrastructure for every dense AI rack.

The 1.02-1.05x opportunity: Optical circuit switching

Optical circuit switching has been a good idea for a while (opens in new tab), but the milliseconds it took to reconfigure an optical switch didn’t make sense to gain nanoseconds networking. AI changed that. It allows us to set up and leave the switch. At the same time, transceivers have grown into roughly half the cost of the network and more than half its power, so deleting an entire electrical-optical conversion layer went from nice-to-have to structural.

Solutions are beginning to hit the market. Lumentum and Coherent both ship optical circuit switches, and NVIDIA put $2B into each (opens in new tab) in March 2026.

Beyond the stack: Storage and power electronics

More efficient consumption is one route to navigating power scarcity. But there’s also an opportunity for us to provision more watts from facilities. Sites are sized for peak draw and training load swings, and the common fixes for those swings (dummy workloads, GPU power floors) burn roughly 6-7% (opens in new tab) of a facility’s energy that produces no tokens. Generative AI has exacerbated this burn, since grids were built for the relatively predictable consumption needs of factories and office buildings, not the intense computation bursts that characterize GPU usage. Reliability is similarly important when you take the provisioning lens: as data centers scale, time offline caused by a disturbance incurs greater and greater costs.

Storage and power electronics can close the gaps in energy production and tokens provisioned at the site-level. Medium-voltage batteries that sit between the grid and the campus smooth out the fluctuating swings in power consumption that generative AI causes, eliminating energy waste and stabilizing the grid. This is why Crusoe is deploying 5 GW of ON.energy’s AI UPS (opens in new tab) across its campuses through 2027. It’s a way to make sure every watt we;re allowed to provision gets the chance to become a token.

Beyond the stack: Behind-the-meter generation

To address power scarcity, we have to look for new ways to produce more raw watts of power beyond what the public grid can provide. Behind-the-meter generation barely existed as a category two years ago. Now, it’s become essential. SemiAnalysis now counts 75 GW of firm, binding orders (opens in new tab) for behind-the-meter AI compute. Roughly 20 GW of that was booked in the second quarter of 2026 alone.

The business logic behind this is sound. A power plant for a gigawatt-scale island campus costs about $5B. Inference revenue pays back the power plant alone in months, and the whole campus in just a few years. What we’re interested in is how to supply that power. Felicis has made three investments there: Radiant (opens in new tab), whose one-megawatt reactor ships in a container; Exowatt (opens in new tab), which banks concentrated sunlight as heat and converts it back on demand; and Endurance Energy (opens in new tab), which is going after the heat at the mid-ocean ridges. All use different physics but are premised on the watt arriving from somewhere other than the queue.

What we're looking for

The AI buildout will be constrained by physical infrastructure (opens in new tab) for the rest of this decade. If you're building anywhere along this chain, we'd like to talk (opens in new tab). How much does your technology actually move tokens per watt? How are you contributing to the path to 10x token per watt? We're most interested in the unglamorous parts, and we can often be the most helpful during the conversations that might feel too early.

Thank you to the Crusoe team for your thought partnership and review on this post.

Authors

  • Feyza Haskaraman

    Partner

Tags

  • Global Resilience

Share