Why does AI use so much electricity?

Data centres have consumed serious amounts of electricity for two decades. What changed with AI is the density: how much power a single rack demands, and therefore what a building has to be able to deliver and remove.

Where the power actually goes

Four places, in descending order:

The accelerators. GPUs and purpose-built AI chips are designed to keep thousands of arithmetic units busy simultaneously. A high-end accelerator draws several hundred to over a thousand watts by itself, and a training node packs several of them together. Whereas a conventional server rack might draw 5–15 kW, an AI rack is specified in the tens of kilowatts and, in the newest designs, above 100 kW.

Moving data. Large models are split across many chips, which must exchange intermediate results constantly. High-bandwidth memory and the interconnect fabric consume a significant share of the total — and unlike compute, that share does not fall as quickly with each generation.

Cooling. Every watt that goes in comes out as heat. Air cooling stops being practical somewhere around 30–40 kW per rack, which is why liquid cooling has moved from exotic to standard in new AI halls. Efficiency is tracked with PUE (power usage effectiveness): total facility power divided by IT power. A well-run modern facility sits near 1.1–1.2; older air-cooled sites are considerably worse.

Everything else. Power conversion losses, uninterruptible supplies, networking, storage.

Training is a spike, inference is a load

Training a frontier model runs thousands of accelerators flat out for weeks. It is expensive, visible, and finite.

Serving that model is different in kind. Every request performs computation, and a popular product handles millions per day, every day, for years. The per-request energy is small; the multiplication is not. For any model in real production use, cumulative inference energy overtakes training energy well within its service life — and unlike training, the load never ends.

Two things push inference energy up further:

• Longer outputs. Generation is sequential — each token requires a full pass through the model. Reasoning-style models that produce long internal chains before answering do proportionally more work per request.
• Agents. One user instruction can become dozens of model calls plus tool invocations. From the data centre's perspective that is dozens of requests, not one.

And one thing pushes it down hard: model choice. Energy scales with computation, so serving a task with a small model instead of a large one is not a marginal saving. Routing — cheap model first, escalate only when needed — is the single most effective lever most teams have.

The constraint moved from chips to the grid

For a period the bottleneck was accelerator supply. Increasingly it is electrical interconnection: the queue to connect a new large load to the transmission network, and the lead time for transformers and switchgear. In several major markets those queues are measured in years.

That reshapes the industry in ways worth watching:

• Siting follows power. Operators build where generation and grid capacity exist, rather than near users — acceptable for training, harder for latency-sensitive serving.
• Long-term generation deals. Multi-year power purchase agreements, including with nuclear and geothermal operators, have become standard infrastructure news rather than sustainability announcements.
• On-site generation and storage. Behind-the-meter generation to avoid the interconnection queue entirely.
• Local politics. Electricity price rises and water use are now live municipal issues wherever large facilities are proposed.

Efficiency is improving — and consumption is still rising

Both statements are true, and the tension confuses a lot of coverage.

Energy per token has fallen sharply through better hardware, quantisation, batching, caching, sparse architectures that activate only part of the model per token, and distilled small models that handle routine work. These are not marginal gains; they are large.

Total consumption has risen anyway, because usage has grown faster than efficiency. This is the standard rebound pattern: when a unit of something becomes cheaper, more of it gets used. Efficiency is necessary and it is not sufficient.

What to actually measure

If you operate a service and want a defensible number rather than a headline:

• Facility PUE, from the operator.
• Carbon intensity of the grid where and when you run. The same workload can differ severalfold between regions and between times of day.
• Tokens served per unit of energy, tracked over time. This captures your model and routing choices, which are the parts you control.
• Water usage effectiveness, if you operate in a water-stressed region.

The uncomfortable summary: most of the levers that reduce AI energy use are the same ones that reduce AI cost — smaller models where they suffice, caching, batching, shorter outputs. That alignment is the reason to be moderately optimistic about the per-request trend, and it is also why total demand keeps climbing.