Why AI needs GPUs

A processor designed for general computing has a handful of very fast cores that execute instructions one after another, with elaborate machinery for guessing what comes next. It is excellent at branching, sequential work.

A graphics processor has thousands of simple cores that all perform the same operation on different data simultaneously. That design exists because rendering a screen means doing the same arithmetic for millions of pixels independently.

A neural network layer is a matrix multiplication: every output is a weighted sum of every input. Millions of multiplications, all independent, all at once. It is structurally the same problem as rendering, which is why the chip built for games became the engine of artificial intelligence.

The part that actually limits speed

Not arithmetic. Memory bandwidth. To compute anything, the model's parameters have to be read from memory. A model with 70 billion parameters at two bytes each is 140 gigabytes that must be read for every token generated.

This is why AI accelerators are built around very fast memory stacked next to the processor, called HBM, and why memory supply has repeatedly been the bottleneck for the whole industry rather than wafer capacity. When you read a chip specification, memory bandwidth and capacity predict real performance better than the headline arithmetic figure.

Training and inference are different problems

Training processes enormous batches and needs to hold intermediate values for the backward pass. It is throughput-bound, runs for weeks across thousands of chips connected by very fast networking, and the networking is as important as the chips.

Inference generates one token at a time for many users at once. It is memory-bandwidth-bound and latency-sensitive. Different hardware can win here, which is why competitors have made more progress in inference than in training.

Inference now consumes more total compute than training, because training happens once and inference happens forever.

Why the software matters as much as the chip

The dominant vendor released a programming platform in 2007 and spent nearly two decades building libraries, compilers and tuned routines on top of it. A competitor with better silicon still has to make thousands of existing programs run well on day one. That gap is the moat, and it closes fastest for simple repetitive workloads.

What else exists

Custom accelerators built by large cloud operators for their own workloads, which never have to win a public benchmark. Startup chips optimised for fast inference of transformer models. And ordinary processors, which run small models perfectly well and are what your phone uses.