The model in your pocket

Every AI feature has a placement decision: run it in a data center or run it on the thing in the user's hand. The cloud gives you the largest models and a per-request bill. The device gives you zero marginal cost, no network round trip and data that never leaves, at the cost of a model perhaps a hundredth the size.

What makes a model fit

Three techniques shrink models enough to run locally, and all three trade a little quality for a lot of size.

Quantisation stores weights at lower precision, four bits instead of sixteen. That is a fourfold size reduction for a small and usually acceptable quality loss, and it is the single most effective lever.

Distillation trains a small model to imitate a large one on a target distribution. The small model never matches the teacher generally, but on the narrow band of tasks it was distilled for it can come close.

Architecture choices such as activating only part of the network per token, or sharing attention keys across heads, reduce the memory bandwidth needed per token, which is what actually limits speed on a phone.

The constraint that surprises people

It is not compute. Modern phone and laptop chips have dedicated neural accelerators with more than enough arithmetic. It is memory bandwidth and thermal budget. Generating a token requires reading the model's weights, and a phone's memory bus is slow compared with the memory next to a data center accelerator. Sustained generation also heats the device, and a phone that gets hot throttles within a minute.

This is why on-device models are excellent at short bursts, summarising a message, transcribing speech, rewriting a paragraph, and poor at anything that runs for minutes.

Where local wins outright

Privacy-bound data. Medical notes, messages, photographs, anything where sending the content elsewhere is a legal or reputational problem. This is the strongest argument, and it is increasingly a regulatory one in Europe.

Offline and unreliable networks. Translation abroad, a machine on a factory floor, a vehicle in a tunnel.

High-frequency small tasks. Anything a user triggers hundreds of times a day, where a cloud call costs a fraction of a cent each and the fractions add up to a real bill.

Hard latency limits. Anything in a control loop, which is why robot policies run locally even when a bigger model is available remotely.

The hybrid pattern that actually ships

Almost every serious product ends up doing both. A small local model handles routine requests and decides, in a few milliseconds, whether the question needs the large one. The user sees an instant answer most of the time and a slightly slower, better answer occasionally, and the operator pays for the cloud only on the fraction that needs it.

Getting the routing right is the whole engineering problem, and the honest way to evaluate it is to measure how often the local model answers something it should have escalated.

What it means for the economics

If a meaningful share of inference moves to devices that users already paid for, the cost of delivering AI features falls for the operator and the demand for data center capacity grows more slowly than the straight-line forecasts assume. Nobody knows how large that share becomes. It is the most underrated variable in every projection of AI electricity demand.

What to watch

Watch memory capacity in mainstream phones and laptops, because model size on device is gated by it. Watch whether operating systems expose a shared system model that every app can call, which would remove the duplication of each app shipping its own. And watch European privacy enforcement, which is quietly the strongest commercial argument for local inference.