AI chips

What it is

An AI chip trades general-purpose flexibility for throughput on a narrow set of operations, mainly dense matrix multiplication at low numerical precision. Performance depends as much on memory bandwidth, on-chip memory and interconnect speed as on raw compute, because large models are frequently limited by how fast weights and activations can be moved. Different parts are tuned for different jobs: training accelerators favour bandwidth and cluster scaling, while inference parts often optimise for cost, latency and power per request.

Why it matters

Chip supply, cost and efficiency set the economics of every AI product, which shapes how much inference a vendor can give away free and therefore how AI answer surfaces behave. For marketers, this filters down into token pricing, rate limits and model choice for retrieval and generation pipelines, all of which affect what it costs to publish or to run visibility testing at scale. It is also a heavily searched topic, so accurate explanation of the category is a credibility marker for technology publishers.

How it works

Teams choose accelerators by benchmarking their actual workload rather than by peak specification, measuring throughput and latency at a target batch size and precision. Practitioners compare tokens per second, memory capacity for the model they want to serve, cost per million tokens on hosted endpoints, and power draw where energy or cooling is constrained. Many organisations avoid buying hardware at all and instead select cloud instance types or managed model endpoints, treating chip choice as a pricing and latency decision.

When it applies

It applies when specifying or hosting your own models, choosing cloud instances for retrieval and embedding workloads, or budgeting for large-scale generation and testing. It also applies whenever you publish content that explains AI infrastructure to a technical or executive audience.

Examples

• A data team benchmarks an open-weight model on two GPU instance types and finds memory capacity, not compute, decides how many concurrent users it can serve.
• A SaaS company moves batch embedding of its documentation to a cheaper inference-focused instance and keeps interactive chat on a low-latency one.
• A publisher writes a plain-English explainer of the difference between GPUs, TPUs and custom ASICs for a non-technical executive audience.

How it is measured

• Throughput in tokens per second per accelerator at your target latency
• Cost per million input and output tokens for the workload you actually run
• Accelerator utilisation and queue wait time during peak hours
• Energy or power draw per thousand requests where efficiency is reported