You keep hearing that AI runs on "GPUs" and also on Google "TPUs." They're both chips that do AI math. Here's how they differ, using Google's own explanations.
The one-line version
From Google's explainer:
- CPU: a general-purpose chip. Handles almost anything, but slower at specialized work.
- GPU: the workhorse for accelerated compute, from graphics to AI.
- TPU: Google's own chip, designed from the ground up for AI.
What a TPU actually is
Google's Cloud TPU docs describe TPUs as custom ASICs (chips built for one job) that accelerate machine learning. They're designed for the large matrix operations at the heart of neural networks, and they have on-chip high-bandwidth memory (HBM) so they can hold bigger models.
Two practical differences from a GPU:
- You can't buy one. Google says TPUs live in its data centers. You rent them through Google Cloud.
- Your code gets compiled for it. Code for TPUs must go through Google's XLA compiler, which turns your model graph into TPU machine code. GPUs run a wider range of hand-written code.
When Google says to use which
Google's own docs give a decision guide. Paraphrased:
| Use... | When your workload is... |
|---|---|
| CPUs | quick prototyping, simple or small models, lots of custom C++ ops, I/O-limited |
| GPUs | many custom PyTorch/JAX ops, ops not available on TPU, medium-to-large models with bigger batches |
| TPUs | dominated by matrix math, no custom ops in the training loop, trains for weeks or months, large models and batches, huge embeddings (ranking, recommendations) |
Google also lists what TPUs are not suited to: programs with frequent branching, lots of element-wise operations, high-precision arithmetic, or custom ops in the main training loop.
Today's top TPU vs a GPU you know
Google's TPU7x (Ironwood) docs list, per chip:
- 192 GiB of HBM at about 7.37 TB/s
- 2,307 TFLOPs BF16 and 4,614 TFLOPs FP8 peak
- Pods of up to 9,216 chips
For scale, a consumer RTX 5090 has 32GB of GDDR7 at 1,792 GB/s. Different leagues: one's a data-center chip built to be linked by the thousand, the other's a desktop card. Why memory type matters so much is in GDDR7 vs GDDR6 vs HBM and our HBM explainer.
The docs also note Ironwood supports JAX and PyTorch (not TensorFlow), and serving with vLLM and SGLang. If you already use vLLM's OpenAI-compatible server, there's a TPU backend.
Why this matters if you're not Google
- Price competition. More chip options give cloud buyers leverage. See what it costs to rent an H100 or B300.
- Portability. If you write plain PyTorch or JAX without exotic custom ops, you keep the option to move between GPUs and TPUs.
- The AI build-out. Custom chips are part of why spending is so huge; see the AI bubble capex math.
