What is a tokenizer, and why it changes your AI bill

Every AI API bill is written in tokens. You pay a price per million input tokens and another per million output tokens, and your context window is measured in tokens too. But a token isn't a word, a character or a syllable. It's whatever the model's tokenizer decides it is, and that decision changes what you pay.

What a tokenizer does

A model can't read text directly. Before your prompt reaches it, a tokenizer chops the text into pieces from a fixed vocabulary and turns each piece into a number. The model works on those numbers and writes its answer the same way, one token at a time.

Most current tokenizers use some form of byte-pair encoding (BPE). OpenAI describes its open-source tiktoken library as "a fast BPE tokeniser for use with OpenAI's models". The idea: frequent strings of characters get their own token, rare ones are built from smaller pieces.

You can see it in action. With tiktoken's o200k_base encoding, which the library maps to GPT-4o, we got these splits on Oct 29, 2026:

• "Toronto" → one token
• "tokenizer" → "token" + "izer"
• "unbelievably" → "un" + "bel" + "ievably"

A leading space usually belongs to the token too, so " hello" and "hello" are different tokens.

Same text, different counts

Because the vocabulary decides the split, the token count depends on the tokenizer and on the kind of text. Here's what we measured with tiktoken 0.14.0 on Oct 29, 2026, using two of OpenAI's encodings:

Three things stand out. English prose came in at about one token per word. The French translation had fewer words but more tokens. And code and JSON, full of brackets, quotes and underscores, used two to four times as many tokens as words. These are tiny samples, so treat them as an illustration, not a benchmark.

Google's rule of thumb for Gemini is similar: about 4 characters per token, and 100 tokens for roughly 60 to 80 English words, according to its token docs.

Why it changes your bill

Your cost is price per token × number of tokens. The list price is only half of that.

The clearest recent example is Anthropic's own. In its Opus 4.7 announcement (Apr 16, 2026), Anthropic said the model "uses an updated tokenizer" and that "the same input can map to more tokens—roughly 1.0–1.35× depending on the content type". Its token counting docs now say Claude 4.7 and later models produce "approximately 30 percent more tokens than on earlier models" for the same text, and that the Fable and Mythos models share that tokenizer. The docs add that usage and billing on those models "reflect this tokenizer's counts".

Independent measurements landed near or above the top of that range for some content. Simon Willison reported on Apr 20, 2026 that the Opus 4.7 system prompt counted 1.46 times as many tokens on the new tokenizer as on Opus 4.6, while a text-heavy PDF came in at 1.08 times.

Here's what that means in plain arithmetic, with a made-up price. Suppose two models both list $5 per million input tokens, and your app sends text that counts as 10 million tokens a month on the old tokenizer. If the new tokenizer turns it into 30% more tokens, you're billed for 13 million, or $65 instead of $50, at the same sticker price.

Anthropic itself notes the trade-off can run the other way at the task level: it said the net effect on token usage in its own coding evaluation was favourable, and it recommends "measuring the difference on real traffic". That's the right instinct for any model switch.

The same logic applies to your context window. A context window of any given size holds less of your text if each sentence costs more tokens. Our context windows explainer covers what fills that space.

How to count before you pay

Count with the tokenizer of the model you plan to use, not a rule of thumb.

• OpenAI models: install tiktoken and call tiktoken.encoding_for_model("gpt-4o"), then len(enc.encode(text)). It runs locally, so it costs nothing.
• Claude: Anthropic's POST /v1/messages/count_tokens endpoint takes the same input as a normal message, including system prompt, tools, images and PDFs. Anthropic's docs say it's free, rate-limited by usage tier, and returns an estimate that "might differ by a small amount" from actual usage. To compare two models, send the same request twice with different model values.
• Gemini: call count_tokens before sending, or read the usage fields on the response, which split input, output, thinking, cached and tool tokens.

Non-text input counts too. Google says a Gemini image of 384 pixels or less in both dimensions is 258 tokens, larger images are tiled at 258 tokens each, and audio is 32 tokens per second.

Ways to spend fewer tokens

• Cache what repeats. Long system prompts and documents you send on every call are the best candidates. See prompt caching, explained.
• Trim structured data. Pretty-printed JSON spends tokens on whitespace and repeated keys. Send only the fields the model needs.
• Recount after every model change. A newer model can be cheaper per token and still cost more per task.
• Compare per task, not per token. Our LLM API pricing comparison lists list prices, and a tokenizer count on your own prompts turns them into real costs.

A tokenizer is an unglamorous piece of plumbing. But when two models list the same price, it's often the reason your bills don't match.