How many tokens a prompt has, and how to count them

What counts as a token

A model does not read words: it reads tokens, chunks the tokenizer learned from the text it was trained on. Very common words tend to be a single token; rare ones get split into several.

That explains behaviour that throws people at first: «of» is one token and «disproportionately» may be five, even though one has two letters and the other eighteen. What decides is not length, it is how often that sequence appeared during training.

Spaces, line breaks and punctuation count too. A prompt with a lot of decorative structure —rows of dashes, separators, deep indentation— pays for all of it without any of it reaching the instruction.

Why the four-character rule is only an estimate

The «four characters per token» figure comes from measuring English text, and the tokenizers of the big models were trained on corpora where English dominates, so their chunks are tuned for English.

Any other language does worse. Longer words, richer conjugation and accents in less frequent sequences all push the count up: the same content usually takes 15 % to 30 % more tokens in Spanish than in English.

That is why a character count works as an order of magnitude and not as a measurement. If you are deciding whether a document fits in the context window, the difference between estimating and measuring is the difference between the call working and the call getting cut.

How to actually count them

There are three ways, and which one serves depends on what you need.

• ·The model's own tokenizer. It is the only exact figure, because each model family splits text its own way: the same prompt does not give the same number on GPT as on Claude. Anthropic and OpenAI both expose a counting endpoint.
• ·A local library, such as tiktoken for OpenAI's models. It runs offline and is exact for that family, but you have to keep it installed and up to date.
• ·A character estimate, dividing by four. Good for knowing whether you are in the hundreds or the thousands; not good for deciding whether something fits.

When the exact number matters

Not always. If you write a one-off prompt in a conversation, the figure hardly matters: you will notice it when the chat starts forgetting the beginning.

It matters in two cases. The first is budget: if you call an API, every input token is billed on every call, so a bloated system prompt multiplies its waste by the volume of your product. The second is the limit: when you build a system that puts retrieved documents into the context, you need to know how much the fixed part takes to work out how much is left for the variable one.

In both, the number you need is the one from the model you are going to use, not an average.

And once counted, cut them

Counting is the diagnosis; trimming is the treatment. An average prompt carries between 30 % and 50 % of text that does not change the answer: pleasantries, filler, roundabout phrasing and repetition.

You can do that by hand, and for a fixed system prompt that is the sensible thing: go over it once and leave it written. It stops being sensible when the text changes on every call, because then there is nothing to review in advance.