What a model API costs, and where the bill comes from

How it is billed, in two line items

Every provider charges per token and separates two prices: input and output. Input is everything you send —system prompt, history, documents, user message— and output is what the model answers.

Output always costs considerably more per token than input, between three and five times depending on the model. That makes people assume the spend is in the answers, and in a real product it almost never is: input repeats on every call and output is not always long.

The maths is direct: input tokens times their price, plus output tokens times theirs, all divided by a million, which is the unit rates are published in.

The line item you do not see

The system prompt is fixed, it was written once, and nobody has looked at it since. But it travels on every call, so its cost is not its size: it is its size multiplied by the volume of your product.

A thousand-token system prompt in a product making ten thousand calls a day is ten million input tokens a day that have nothing to do with what the user does. Trimming it by 30 % does not save 30 % of one call: it saves three million tokens a day, every day.

Conversation history works the same way and is worse, because it grows. In a chat, every turn resends everything before it: message number twenty pays for the previous nineteen all over again.

How to estimate the bill before it arrives

Three numbers settle it, and all three can be had in an afternoon.

• ·How many tokens your fixed part takes: system prompt plus any instruction that always travels. Measured once with the tokenizer of the model you use.
• ·How many tokens an average interaction takes: what the user writes plus whatever you retrieve from documents, if you do that.
• ·How many calls a day you expect. It is the number that turns the other two into money, and the one most often underestimated.

The three ways to make it cheaper, by effort

In this order, because the first is free and the last changes the product.

• ·Trim what repeats. It is the only lever that acts on every call without touching the quality of the answer. An average prompt carries between 30 % and 50 % of text that does not change what the model does.
• ·Cache the prefix. The big providers charge far less for the part of the prompt they have already seen, if it repeats literally at the start. It requires ordering the prompt so the fixed part goes first, but it changes nothing about what it does.
• ·Change model. A smaller model costs a fraction and is enough for many tasks. It is the most effective and the only one that requires proving again that quality holds.

And if you do not pay per token, it still costs you

On a free tier there is no bill, but a token is still budget: each one takes up context window and brings forward the moment the conversation gets cut or the model starts forgetting the beginning.

The same trim that saves money for someone paying gives someone not paying more conversation before the limit. It is the same operation in two different currencies.