How to write a system prompt that does not bloat

What it actually does

The system prompt sets the default behaviour: who the model is, what it should do, what format it answers in and what it must never do. It sits apart from the user message because it has a different lifespan: the message changes on every call, the system prompt does not.

That asymmetry decides everything else. A word in the user message is paid for once. A word in the system prompt is paid for on every call your product makes for as long as it exists, multiplied by all your users.

What belongs in it and what is spare

What actually changes the answer is not much: the role, the output format, the hard constraints, and examples when the format is hard to describe in words.

• ·The role, in one line. «You are a financial analyst» does it; «I want you to act as if you were an experienced financial analyst with many years of experience» says the same with four times the tokens.
• ·The output format, specifically. «Answer in JSON with the keys name, amount and date» works; «answer in a structured way» says nothing you can act on.
• ·The hard constraints, as imperatives. «Do not invent data that is not in the document» is an instruction; «it is very important that you are very careful not to invent» is the same with filler.
• ·Examples, only if the format cannot be described. One well-chosen example saves paragraphs of explanation; three nearly identical ones are three times the same cost.

Why it always ends up growing

The pattern is always the same. The model fails on one case, someone adds a sentence to cover it, it works, and the sentence stays. Six months later the prompt is forty lines long and nobody remembers which of them are still needed.

The problem is not only cost. A long prompt dilutes the important instructions among the incidental ones, and there is a point past which adding rules makes obedience worse rather than better.

The discipline that works is simple and almost nobody applies it: when you add a sentence because of a specific failure, note next to it what that failure was. Three months later that is the only thing that lets you know whether it can go.

How to trim it without breaking it

Trimming a system prompt has a hard limit, and crossing it does not save money: it breaks the behaviour. The safe order runs from what changes nothing to what does.

• ·Pleasantries and wish wrappers. «Please», «I would like you to», «if it is not too much trouble». The model does not obey better for politeness.
• ·Filler and reinforcement. «It is very important that», «always remember», «under no circumstances». If the instruction is already an imperative, the reinforcement adds no obedience.
• ·Roundabout phrasing. «In order to» for «to», «in the event that» for «if». It changes the length, not the meaning.
• ·Repetition. The same rule stated twice in different words is the most common residue of months of patching, and the one that takes up most room.

Measure before and after

Trimming by eye is how prompts get broken. Before touching anything it helps to have two figures: how many tokens it takes today and how many calls a day send it. The second turns the saving per call into a monthly figure, which is the one that decides whether it is worth it.

And afterwards, check the result: a word-by-word diff shows exactly what disappeared, which is the only thing that separates a trim from an amputation.