prompt caching

In depth

The mechanism

The mechanism follows from the architecture. To produce an answer, the model first computes an internal representation of the whole input, and that computation depends only on what precedes each position. If two requests begin with the same thousand tokens, this work is identical: it can be kept and taken up again from there. The practical consequence is a reduction in price and in time to first answer, often very large for systems that repeat a long instruction at every call.

The rule of composition

This imposes a rule of composition that runs against habit: put the stable in front and the variable behind. Standing instructions, tool definitions, reference documents, examples: all of that at the head, identical to the character. The user’s question, the timestamp, their identifier: at the end. A single variation at the head, a date inserted into the first line for instance, is enough to invalidate the whole cache and bring the cost back to its initial level, without any error showing.

Two effects to watch

Two effects deserve watching. The first is temporal: these caches expire within a few minutes, so that the gain depends on the rhythm of the requests and a lightly used service benefits little from it. The second touches confidentiality: the cache belongs to one account and is not shared between customers, but it implies that a portion of your inputs stays stored for a few minutes at the provider’s, which is among the questions to ask during a data processing analysis.