trace

In depth

What it records

A useful trace records the complete sequence of a run: the initial request, the context actually assembled, every call to the model, every tool called with its parameters and its result, then the final output. Each turn carries an identifier tying it to the whole run, failing which you get fragments impossible to stitch back together. To this are added the measurements that make reading back usable: duration, volume read and produced, version of the instructions in force that day. The question a trace has to answer is simply stated and hard to tool up for: what was the model reading at the precise moment it took that decision.

When it becomes necessary

On a system that answers in one turn, you often do without a trace, since the input and the output are enough to understand. On an agent it is impossible, because the faulty decision was taken several turns before the visible result, from an observation that arrived along the way. Without a trace, all that remains is conjecture and an attempt at reproduction, which often fails since two runs of the same case can diverge. Nor is the trace to be confused with memory: memory is what the system gives itself to read again in order to continue, the trace is what people read back in order to understand. It is also what feeds evaluations, since every real incident can become a test case.

The traps

The first trap is the incomplete trace: you record the tool calls but not the assembled context, so that you see what was done without ever knowing why. The second is the trace becoming a risk in itself, because by construction it contains personal data, secrets and internal documents, which makes it a processing activity to declare, to protect and to purge. The third is volume, since fully tracing a talkative agent sometimes costs more than running it, hence a sampling that is better owned than endured. A trace is therefore designed with a scope and a retention period decided in advance, not accumulated by default.