compounding reliability

In depth

The arithmetic

The arithmetic is brutal and worth doing once. A step that is ninety-five per cent reliable gives, over five steps, about seventy-seven per cent success end to end; over ten steps, about sixty; over twenty, thirty-six. That is what explains the gap between a convincing demonstration, where you watch one run that worked, and a move into production, where you are subject to the whole distribution. A demonstration never proves a rate, it proves that a success is possible.

Three levers

Three levers act, and only the third really changes the order of magnitude. Reducing the number of steps is the most effective, which argues for cutting a long agent into several shorter tasks with points of recovery. Raising the reliability of a single step helps, but pays little beyond a certain threshold, since you have to go from ninety-five to ninety-nine per cent to double the final success. Adding a check after each step is the decisive lever: an error that is detected and retried does not propagate, which breaks the multiplication.

Reversibility

Finally, steps have to be told apart by their reversibility, since they do not all carry the same cost of failure. Reading a file, looking up a piece of information, producing a draft: an error is recovered there without damage. Sending an email, approving a payment, modifying a database: the error there is final. The reliability calculation must therefore be done separately on the irreversible actions, and it is at exactly those points that a human approval is placed, never uniformly along the whole path.