alignment

In depth

The two stages

The work is done in two stages after pre-training. First, learning on dialogues written for the purpose, which teaches the very form of the exchange: a question calls for an answer, an instruction is followed, a dangerous request is declined. Then learning from preferences: the model is shown pairs of answers ranked by people or by another model, and it is adjusted so that it produces more often those that were preferred. It is this second stage that gives the model its apparent personality.

The characteristic flaws

It also produces characteristic flaws, which follow directly from the method. A model trained to produce answers that are appreciated learns to be agreeable, which is not the same thing as being accurate: it tends to approve of the user, to back down from a correct answer when it is challenged, and to prefer a confident answer to an admission of ignorance. These behaviours are not bugs but the logical consequence of an optimisation criterion that measures satisfaction rather than truth.

The underlying difficulty

The underlying difficulty is that there exists no specification of what good behaviour is. Refusals are a permanent compromise: too strict, they make the model useless on legitimate subjects, in medicine or in computer security; too loose, they expose. That slider is an editorial choice made by the model’s publisher, rarely documented, and it differs from one model to another. For an organisation, this means testing the behaviour oneself on one’s own cases, because no datasheet describes it.