bias

In depth

Where they come in

Biases come in through several doors. Through the data first, which reflects the world as it is written, with its imbalances of representation: an occupation practised mainly by one group will end up associated with it. Through filtering next, which sets aside content according to criteria that are never neutral. Through alignment finally, which installs preferences of behaviour on the basis of human judgements, themselves situated. Each of these stages is a choice, and the absence of an explicit choice is one too.

When a system sorts people

The heaviest consequence arises when a learned system sorts people: job applications, credit files, treatment priorities. A systematic deviation then becomes discrimination, with legal consequences, and it is passed on at large scale and at speed, which distinguishes it from an individual prejudice. This is why regulations classify these uses among the most tightly framed, and require documentation of the measures taken rather than a declaration of intent.

Measurement

The only solid approach is measurement. A bias is not seen by reading a few answers: it is seen by comparing the outputs on identical inputs of which only one attribute has been varied, over volumes large enough for the deviation to be significant. This protocol belongs to evals, and it must be replayed at every change of version, since a model update can shift a behaviour without warning. A limit must also be accepted: reducing a deviation on one criterion can widen another, and arbitrating between several definitions of fairness is a political decision, not a technical setting.