parameter (weight)

In depth

The figure announced

Counting parameters amounts to measuring the size of the file, not the competence. That size mainly determines hardware constraints: the memory needed to load the model, the cost of each answer, the possibility of running it on an ordinary server or on a phone. Two models of the same size can perform very differently according to what they have read and how they were trained. Announcing a size without announcing measured results therefore says almost nothing.

The decisive distinction

The distinction from training data has immediate legal and practical consequences. Publishing parameters is not publishing a corpus: this is what makes it possible for a model to be released freely while the texts used to train it are not, and it is also the reason why a document cannot be withdrawn from a model as it would be withdrawn from a database. A piece of data is not in the model, it contributed to forming it, which is a relation far harder to undo.

Two technical nuances

Two technical nuances are worth knowing. The first: not all parameters are always called upon. Some architectures activate only a fraction of the network on each request, so that a model announced as very large can cost as much as one far smaller. The second: parameters are stored at a chosen numerical precision, and reducing it, which is called quantisation, divides the memory required at the price of a loss of quality that is often modest. The same model can therefore exist in several formats, of very different sizes.