high-risk use

In depth

Qualification in two stages

The qualification is read in two steps. First a domain: the European regulation on artificial intelligence lists the fields where a decision engages a person’s life. There you find access to employment and the management of workers, education and the assessment of learners, access to credit and to essential services. To these are added several sovereign fields, from law enforcement to justice, migration and border control included, and certain uses of biometrics. To this are added the cases where an AI system serves as the safety component of a product already regulated elsewhere. Then a role: within those domains, what counts is the place the system occupies in the decision, and an arrangement that prepares, filters or ranks does not sit among the accessories once it steers what the person obtains. The text provides that a listed use may be set aside from the category where its contribution stays narrow and without real influence, but that exclusion has to be demonstrated and documented: it is not a silent self-declaration. The practical consequence is that the question is never answered from a supplier’s catalogue: the real use has to be described.

What it triggers

What the qualification triggers reads better as a set of capabilities to be demonstrated than as a list of forms to be filled in. You have to be able to show that risks have been identified and monitored over time, and not estimated once and for all at launch. You have to be able to account for the data used: where it comes from, what it covers and what it leaves out, which is the direct point of contact with the question of bias. You need technical documentation describing what the system does, what it rests on and within what limits it holds, together with logging that makes it possible to reconstruct a contested case months later. You need effective human oversight, that is to say oversight entrusted to someone who has the time, the information and the authority to interrupt: the wording explicitly targets rubber-stamp approvals, and this is the point at which most arrangements empty of their substance. Finally, the people concerned have to know that a system is involved, and to be able to contest the result when a decision is aimed at them. These requirements are distributed along the chain, whoever supplies not answering for the same things as whoever uses, and that is why compliance announced by a supplier never closes the question for the organisation that deploys.

Internal use

The main trap is believing that an internal use escapes the qualification. The intuition comes from consumer law, where the subject is the product sold, whereas here the starting point is the consequence for the person: uses that affect employees, from recruitment to assignment and appraisal, are among the most heavily regulated. The second trap is the reasoning from the “merely assisted” decision. Filing a system among the aids because a person signs at the end proves nothing: you still have to look at what happens before that signature. An arrangement that discards three quarters of the files before any human sees them does steer the decision. The third is treating the qualification as a state once acquired, when a use drifts, a system changes and a pilot becomes a deployment: the question comes back at every widening. The fourth is confusing the tool with the use, which gives the feeling of having dealt with the subject because a supplier has produced a certificate. The useful conclusion is not legal but organisational: what you must be able to produce is an honest description of the real use, of who is subject to the decision and of what the system changes in it, and then to take that description to whoever knows how to qualify it.