Evaluation engineer
The one who writes the mark scheme before reading the paper
An evaluation engineer builds the tests that say whether a system answers well, and writes them before seeing any answers, like an examiner drawing up a mark scheme before marking.
- Floor 2The Harness
- P2Writes the application
- Emerging titleIt appeared recently. What it covers is not yet stable from one employer to the next.
What they answer for
The deliverableA set of tests and their mark scheme, plus the number that does or does not clear a release. If they leave, nobody can say whether the last change improved anything at all.
The profile
Five activities, scored 0 to 3- ResearchProducing knowledge that does not yet exist.
- BuildShipping a system that runs, deploys and breaks.
- OperateKeeping it in production: cost, incidents, drift, on-call.
- VerifyMeasuring, testing, attacking. Producing a verdict that holds.
- LeadDeciding, persuading, driving adoption, answering to others.
What the work is
Why the role exists
Ordinary software is tested: you know the expected output and you compare against it. A system that answers in prose has no expected output, it has answers that are more or less good. The trade consists of making that "more or less" measurable, and repeatable from one week to the next.
The order of operations
What separates it from software quality is the order of operations. You write the mark scheme first, on cases chosen for what they will reveal, and you look at the answers afterwards. Writing the scheme afterwards amounts to scoring what you happened to get, which always produces a good result.
The trap
The trap is that the role looks secondary until something fails in public. It is the most underrated trade on the floor, and the first one demanded after the first incident. An advert that creates it after the fact often describes a role with no authority: measuring without being able to stop a release serves nobody.
A week in the role
- Test cases written with people from the domain concerned, who alone know what a good answer is.
- Code: the chain that replays the tests on every change, and files the results.
- Arguments about the mark scheme, when two domain experts do not score alike.
- A regular conversation with product: does that number clear the release, or not.
Ways in
- From software quality, accepting that you leave behind the pass-or-fail that made it feel safe.
- From the domain itself, when you are the one who knows what a good answer is. This is the least obvious way in and the most useful.
- What the role does not require, whatever the advert says: training a model. You measure; you do not improve it yourself.
Reading an advert
2 signs- The advert talks about "automated testing" and names a software test framework.
- That is classic quality assurance. The role described never meets the mark scheme problem, which is the trade.
- Nothing says who decides when the score is bad.
- Measuring without the authority to block produces a dashboard nobody reads. Ask before you sign.