ModelorProduct
The model is the frozen part that predicts, and nobody uses it directly. The product is what you actually use: the same model, plus instructions, a memory, documents, tools and an interface, all of them chosen by someone.
Point by point
| Criterion | Floor 1Floor 1 · The Modela solid block on its own: the prediction machineModel | Floor 2Floor 2 · The Harnessthe block and its bolted-on plates: what gets added to itProduct |
|---|---|---|
| What it is | A file of parameters | An assembly around that file |
| Who builds it | A model publisher | Whoever builds the service |
| Rate of change | A few versions a year | May change every week |
| What is evaluated | Generic public benchmarks | Your own cases, end to end |
| Where to pin a flaw | Rarely on the model alone | Most often on the assembly |
| What is negotiated | A price per token | A service level and guarantees |
On the ground four situations
An assistant refuses to answer on a medical subject that is perfectly harmless.
Floor 2 · The Harnessthe block and its bolted-on plates: what gets added to itProductThe refusal almost always comes from an instruction or a filter placed by the service, not from the model: the same model, called directly, will often answer.
A model is unaware of an event that happened after its cut-off date.
Floor 1 · The Modela solid block on its own: the prediction machineModelHere it really is the model: what it knows by itself stops at its corpus, and no product setting fills that hole without giving it something to read.
Two applications using the same model give very different answers to the same question.
Floor 2 · The Harnessthe block and its bolted-on plates: what gets added to itProductThe useful counter-example: with an identical model, the whole gap comes from the assembly, standing instructions, documents supplied, format imposed.
A model update by the publisher degrades the behaviour of your application.
BothThe change comes from the model, but the fragility comes from the product: nothing had pinned the version, and no test set caught it before the users did.
The test that settles it
Ask whether the behaviour you observed would survive a change of supplier. What would disappear belonged to the model. What would remain, because you wrote it, belonged to the product.
The trap
“This model is bad” is almost always a judgement on a product. Before changing model, it is worth checking what the harness gives it to read: that is where most of the causes are, and it is the only part you control.
Check click your answer
Level 1 · Recognise
Are you using a model when you talk to an assistant?
Level 2 · Distinguish
Two services use the same model and give very different answers. The explanation?
Level 2 · Distinguish
What has to be evaluated before going into production?