Always test, always research, always challenge: what building AI keeps teaching me

Yesterday I spent the day building "Project X", an offline tool that reads a company's contracts and builds one register of what has actually been signed. It runs on a standard laptop, with no internet connection, because the people who need it most – banks, regulated companies or even enterprises that are reluctant to get their data out on the market (so I'd imagine all of them) – cannot send their contracts to a cloud tool.

By the end of the day the model was fast. It was also, at one point, confidently wrong. That moment is the reason for this article.

The USD 2,000,000 that did not exist

I gave the model a services agreement and asked it for the liability cap. It answered USD 2,000,000.

The contract has no liability cap. The number was real, but it sat two clauses away, in the insurance section: the supplier must hold public liability insurance of USD 2,000,000. The model found the word "liability", found a number, and put them together.

Nothing crashed. There was no warning. The output looked exactly like a correct answer.

In procurement this is the worst type of error. A field the tool leaves blank goes to a person for review. A wrong liability cap goes into a renewal negotiation, a risk report, or a board paper, and nobody questions it because it came from "the system".

I caught it for one reason: I test against contracts where I already know every answer. And I score "gave a value where the contract states none" as its own failure, separate from a miss. On the three fields that matter most (liability cap, contract value, governing law), one invented value fails the whole run.

Fix the cause, not the output

The easy fix is to tell the model "ignore insurance". The real fix was one level down. The tool now splits each contract into its clauses and gives every field only the clauses that match it, minus a short list of phrases that must never pull a clause in. For the liability cap, insurance is on that list.

Same contract, new answer: not stated. Correct.

That change did something else. My first version sent the whole contract to the model for every field, about 146,000 tokens of reading per agreement, which was far too slow on a laptop. Instead of tuning it, I questioned the design. Each field now gets only what it needs: about 1,750 tokens, and the full contract runs in 2.5 minutes on an ordinary office laptop, with the network switched off.

Faster and more accurate came from the same decision. That only happened because I stopped polishing the first approach and challenged it.

Always test, with a test that can fail

Most checks I see in AI projects are checks that pass when nothing happened. A search that returns zero results is treated as proof that a problem is absent, when the page might simply not contain the table being searched.

So every check I build now comes with a planted fault. If I say a duplicate detector works, it has to catch a duplicate I put there on purpose. If it can't fail, it proves nothing.

Always research, at the source

Earlier this year I asked an AI agent to collect energy prices across Europe. It came back with a complete dataset: every country, every field, the right format. It passed every validation rule I had.

It was wrong. It priced a country that is not in the source data at all, used a period a year out of date, and left out the only two countries the cited article actually listed.

A format check tells you the shape is right. It tells you nothing about the truth. Since then my rule is simple: where the publisher has the data, fetch it from the publisher. Only use a model where there is no other source, and then check three rows against the original by hand.

Always challenge, including the records

Last night I found that the instruction file every AI session on my build machine reads first had not been updated since April. Six months of sessions started from facts that were no longer true. Nothing broke loudly. Every answer still sounded sure of itself.

Records go wrong quietly. Two true-sounding notes disagree, or something true was never written down. The only defence is to go and look at the thing itself, not at what the notes say about it.

Bring more models, and let them disagree

On the same contract, a lighter version of the same model avoided the liability error. It is also faster. Tempting to switch on the spot.

I didn't. One contract is an anecdote. Both versions now go through five contracts with known answers before I choose.

The product works the same way. More than one model reads every field, and a value is only accepted when they agree. When they don't, it goes to the contract manager with the clause and the page, and a person decides. Agreement between models is not truth, but disagreement is a reliable signal that a human needs to look.

Refine, reiterate

None of this is new to anyone in procurement. We don't accept a supplier's claim because it is presented with confidence. We ask for the evidence, check it against the source, get a second quote, and challenge the first answer that sounds too clean.

Building with AI needs exactly the same discipline:

• Always test, with a test that can fail.

• Always research, at the source.

• Always challenge, including your own design and your own records.

• Bring more models, and let them disagree.

• Refine, and go round again.

Verify the claim, not the confidence.

#ProcurementUnion #ProcurementExcellence #ProcurementWay