ArticlesProjectsRuminationsCredentialsAbout
← Ruminations
2026-09-02

Who checks the checker?

This is how a lot of AI gets tested today. One model gives an answer.

This is how a lot of AI gets tested today. One model gives an answer. Another model scores it. If the score goes up, we ship. A person looks at a sample sometimes.

I do it too. People are slow and expensive and they disagree with each other.

But the judge learnt from the same internet as the thing it is judging. Where one has a blind spot, the other probably has it too. When both are wrong the same way, the score goes up.

At the startup we now keep one check that only a human does, every week. It is slow, and it has caught things the models agreed on.