Evals

What is an eval?

Evals are structured tests that measure how well an AI system performs a task against criteria set in advance. OpenAI describes them as structured tests for measuring a model's performance. An eval runs the AI on a set of real examples with known good answers, then grades the output by code, by another model or by people. Anthropic's documentation recommends writing test cases before refining prompts.

Why it matters to a revenue leader

Evals are how you know an AI tool works on your data as well as it did in the demo. Generative AI can give different answers to the same input, so one good result proves little. Before rolling out a tool that summarizes deals, flags risk or drafts customer messages, ask for results on a test set built from your own records, and rerun it when the vendor changes models. Evals also catch drift, since a tool that worked in March can degrade after an update.

A RevOps lead at a mechanical services company tests an AI tool that pulls renewal dates and prices from service agreements. On 200 past agreements with known answers, it gets 184 right, or 92%. Most misses are scanned contracts with handwritten changes, so the team routes scanned files to a person and reruns the test each time the vendor updates its model.