LLM as a judge

What is LLM as a judge?

LLM as a judge means using one AI model to grade another model's output against standards set in advance. The judge scores an answer or picks the better of two. A 2023 study led by Lianmin Zheng found that strong judge models reached over 80% agreement with human preferences, similar to the agreement between people. The same study documented biases, such as favoring longer answers or whichever answer appears first.

Why it matters to a revenue leader

LLM as a judge makes it practical to check AI quality at scale. A model can grade thousands of deal summaries or email drafts in the time it takes a manager to read a handful. The judge is still a model with blind spots, so its grades deserve trust only after someone compares them with your own people's judgment on a sample. OpenAI's evaluation guidance calls model judges cheaper and more scalable than human review, and advises checking their agreement with human labels first.

A fire and life safety company tests an AI that writes inspection summaries for customers. A judge model scores 1,000 summaries for accuracy and clarity. Two service managers grade 50 of the same summaries, and the judge agrees with them on 41, or 82%. Six of the nine disagreements involve summaries that leave out open deficiencies, so the team tightens the judge's standard on that point before relying on it.