Every contact center has had this meeting. A supervisor tells an agent they scored a 62 on Tuesday's call. The agent asks which part. The supervisor scrolls through a form, finds a "No" next to "Confirmed callback number," and the agent says, "I did confirm it — right at the end." Nobody can settle it without pulling the recording, so the meeting is now about the score, not the behavior.

Automating QA doesn't fix this on its own. Run a language model over every transcript and you get the same argument, a thousand times a night. The verdict is only part of a QA system. The other part is the evidence, and most automated QA tools skip it.

The Verdict Is the Cheap Part

Getting an LLM to answer "Did the agent confirm the callback number?" is easy now. The research backs it up. In the most-cited study on the subject, Zheng et al.'s Judging LLM-as-a-Judge (2023), strong LLM judges matched human preferences at over 80% agreement, about the same rate at which humans agree with each other. The same paper also lists the failure modes: position bias, verbosity bias, self-enhancement bias, and limited reasoning on harder questions.

Look at what that 80% means in practice. Even when the judge is as good as a human reviewer, it disagrees with some human a meaningful share of the time. That's just how subjective evaluation works. Two seasoned QA leads won't score the same call identically either. The question isn't whether the AI will sometimes be wrong. The question is whether you can tell when, and how cheaply.

A bare "No" gives you nothing to check. A "No" with the line the model relied on gives you something you can check in about five seconds.

Evidence Changes What the Score Is

When every verdict carries a quote from the transcript, the score stops being a claim and becomes a citation. That changes three things.

Disputes get cheaper. The agent who says "I confirmed it at the end" can be checked against the quoted evidence immediately. Either the model found the wrong part of the call or the agent misremembers. Either way, you settle it in seconds without pulling the recording.

Bad criteria show themselves. If a scorecard prompt keeps producing verdicts whose quotes don't really support them, the prompt is ambiguous. "Did the agent show empathy?" tends to pull in a different random pleasantry every time. "Did the agent acknowledge the caller's stated problem before offering a solution?" pulls in the same kind of line every time. The evidence field is how you debug your rubric.

Unsupported verdicts are easy to spot. A model asked to justify itself with a quote has to point at something. If the quoted line isn't in the transcript, or doesn't say what the verdict claims, you've caught an error that a bare score would have hidden. My opinion: any serious auto-QA pipeline should treat an unverifiable quote as a failed verdict, not a low-confidence one.

One caveat: evidence is only as good as the transcript under it. A misheard digit in a callback number can make a correct confirmation look wrong. That's a reason to check, not a reason to skip the AI.

The Override Rate Is Your Calibration Metric

Here's the part that gets the least attention. Once human reviewers can override AI verdicts, every override is a labeled data point telling you where the machine and your standards disagree.

Most teams treat overrides as cleanup. They should treat them as telemetry. Break them down three ways:

  • By criterion. If one scorecard question gets overridden far more than the rest, rewrite the question. The AI isn't failing; the question is.
  • By agent. If one agent's scores get overridden much more than their peers', either their calls are unusual (a different queue mix, harder callers) or a reviewer is grading them on a different curve. Both are worth knowing.
  • By reviewer. If one supervisor overrides far more than the others, you've found a calibration gap among the humans, which manual QA could never show you because it had no baseline to compare against.

This is where QA scoring at full coverage pays off in a way the 2%-sample model never could. With census-level AI scoring as a constant baseline, human judgment is finally measured against something consistent.

How We Built It at Dial800

In Dial800 CallCenter, scorecard criteria are written as plain-language prompts, such as "Did the agent confirm the callback number?", and every transcribed call is scored automatically each night. Every verdict cites Evidence: a quote from the call itself. Human reviewers can override any AI verdict, and QA Summary reports show auto versus overridden scores per agent, so you can see where the AI and your reviewers agree and where they don't.

We made evidence mandatory rather than optional for the reason above: a score nobody can check is a score nobody trusts. And because the same platform carries the live transcript and the call's attribution, the coaching conversation can start from the actual line, on the actual call, from a known campaign, instead of from a number on a form.

The Takeaway

If your auto-QA vendor can't show you why a call failed a criterion, in the caller's and agent's own words, you haven't automated QA. You've automated the argument. Ask for the quote. Then watch your override rate, because that's where your scorecard tells you what it's getting wrong.