---
title: "An Approval Rate Is Not a Cost Reduction"
pubDate: 2026-04-21T00:00:00.000Z
updatedDate: 2026-10-02T00:00:00.000Z
tags:
  - AI
  - Customer Support
  - Evaluation
image: /images/blog/approval-rate-is-not-savings/featured.png
---

## TL;DR
An approval rate is not a cost reduction. Unless someone can tell in advance *which* answers are the bad ones, a human still reads all of them, and the saving is a fraction of what the headline number suggests. Do that arithmetic, queue volume included, before the project starts.

## The number everyone reaches for

The natural way to measure an LLM support automation is an approval rate. Run the system in **shadow deployment** — it drafts a reply to every incoming ticket, subject-matter experts grade the drafts, and nothing is sent to a customer — and count how many drafts the experts would have sent as written.

It is a good number for steering development. It is a poor number for a business case, because it invites a reading it can't support: an 80% approval rate sounds like 80% of the work done.

## Why the two come apart

Unless something can flag in advance which fifth is wrong, a human still reads all of them.

What the automation reliably replaces is *composing* a reply, not *deciding* whether a reply is correct — and the reviewer still has to load the customer's situation into their head to make that decision.

So the honest accounting looks less like "80% of the tickets are handled" and more like "every ticket now costs a read instead of a write." How big a saving that is depends entirely on how much of the original work was the deciding rather than the writing, which is a question about your queue, not about the model.

## What would have to be true

For an approval rate to become a cost reduction, one of these has to hold:

- **The system knows when it is unsure.** If the model can separate the cases it will get right from the ones it won't, you can auto-send the confident slice and route the rest to a human. Then the useful metric is not the overall approval rate but the approval rate *within the auto-send slice*, and how large that slice is. Auto-sending half the queue with near-zero errors beats an 80% approval rate you can't act on.
- **Review is genuinely cheaper than authoring.** True where the reply is long, formulaic, or requires assembling data the reviewer would otherwise have to look up by hand. Less true where the hard part was the judgement call.
- **The errors are cheap.** If a wrong reply costs little to correct after the fact, you can accept some bad sends and skip the review. In regulated domains like banking, that assumption rarely survives contact with compliance.

A headline approval rate measures none of these. It is often a proxy for a proxy, too: if the graders are judging whether a draft matches an established message template rather than whether the customer's issue got resolved, the number sits one step further from the outcome anyone is paying for.

## Volume decides the rest

Even with all of that fixed, the arithmetic has a second term: how many tickets there are.

Take a low-volume queue. Even a flawless system is competing for a small slice of one person's time, against the cost of building and maintaining it — and maintenance is not optional, since the business rules the system encodes keep changing.

## Do the arithmetic first

Whether an automation like this pays for itself is an ROI calculation. The parameters worth pinning down before the project starts:

- **Volume.** Tickets per day in the queue you are actually automating, not the whole support inbox.
- **Handling time.** Minutes per ticket today, and minutes per ticket to *review* a generated draft.
- **The cost of being wrong.** What happens to a bad reply that goes out, and who carries it.
- **Reviewer availability.** [The feedback loop](/post/the-missing-compiler) runs on subject-matter expert time, both during the build and afterwards.
- **Maintenance.** Who edits the rules when the business changes them, and how often that is.

None of this is hard. It is uncomfortable, because the honest version of the calculation sometimes says the project shouldn't happen — which is exactly the value of doing it before rather than after.