You measure the ROI of an AI agent on the cost of the entire process, before and after, not on how fast the agent is. That includes the time someone spends reviewing its output, the cost of fixing what slipped through, and the monthly bill for the model and infrastructure. If your company didn't measure the process before turning the agent on, there's no ROI to calculate on day 90. There's only an impression.
The gap between using AI and making money from it is wide. In McKinsey's The state of AI in 2025, only 39% of respondents report any enterprise-level EBIT impact from AI, and most of them say it accounts for less than 5% of EBIT. It's not that agents don't work. Often they do, and nobody can show what they're worth.
Speed isn't enough: the Klarna case
In February 2024, Klarna announced that its AI assistant had handled 2.3 million conversations in its first month, two-thirds of its customer service chats, doing work equivalent to 700 full-time agents. Average resolution time dropped from 11 minutes to under 2, and the company estimated a $40 million profit improvement for 2024. The release also said customer satisfaction was on par with human agents and repeat inquiries had dropped 25%.
A little over a year later, CEO Sebastian Siemiatkowski told Bloomberg the company would start hiring human agents again, because cost had weighed too heavily in the evaluation and the result was "lower quality" (as reported by Fortune, May 2025).
The lesson isn't that the agent failed. It's that quality measured in month one doesn't guarantee quality a year later, and a quality metric only protects operations if it carries the same weight as cost and speed when decisions are made. That's what a good 90-day dashboard needs from the start, and keeps needing afterward.
Before day 1: the baseline
Two to four weeks before you switch the agent on, measure the process as it runs today. You'll need:
- Volume: how many cases per month (orders, tickets, invoices, contracts) go through the process.
- End-to-end time: from the moment a case arrives until it's resolved, not just the step the agent will take over.
- Labor hours per case: all the people time, across every hand the case passes through.
- Fully loaded hourly cost: salary plus taxes and benefits, not base salary.
- Error and rework rate: how many cases come back, get fixed later or trigger a complaint.
If these numbers don't exist, producing them is the project's first deliverable. A timed sample of 200 cases is far better than nothing.
The five dashboard metrics
1. Total cost per resolved case. Add up people, model, infrastructure and the time of whoever operates the agent, then divide by resolved cases. This is the number that goes to the board.
2. End-to-end cycle time. An agent that answers in 3 seconds but leaves the case sitting in a review queue for 2 days hasn't shortened anything.
3. Escalation rate. The share of cases the agent hands to a human. It should fall over the 90 days. If it stays flat, the agent isn't getting the right context, or the process has more exceptions than anyone thought.
4. Escaped error rate. Errors found after a case was marked resolved: an invoice booked with the wrong amount, a reply that led to a complaint. It's the metric that decides whether you can expand the agent's autonomy.
5. Adoption. How many people on the team use the agent and trust its output. If the team redoes the agent's work "just in case," the gain exists in the spreadsheet and not in operations.
The 90 days in three phases
Days 1 to 30, assisted mode. The agent suggests and a human approves everything. The escalation rate is high on purpose. The goal is to measure accuracy safely and build a library of cases the agent got wrong.
Days 31 to 60, partial autonomy. Case types where the agent was consistently right are resolved without approval, with sample-based review (10% is a common starting point). Everything else stays assisted.
Days 61 to 90, steady state. The process runs the way it will at scale. This month's numbers go into the ROI calculation. Earlier months carry a learning cost that won't repeat.
On day 90, there are three possible decisions: scale to more volume or other processes, adjust and measure for another 30 days, or shut it down. Shutting down with data is a legitimate outcome, and a cheap one compared to running an agent nobody can prove pays for itself.
Hypothetical example: supplier invoice triage
The numbers below are a hypothetical example built to show the math. They are not client results.
A company receives 6,000 supplier invoices a month. Today each one takes 8 minutes of manual work (match to the purchase order, classify, book). That's 800 hours a month. At a fully loaded cost of $40 an hour, the process costs $32,000 a month.
| Metric | Before the agent | With the agent (30% escalation) | With the agent (50% escalation) |
|---|---|---|---|
| Volume | 6,000 invoices a month | 6,000 invoices a month | 6,000 invoices a month |
| Hours of work | 800 hours | 229 hours | not itemized |
| Cost per hour | $40 | $40 (analyst) | $40 (analyst) |
| Monthly cost | $32,000 | $13,460 | about $17,700 |
| Monthly savings | n/a | $18,540 | about $14,300 |
| Payback | n/a | about 5.4 months | 7 months |
With the agent, in month 3 (steady state):
- 70% of invoices (4,200) are resolved without intervention. An analyst reviews 10% of them at 3 minutes each: 21 hours.
- 30% (1,800) are escalated. The analyst resolves each in 6 minutes because the agent has pre-filled most of it: 180 hours.
- 2% of the autonomous invoices have an escaped error that's fixed later, at 20 minutes each: 28 hours.
- Model and infrastructure: $2,700 a month. Monitoring and tuning: 20 engineering hours at $80, or $1,600.
Total: 229 analyst hours ($9,160) plus $4,300 in operating costs, or $13,460 a month. Savings come to $18,540 a month. If building and deploying the agent cost $100,000, payback is about 5.4 months and 12-month ROI is roughly 122%.
Now change a single assumption: the escalation rate settles at 50% instead of 30%. Monthly cost rises to about $17,700, savings drop to about $14,300, and payback stretches to 7 months. That's why the escalation rate deserves the most attention in the first 60 days. For simplicity, the example ignores errors that already existed in the manual process.
There's one detail your CFO will ask about: 571 freed-up hours a month only become savings if they're reallocated or not rehired. If the team stays the same size doing the same work, the gain is capacity, not cash. Both have value, but they belong on separate lines in the report.
Common mistakes
- Measuring tasks processed instead of cases resolved. An agent can process 10,000 items and resolve 4,000.
- Leaving out the human cost of review. After go-live it's the largest cost line, and it's almost always missing from the spreadsheet.
- Comparing the agent's month 1 to a mature process. The first month always looks bad.
- Having no quality metric with the same weight as cost and speed.
- Forgetting the variable cost of the model. It grows with volume, and a pilot with 500 cases a month won't show you the bill at 50,000.
Day-0 checklist
- The process baseline has been measured and has an owner.
- The day-90 success criterion is written down with a number (for example: cost per case down 30% with no increase in escaped errors).
- One person has the authority to decide to scale, adjust or shut down.
- The dashboard with the five metrics updates at least weekly.
- Model and infrastructure cost is tracked per case, not just on the monthly invoice.
- The team knows how to report an agent error, and that report becomes data.
Next step
If you're about to put an agent into production, write down the day-90 success criterion and the baseline first. Without those two numbers, any result will be argued as opinion.
That's how we run the PoC Factory: the success criterion is agreed before day one, and the PoC ends with a Decision Package recommending whether to scale, adjust or discard. To see how we measure each stage, read our methodology.