Evaluate and Measure: Tools, ROI, Impact
This content was developed with AI assistance and is regularly reviewed for accuracy.
Snapshot: October 2026. Model names, prices, and product features change monthly; What's New in AI is the maintained chronology. Survey figures below are as reported, with dates; several come from secondary coverage.
Almost every organization now uses AI, and few can show what it returned. McKinsey's 2026 survey found 89% of organizations regularly use AI in at least one function, while the share reporting a positive effect on earnings stayed flat at 37% (reported). The gap is rarely the model. It is the absence of a baseline, a fair comparison, and a decision to stop what is not working.
This page replaces six earlier lessons on tool evaluation, impact measurement, ROI, automation strategy, getting started, and best practices with one five-step method and one worked example carried all the way through. It ends with a straight reading of the two failure statistics leaders quote most often.
Learning focus:
- Pick a use case worth measuring and capture a baseline before anything changes
- Score tools against your own test set, weighted by what matters to you
- Build a business case with every cost category, including usage-based AI costs
- Pilot against a control, then decide to expand, optimize, or sunset on real data
The method in one view
| Step | Question it answers | Output |
|---|---|---|
| 1. Choose and baseline | Is this worth doing, and what does it cost today? | A one-page baseline with time, error, and cost figures |
| 2. Choose the shape, then the tool | Workflow, AI step, or agent? Which product? | A weighted scorecard from a test on your own data |
| 3. Build the business case | What will it cost and return? | ROI, payback, and labeled soft benefits |
| 4. Pilot against a control | Did the change cause the result? | Measured figures from a limited, parallel run |
| 5. Measure and decide | Keep, fix, or stop? | A monthly dashboard and a quarterly decision |
The steps are deliberately boring. The organizations that get value from AI are not the ones with exotic tools; Microsoft's 2026 Work Trend Index attributes 67% of the variation in AI impact to organizational factors and 32% to individual ones. Measurement discipline is one of those organizational factors.
Step 1: Choose a use case and capture a baseline
Look for work that is frequent, rule-heavy, and annoying. Four lenses find candidates quickly: a week of time tracking (anything over two hours a week with predictable steps), a pain survey (what people hate doing), system hopping (copying data between applications), and error hot spots (where mistakes are costly). Then check readiness. Is the process stable, with no major change in six months? Can someone explain the rules? Is it digital end to end? Do the people who own it want help?
Plot the survivors on value against complexity and start in the high-value, simple corner. Quick wins build the credibility you need for harder projects. And fix the process before automating it: a fast version of a bad process produces bad results faster.
Then measure the current state for one to two weeks before you touch anything. Without a baseline you cannot prove value later, and every after-the-fact estimate will be challenged. Record hours spent, volume, error rate and the cost of each error, cycle time, and any money lost to delay.
The worked example, part 1
A mid-size company's accounts payable team processes 800 invoices a month. Three clerks spend about 15 hours a week each on data entry and matching invoices to purchase orders.
| Baseline item | Calculation | Annual cost |
|---|---|---|
| Staff time | 3 clerks × 15 h × 52 weeks × $38 loaded rate | $88,920 |
| Error correction | 4% of 800 = 32 errors a month × 12 × 0.75 h × $38 | $10,944 |
| Missed early-payment discounts | $2.4M payables × 6% paid late × 2% discount | $2,880 |
| Total | $102,744 |
"Loaded rate" means salary plus benefits and overhead, typically 1.3 to 1.5 times base pay.
Step 2: Choose the shape, then the tool
Before comparing products, decide what kind of automation the task needs. A deterministic workflow follows fixed steps; a workflow with an AI step hands one bounded judgment (classify, extract, draft) to a model; an agent decides its own steps toward a goal. When work must be approved, audited, or repeated identically, start with a workflow and add an AI step only where the input is messy. Workflows vs Agents sets out the full decision rule.
Then evaluate tools on five criteria, weighted for your situation rather than a generic ranking:
- Accuracy on your data. Test every candidate on the same 30 to 50 real examples, including the ugly ones. Demos show ideal conditions; consistency matters more than an occasional brilliant result.
- Integration and connectors. Does it connect to the systems where the work lives, with scopes you can limit?
- Security and data terms. Retention, training use, certifications, audit logs, and a signed data processing agreement.
- Total cost. Licences plus usage, implementation, training, and maintenance (see Step 3).
- Control and exit. Can you see and approve what it does? Can you export your data and leave?
Score each tool from 1 to 5 per criterion, multiply by the weight, and add.
The worked example, part 2
The team tested three options on the same 50 invoices from last quarter.
| Criterion (weight) | A: workflow tool with an AI extraction step | B: AP platform with built-in AI | C: general-purpose agent in an assistant |
|---|---|---|---|
| Accuracy on test set (30%) | 4 | 4 | 3 |
| ERP integration (20%) | 4 | 5 | 2 |
| Security and data terms (20%) | 4 | 5 | 3 |
| Total cost (15%) | 4 | 3 | 4 |
| Approvals and audit trail (15%) | 5 | 4 | 2 |
| Weighted score | 4.15 | 4.25 | 2.80 |
Option C was the most flexible and the weakest fit: it read invoices well but took a different path on similar inputs, and its log was harder to audit. B edged A. When scores are that close, break the tie on lock-in and exit cost, then confirm in the pilot. The team chose B, with a rule that any invoice the model flags as uncertain goes to a clerk for approval.
Step 3: Build the business case
The core formulas are simple; the rigor is in capturing every cost and labeling every benefit honestly.
ROI (%) = (annual benefits - annual costs) / annual costs × 100
Payback (months) = one-time costs / monthly net benefit
For multi-year projects above roughly $50,000, add net present value (NPV), which discounts future benefits at your organization's standard rate, typically 8% to 12%.
Count every cost
Five categories cover most projects. Software and usage: licences plus usage-based charges, which in 2026 often means tokens or agent activity, and which grow with volume. For scale, Anthropic's documentation reports enterprise Claude Code use averaging about $13 per developer per active day (October 2026), and a Ramp report says average token spend among its customers rose 13-fold from January 2025 to May 2026 (reported). Implementation: internal hours at loaded rates, plus any consultants and integration work. Training: session time, documentation, and the productivity dip while people learn. Maintenance: monitoring, fixes, prompt and connector updates when systems change; budget 10% to 20% of implementation cost a year. Opportunity cost: what the builders did not build instead.
Separate hard and soft benefits
Hard benefits are measured: labor hours saved, errors avoided, money recovered from faster processing. Soft benefits are real but estimated: capacity to absorb growth without hiring, lower turnover from removing tedious work, reduced compliance exposure. Report both, label soft benefits clearly, and make the decision on the hard ones. Project conservatively (70% to 85% improvement, not 100%), and let results beat the projection.
The worked example, part 3
| Item | Projection |
|---|---|
| One-time costs | Implementation 120 h × $55 = $6,600; ERP integration 40 h × $75 = $3,000; training 8 h × 5 people × $38 = $1,520; documentation 12 h × $55 = $660. Total $11,780 |
| Running costs per year | Platform and usage $4,800; maintenance $1,440. Total $6,240 |
| Benefits per year | 85% of invoices processed straight through frees 38 h a week: 38 × 52 × $38 = $75,088. Errors down 90%: $9,850. Discounts recovered (80%): $2,304. Total $87,242 |
| Year-one ROI | ($87,242 − $11,780 − $6,240) ÷ $18,020 = 384% |
| Year-two ROI | ($87,242 − $6,240) ÷ $6,240 = 1,298% |
| Payback | $11,780 ÷ ($81,002 ÷ 12) = about 1.7 months |
Soft benefits, listed but not counted: the team can absorb a projected 30% volume increase next year without a fourth hire.
A one-page business case for an executive needs only five parts: the problem in numbers (from the baseline), the proposed change, the financial impact, the risks and how they are mitigated, and the specific approval requested. Tailor the emphasis: finance wants payback and cost certainty, operations wants capacity and error rates, IT wants integration and security, HR wants the effect on roles.
Step 4: Pilot against a control
A pilot exists to test the business case, not to celebrate it. Agree the success criteria with every stakeholder before it starts, in writing, so nobody relitigates them afterward. Limit scope to one team, one process, or one slice of volume. Run the new and old processes in parallel at first, or compare against a similar team that has not changed, so you can attribute results to the change rather than to a quiet month.
Collect two kinds of evidence: the operational numbers from your baseline, and what users say about where the tool helps and where it creates new work. Spend real effort on the people side; a common rule of thumb is 30% to 50% of project effort on communication, training, and support, because a tool that works but is not adopted is a failed project. Drive AI Adoption on Your Team covers that half.
For agents and AI steps, add one more check: verify the output, not the status. A scheduled task can report success on work that failed, and a model that grades its own results will grade generously. Sample outputs by hand. Teams building agent evaluation suites should read Evals for Agents.
The worked example, part 4
After 90 days the measured figures came in below projection. Straight-through processing reached 75%, not 85%, because handwritten and multi-currency invoices went to review more often than expected, so the team freed 32 hours a week, not 38. Errors fell 80%, not 90%. Usage ran higher than quoted, putting running costs at $8,040 a year.
| Measured | Calculation | Result |
|---|---|---|
| Benefits per year | $63,232 labor + $8,755 errors + $2,304 discounts | $74,291 (85% of projection) |
| Year-one ROI | ($74,291 − $11,780 − $8,040) ÷ $19,820 | 275% |
| Payback | $11,780 ÷ ($66,251 ÷ 12) | about 2.1 months |
Still a clear win, and now a credible one, because it was measured against a baseline and the shortfall is explained.
Step 5: Measure after launch and decide
Calculating ROI once is useful; tracking it is what keeps a portfolio healthy. Keep a simple dashboard per automation, a shared spreadsheet is fine, with execution volume, success rate without manual intervention, hours saved against baseline, cost savings, usage spend, maintenance hours, and cumulative ROI.
Review on three cadences. Monthly, spend 30 minutes per automation comparing actuals to projections, and flag anything delivering under 80% of projected benefit. Quarterly, recalculate ROI on real data and make a decision. Annually, review the whole portfolio, update your internal benchmarks from your own results, and set next year's priorities.
The quarterly decision has three outcomes. Expand what consistently beats projections and has adjacent use cases. Optimize what is positive but under projection; small fixes usually close the gap. Sunset what is still negative after its payback period, whose maintenance is climbing, or whose underlying process has changed. Sunsetting is a success of the method, not a failure of the team.
The worked example, part 5
At 85% of projection the invoice automation sits above the 80% flag, so the decision was "optimize": the team added a template for the two invoice formats that most often went to review. The next quarter it reached 81% straight-through, and the team proposed expanding the same pattern to expense reports.
Reading the failure statistics honestly
Two numbers dominate leadership conversations about AI value. Both are worth knowing, and both are routinely misquoted.
Gartner: over 40% of agentic AI projects canceled by the end of 2027. This is a forecast published June 25, 2025, not a measurement of past failures. Gartner attributes the expected cancellations to escalating costs, unclear business value, or inadequate risk controls, and in the same release described widespread "agent washing", judging only about 130 of thousands of vendors marketing agents to be real. It was still the operative forecast in October 2026; no updated figure was found. Use it as a caution about agent projects specifically, and note that its three causes map onto Steps 2, 3, and 5 of this method.
MIT NANDA: "95% of generative AI pilots fail." The August 2025 report "The GenAI Divide" found about 5% of pilots delivered rapid revenue acceleration. Its basis was reported as 150 leader interviews, a 350-person employee survey, and 300 public deployments, and one critic says the organizational sample was 52. The definition does most of the work: any pilot that did not reach full production with measurable revenue impact counted as a failure, which sweeps in learning exercises and vendor evaluations that were never meant to ship. One analyst estimates a real failure rate of 25% to 30% for deployments intended for production, and the report itself found a 67% success rate for projects done with external partners. Paul Roetzer called the figure "not a viable, statistically valid thing."
Two further data points complete the picture. Menlo Ventures' December 2025 enterprise survey found only 16% of enterprise deployments qualified as true agents, systems that plan, act, and adapt; the rest were copilots and workflows with AI steps. And McKinsey's 2026 survey found 40% of organizations with over $1 billion in revenue scaling agents, against a flat 22% at smaller ones (reported).
The honest reading: many AI projects do not pay off, mostly for organizational reasons, and agent projects carry extra cost and risk. That is an argument for baselines, fair tests, conservative cases, and a willingness to stop, not an argument against trying. When someone quotes "95%" in a meeting, ask what counted as failure.
Common mistakes
- Counting only labor. In the example, error reduction and recovered discounts added about $12,000 a year on top of labor savings.
- Leaving out internal hours and usage growth. A case that lists only the licence is a case that will miss.
- Skipping the baseline. Without it, every result is an opinion.
- Evaluating on demos. Test on your own data, edge cases included, with identical inputs for every tool.
- Never sunsetting. An automation nobody reviews becomes a cost nobody sees.
Hands-on exercise
Apply the method to one real task this week.
- Pick one candidate from your team using the four lenses and the readiness questions.
- Record a one-week baseline: hours, volume, errors and their cost.
- Choose the shape with the workflow-versus-agent rule, and list two or three tools.
- Build a weighted scorecard and test each tool on 30 real examples.
- Write a one-page business case with every cost category and labeled soft benefits.
Then define the pilot's success criteria and the review date before anyone starts building.
Key takeaways
- Use one method end to end: baseline, shape and scorecard, business case, controlled pilot, ongoing review
- Test tools on your own data with weighted criteria; close scores break on lock-in and exit cost
- Count every cost, including usage-based AI spend and maintenance, and label soft benefits separately
- Measure against a control and verify outputs, not status lights; expand, optimize, or sunset each quarter
- Gartner's 40% is a forecast about agent projects and NANDA's 95% depends on a broad definition of failure; both argue for discipline, not avoidance
Next steps
Continue to: Enterprise AI Reference
- Deciding between a workflow and an agent first? Read Workflows vs Agents
- Want the agent-specific version of this argument? Read When a Workflow Beats a Loop
- Need the people side of the pilot? Go to Drive AI Adoption on Your Team