Measuring revenue lift from an AI tool without fooling yourself

5 days ago
3 min read

Updated: 5 days ago

Introduction

Twelve months after buying something, the renewal invoice arrives and somebody asks whether it worked. The honest answer in most businesses is that nobody knows, because revenue went up, several other things changed at the same time, and no measurement was defined at the outset.

This is not a small failure. It means the tools that are working cannot be defended when budgets tighten, and the ones that are not working get renewed indefinitely because there is no evidence against them. The fix is almost entirely front-loaded: decide what you will measure, and record it, before anything is switched on.

1. Measuring revenue lift from an AI tool starts before implementation

The baseline cannot be reconstructed.

Two or three numbers, recorded for the three months before you begin: conversion rate, average order value, response time, margin, whatever the tool claims to affect. Ten minutes of work that makes the whole question answerable later.

2. Pick one primary metric and commit to it

Not a dashboard.

The metric the tool is supposed to move, chosen in advance and written down. Multiple metrics guarantee that at least one will have improved by chance, which is how tools get renewed on the strength of an unrelated number.

3. Prefer a holdout group over a before-and-after comparison

The single most useful design choice.

Half the sales team, half the regions, half the product lines. A comparison group running the old way during the same period removes seasonality, market movement and everything else that changed. Where a holdout is possible, use it.

4. Where a holdout is impossible, use a long enough period

Time is the substitute.

For a small business a single month proves nothing. Twelve weeks minimum, ideally compared against the same period a year earlier as well as the period immediately before, so you can see seasonality rather than mistake it for effect.

5. Count the cost properly

Licence fees are the smallest part.

Setup time, data cleaning, training hours, ongoing administration, the time spent reviewing output, and whatever people stopped doing to accommodate it. Tools that look strongly positive on licence cost alone frequently look marginal on total cost.

6. Distinguish time saved from money saved

The most common overstatement.

Six hours a week saved is worth nothing unless those hours produced something else. If the person now finishes earlier, the saving is real for them and invisible in your accounts. Say which of the two you have.

7. Look for revenue that moved rather than appeared

Ask where it came from.

A tool that pulls forward sales you would have made anyway shows a strong first quarter and a weak second. Checking whether the total grew, rather than the timing changed, requires a longer window than most evaluations use.

8. Write down what result would make you stop

Before you start.

A threshold and a date. Without them, the decision at renewal is made by whoever is most invested in the tool, and sunk cost reliably wins. A pre-agreed number removes the argument.

9. Repeat the measurement at each renewal

Effects decay.

Novelty, initial enthusiasm and the easy wins all fade, and a tool that earned its cost in year one may not in year three. Rerunning the same comparison annually takes an hour if the baseline conventions were kept.

Watch for the case where the gain was a process change rather than the tool. Firms frequently redesign a workflow during implementation, and the redesign is what produced the result. That is worth knowing, because you keep the gain and stop the subscription.

Conclusion

Record the baseline before implementation, because it cannot be reconstructed afterwards.

Commit to one primary metric in advance, use a holdout group wherever it is possible and a long enough period where it is not, count setup, training and review time in the cost rather than the licence fee alone, be explicit about whether saved time became money, check whether revenue grew or merely moved earlier, write down in advance the result that would make you stop, and repeat the measurement at every renewal because the effect decays.