Funnel changes that need testing first, and ones that do not

5 days ago
3 min read

Updated: 2 days ago

Introduction

There is a version of testing discipline that becomes an obstacle. A business with forty enquiries a month cannot run a statistically valid experiment on anything, and insisting on one before fixing a form that loses submissions means the form stays broken for another quarter.

There is also a version of decisiveness that destroys things that were working. Changing five parts of the process at once, then finding conversion has fallen, leaves you with no way to identify which change caused it. The useful distinction is not between testing and acting; it is between changes where the direction is obvious and changes where it genuinely is not.

1. Funnel changes that need testing first are the ones with an uncertain direction

The sorting rule.

If a reasonable person could argue the change either way, test it. If nobody would argue that a broken notification should stay broken, fix it and move on. Most of the changes a small business contemplates fall clearly into one camp or the other, and the ambiguous middle is smaller than it looks.

2. Just fix the outright failures

No test required.

Forms that do not deliver, calls that go unanswered, links that break, quotes that are never sent. These have no upside to preserve and testing them wastes the time they are costing you. Treat the delay itself as the cost: a quarter of testing a broken form is a quarter of lost enquiries.

3. Test anything that reduces volume deliberately

The classic uncertain case.

Publishing prices, adding qualifying questions, raising minimum job size, requiring a deposit. Each trades volume for quality and the net effect depends on your business, which means it has to be observed rather than argued. These are also the arguments that recur indefinitely in a business until somebody produces a number.

4. Test price changes with particular care

Highest stakes, hardest to reverse.

A price rise that reduces volume more than expected takes months to recover from, and reversing it publicly is difficult. Test on a segment or a period rather than across everything at once.

5. Change one thing at a time

The discipline that makes any test readable.

Two simultaneous changes produce one result and no attribution. Where you must change several, sequence them across quarters rather than bundling them.

6. Use time periods when you cannot split traffic

The small business method.

Most firms cannot run a proper split test. This quarter one way, next quarter the other, with the seasonal difference accounted for, is imperfect and considerably better than nothing.

7. Decide the measure and the duration in advance

Otherwise the result is whatever you wanted.

Which number, over what period, and what difference counts as real. Written down before you start, because tests evaluated afterwards are always successful.

8. Accept that small numbers limit what you can learn

The honest constraint.

At low volumes only large effects are detectable. This means testing small tweaks is a waste and testing substantial changes is worthwhile, which is a useful filter on what to attempt at all.

9. Keep a written record of every change and its date

The most underrated practice.

A dated log of what changed. When conversion moves three months later, this is the only way to work out what caused it, and it costs a line of text per change.

Be careful about testing things with a customer cost. Deliberately slowing responses for half your enquiries, or withholding information from a group, tests something real and does so at the expense of people who did not agree to participate.

Conclusion

Test what is genuinely uncertain and simply fix what is obviously broken.

Repair outright failures immediately without ceremony, test any change that deliberately trades volume for quality, be especially careful with price changes and confine them to a segment or a period, change one thing at a time, use consecutive periods when you cannot split traffic, fix the measure and the duration before starting, accept that low volumes only reveal large effects, and keep a dated log of every change you make.