Deciding a page change without a test you cannot afford to run

4 days ago
3 min read

Introduction

Most small businesses cannot test anything meaningfully. A site with forty enquiries a month cannot detect a small improvement in any reasonable period, and waiting for statistical certainty means never changing anything. The usual response is either to change everything on instinct or to change nothing on principle, and both waste opportunities.

There is a middle position that is neither reckless nor paralysed. Some changes are safe to make without evidence because the direction is not in doubt. Others carry real risk and deserve caution. Sorting them, and knowing what to do in each case, is more useful than any amount of discussion about sample sizes.

1. Deciding a page change without a test starts by sorting the change

The triage.

Is this fixing something broken, applying a well-established principle, or expressing a preference? The three call for completely different levels of caution. Most proposed changes fall clearly into one of the three, and the sorting takes a minute.

2. Just fix what is broken

The first category.

A form that fails, a link that breaks, text unreadable on a phone, a missing price. These have no upside to preserve, and testing them wastes the weeks they are costing you. Treat the delay itself as the cost of deliberating.

3. Apply the settled principles without ceremony

The second category.

Fewer required fields, costs shown early, guest checkout, a stated response time, preserved data on error. The direction of these is not seriously in dispute, and a small business should simply make them. Spending months proving them locally is effort better used elsewhere.

4. Be cautious with anything that trades volume for quality

The third category.

Publishing prices, adding qualifying questions, requiring a deposit. The net effect genuinely depends on your situation, and these deserve a period of observation rather than an assumption. They are also the changes that generate the longest internal arguments, which is a reasonable signal that the answer is not obvious.

5. Use before and after periods honestly

The available method.

A quarter before, a quarter after, compared against the same quarters last year to allow for seasonality. Crude, slow, and the best a low-volume business has. State the limitation when you report the result.

6. Change one thing at a time

The discipline that makes any observation readable.

Several simultaneous changes produce one result and no attribution. Sequencing across quarters is slower and it is the only way to learn anything.

7. Keep a dated log of every change

The practice that costs nothing.

One line per change with a date. When a number moves three months later, this is the only way to work out what caused it, and almost nobody keeps it.

8. Prefer reversible changes

The risk management.

Where you cannot be confident, choose changes you can undo quickly. This lowers the cost of being wrong, which is the real constraint when you cannot measure.

9. Accept the limits of what you can know

The honest position.

At low volumes you will never be certain about a small change. Recognising that prevents both false confidence in a result and the paralysis of waiting for evidence that will not arrive.

Be careful about copying results from published case studies. A change that produced a large improvement for a different business, with different traffic and a different offer, carries very little information about yours, and treating it as evidence is a common error.

Conclusion

Sort the change first, then choose how much caution it deserves.

Fix outright faults immediately without testing, apply the settled principles as a matter of course, reserve caution for changes that deliberately trade volume for quality, use before-and-after quarters compared year-on-year where you must observe, change one thing at a time, keep a dated log of every change, prefer reversible changes when uncertain, and accept honestly that small effects will remain unmeasurable at your volume.