Testing with low traffic means giving up on statistics

Aug 29
3 min read

Updated: 4 days ago

Introduction

A business reads about split testing, installs a tool, and runs a test on its enquiry form. After six weeks one version has eleven enquiries and the other has nine, and the tool reports no conclusion.

This is the entirely normal outcome for a site with a few hundred visitors a month, and it was predictable before the test started. Statistical testing needs volume that most small businesses do not have, and running underpowered tests for months is worse than not testing, because it consumes the time that other methods would have used productively.

1. Testing with low traffic cannot produce statistical confidence

Accept the constraint first.

Detecting a modest improvement requires hundreds of conversions per variation. A site producing twenty enquiries a month will not reach that in any useful timeframe, and no tool changes the arithmetic.

2. Calculate the required sample before starting

The check that saves the months.

Sample size calculators are free and take a minute. Entering your actual conversion rate and monthly traffic usually returns a required duration measured in years, which settles the question honestly and prevents six wasted weeks.

3. Make large changes rather than small ones

The only tests worth running at low volume.

A different page, a different offer, a fundamentally different form. Large differences produce effects big enough to see without sophisticated measurement. Button colours and headline variations never will at this scale, whatever the case studies claim.

4. Use qualitative methods instead

Where the real value is at this scale.

Watching five people use the site, asking customers why they hesitated, and reading the questions your inbox receives will identify more problems than any test you can power. Five sessions is a genuinely useful sample for finding faults, even though it is useless for measuring how large they are, and finding them is what you actually need.

5. Apply what is already established

Do not test the settled questions.

Shorter forms, clearer actions, faster pages, visible prices and real photographs are established across many industries. Adopt them directly and spend whatever testing capacity you have on the questions genuinely specific to your own business, where no established answer exists.

6. Compare periods rather than variations

A pragmatic substitute.

Change one thing, then compare the following two months with the two before, allowing for season and campaigns. It is not rigorous and it should not be presented as though it were, but it is a perfectly reasonable basis for decisions at this scale.

7. Test upstream where the volume is

Where numbers do exist.

Advert copy, email subject lines and search listing titles accumulate impressions far faster than page conversions. Meaningful testing is possible there even when it is not possible on the page.

8. Beware tools that declare early winners

The specific trap.

Testing software will report a winner on tiny samples, and those results reverse routinely. Believing an early declaration is precisely how businesses end up adopting a change that actively costs them enquiries for a year before anybody questions it.

9. Prioritise by judgement and record what you did

The practical alternative.

Keep a list of changes with the date and reason, and review it against the numbers every quarter. This is how most small sites genuinely improve, and it is more honest than a test that was never powered.

Revisit the question as the business grows. Traffic that makes proper testing impossible today may support it in two years, and the discipline of recording changes in the meantime makes that transition straightforward.

Conclusion

Accept that a small site cannot reach statistical confidence, and stop spending months proving it.

Calculate the required sample before starting anything, restrict tests to large structural changes, rely on watching real people and asking real customers to find problems, adopt the practices already established across industries rather than retesting them, compare periods before and after a change, run genuine tests upstream where impressions accumulate, distrust early declared winners, and keep a dated record of every change and its reasoning.