What to do when the numbers are too small to draw conclusions

5 days ago
3 min read

Introduction

A business with fifteen enquiries a month cannot use most of the measurement advice written for businesses with fifteen hundred. Split tests will not reach significance. Conversion rates jump ten points from one job going either way. Every chart looks dramatic and almost none of the drama is real.

The usual reactions are both wrong. One is to measure nothing on the grounds that the numbers are meaningless; the other is to treat every fluctuation as a signal and reorganise the business monthly. There is a workable middle, and it involves changing what you measure rather than how much.

Small numbers rule out statistical inference. They do not rule out counting, timing, or asking people questions, and those three carry most of the value anyway.

1. What to do when the numbers are too small is measure differently

The reframing.

Stop trying to detect small effects and start looking for large ones. At low volume only big changes are visible, and big changes are the ones worth making. This is a constraint that improves prioritisation. A change that only shows up in a large sample was probably not worth making.

2. Lengthen the period

The simplest adjustment.

A quarter has three times the data of a month and is still timely enough to act on. Annual comparison for anything seasonal. Patience substitutes for volume. Nothing is lost by waiting except the illusion of responsiveness.

3. Prefer elapsed time to rates

The better measure at small scale.

Days from enquiry to quote is stable with fifteen data points; conversion rate is not. It is also usually what you are actually trying to change. Start every small-business measurement here. Timings can be counted from a diary.

4. Count absolutes alongside ratios

The complementary view.

Two extra jobs is a fact; a nine-point rise in conversion is an artefact of the same two jobs. Reporting both prevents overstatement. The absolute number is easier for everyone to interpret. Put the count first and the percentage in brackets.

5. Use qualitative evidence properly

The underrated source.

Fifteen conversations is a small sample statistically and a rich one practically. The reasons people give for not buying are information even when the counts are not. Write them down verbatim.

6. Run before-and-after rather than split tests

The pragmatic method.

You cannot split fifteen enquiries into two groups usefully. Change one thing, hold everything else, and compare periods. It is weaker evidence and it is the evidence available. Give each period long enough to accumulate a reasonable count.

7. Change one thing at a time

The discipline that makes it work.

With small numbers you cannot untangle two simultaneous changes, ever. Sequence them. The wait is the price of being able to interpret anything.

8. Accept that some questions cannot be answered

The honesty.

Whether one button colour beats another is genuinely unanswerable at this volume, and pretending otherwise wastes months. Direct that effort at the questions your data can answer. There are more of them than people expect.

9. Aggregate across time rather than abandoning the metric

The long view.

Three years of quarterly figures is a real dataset even if any single quarter is not. Keep the series going. The value arrives later and it does arrive.

Be careful about borrowing conclusions from businesses much larger than yours. Findings that hold across ten thousand transactions frequently do not survive contact with a business where the owner speaks to every customer personally.

Conclusion

Look for large effects rather than small ones, and measure over longer periods.

Favour elapsed time over conversion rates when volumes are low, report absolute counts alongside ratios, take the reasons people give as seriously as the numbers, use before-and-after comparison instead of split testing, change one thing at a time so the result stays interpretable, accept that some questions cannot be answered at your scale, and keep the series running so that years accumulate into evidence.