Separating a real change from noise in a small funnel

4 days ago
3 min read

Introduction

A number moves and somebody has to decide whether it means anything. There is no statistical test available at fifteen enquiries a month that will answer this, and waiting for certainty means waiting forever. What is available is a small set of rules of thumb that get the decision right most of the time.

The cost of getting it wrong runs in both directions. Treating noise as signal produces constant reorganisation and destroys the ability to attribute anything to anything. Treating signal as noise means a genuine decline runs for a year before anyone acts.

These rules are crude on purpose. Crude and consistently applied beats sophisticated and abandoned.

1. Separating a real change from noise starts with direction, not size

The first rule.

Three consecutive periods moving the same way is more meaningful than one large jump. Runs are hard to produce by chance. Size is easy. Two periods is interesting; three is worth acting on.

2. Ask what the smallest unit of change is

The arithmetic check.

If one job moves the rate by five points, a five-point move is one job. Work this out once for each metric. It ends most arguments immediately. Write the figure on the report so nobody has to recompute it.

3. Compare against the same period last year

The seasonal check.

Half of all apparent changes in a small business are the calendar. This check costs nothing and eliminates them. It requires only that you kept the records. A second prior year makes it more reliable again.

4. Look for a cause before believing the effect

The plausibility test.

A change with an identifiable cause and the right timing is far more credible than one without. Absence of a cause is not proof of noise, but it raises the bar. Ask what happened in that period. Check the calendar for that month before the meeting.

5. Check whether the counting changed

The measurement check.

New person, new system, new channel being logged. This explains more alarming numbers than any commercial factor. Always check it before anything else. It is also the fastest thing to rule out.

6. Look at the absolute numbers

The sanity check.

Ratios exaggerate at small scale in a way that raw counts do not. Two jobs is two jobs regardless of the percentage. Read both before concluding. Percentages are for comparison; counts are for judgement.

7. Use a wider window when unsure

The default action.

Almost every ambiguous signal resolves itself with another quarter of data. Waiting is usually cheap, and acting on a mistaken signal is usually not. Say explicitly that you are waiting rather than quietly ignoring it.

8. Set the threshold before you look

The discipline.

Deciding in advance what movement would prompt action removes the retrospective reasoning that makes any number meaningful. This is the same idea as agreeing a target before a test. It is much harder to apply after the fact.

9. Take a run of small changes seriously

The asymmetry.

Slow decline is harder to detect than a sudden drop and does considerably more damage, because nothing ever triggers a response. Look at the twelve-month shape once a quarter. That is where slow decline shows.

Be careful about a change that is real but caused by something outside the funnel. A large contract ending, a staff departure or a supplier problem all move these numbers, and the response to those is not a change to the marketing.

Conclusion

Trust a run in one direction more than a single dramatic period.

Work out how many jobs it takes to move each metric by a point, compare against the same period a year ago, look for an identifiable cause with the right timing, check whether the counting method changed before assuming performance did, read absolute counts alongside ratios, widen the window when the signal is ambiguous, agree action thresholds in advance, and check the twelve-month shape each quarter to catch slow decline.