Data quality as the precondition for AI, honestly assessed

5 days ago
3 min read

Updated: 2 days ago

Introduction

Every disappointing AI project in a small business has the same post-mortem. The tool worked as advertised, the process was sensible, and the output was wrong because the underlying records were incomplete, inconsistent or simply not there. The technology was never the constraint.

This is worth establishing before spending anything, because the assessment is quick and the answer changes what you should do. A business whose job records are complete and consistent has options that a business running on memory and a spreadsheet does not, and the second business is better served by six months of recording discipline than by any tool.

1. Data quality as the precondition for AI is checkable in an afternoon

Do it before buying.

Take the specific process you want to improve and ask: is the data recorded at all, is it consistent, is it complete, and is it accurate? Four questions, one process, one afternoon.

2. Ask whether it exists before asking whether it is clean

The first gate.

Many businesses want to predict job durations and have never recorded actual hours, or want to score leads and have never recorded outcomes. If the data does not exist, the project is a recording project first.

3. Check consistency, which is usually the weak point

The same thing recorded differently.

Free-text fields, categories applied inconsistently, dates in different formats, customers under several names. Inconsistent data is worse than missing data, because it produces confident answers rather than obvious gaps.

4. Test completeness on a real sample

Not on the schema.

Pull fifty records and count how many have the fields you need. The proportion is usually much lower than anyone expects, and it is the number that determines whether the analysis is possible.

5. Verify accuracy against reality where you can

The uncomfortable check.

Compare a sample of recorded values against what actually happened. Recorded job times that were entered to match the estimate, or completion dates set by whoever closed the record, are common and they invalidate everything downstream.

6. Judge whether the volume supports the question

Small numbers, modest claims.

Predicting from two hundred records is possible for simple relationships and not for subtle ones. Being realistic about this prevents both wasted effort and confident conclusions drawn from noise.

7. Fix the recording before analysing the history

The pragmatic sequence.

Where the historical data is unusable, start recording properly now and revisit in six months. This is a real and frequently correct answer, and it is cheaper than any attempt to reconstruct the past.

8. Improve quality where it is used, not everywhere

Proportionate effort.

A general data quality programme is unfinishable in a small business. Cleaning the specific fields the specific project needs is achievable and produces the same result for that project.

9. Put the controls in at the point of entry

Otherwise you clean forever.

Required fields, validated formats, controlled lists, and a duplicate check. Without these the data degrades back to its previous state within a year and the cleaning has to be repeated.

Where the data includes personal information, an assessment of quality is also a good moment to check the lawful basis for holding it, how long it has been retained and whether it is accurate — accuracy being an obligation in its own right in several jurisdictions.

Conclusion

Assess the data for the specific process before buying anything, because the check takes an afternoon and explains most failures.

Establish first whether the data exists at all rather than whether it is clean, check consistency because inconsistent records produce confident wrong answers, test completeness on a real sample of fifty records, verify a sample against what actually happened, be realistic about whether your volume supports the question, fix the recording now and revisit the history in six months where necessary, clean only the fields the project needs, and put validation at the point of entry so it does not degrade again.