AI and the risk of quiet errors that nothing flags

5 days ago
3 min read

Updated: 3 days ago

Introduction

Traditional software fails loudly. A field is missing, a calculation divides by zero, a file will not open, and somebody knows immediately that something is wrong. The error is visible, it is fixed, and the damage is contained by the fact that the process stopped.

These tools fail differently. They produce an answer that is well-formed, plausible and wrong, and nothing about the output signals it. A figure is slightly off, a reference is invented, a clause is summarised incorrectly, a category is misassigned. The process continues, the output is used, and the error is discovered by a customer, an auditor or nobody. Designing around this is the central practical skill in adopting them.

1. AI and the risk of quiet errors changes where the checks belong

Move them downstream.

You cannot rely on the system to report its own uncertainty. The check has to be a deliberate step in the process, applied to the output, by somebody with the context to notice.

2. Identify what a wrong answer would cost

The basis for how much checking.

A misrouted internal email costs minutes. A wrong figure in a quote costs the margin on a job. A wrong safety statement costs considerably more. The checking effort should be proportionate and it rarely is.

3. Check the categories where invention is common

Known weak points.

Specific figures, dates, names, citations, references to documents, and quantities. These are precisely the elements that look most authoritative and are most frequently fabricated or transposed.

4. Verify against the source, not for plausibility

The distinction that matters.

Reading output to see whether it seems right catches almost nothing, because plausibility is what these systems produce. Checking a figure against the record that contains it is a different act and it is the one that works.

5. Sample continuously rather than checking once

Behaviour drifts.

A tool validated at implementation may behave differently after an update, a data change or a shift in what it is being asked. A small ongoing sample check is what detects that, and its absence is why problems run for months.

6. Watch for the errors that are systematically in one direction

More damaging than random ones.

A consistent bias — always rounding the same way, always misclassifying one category, always omitting a particular element — accumulates. Random errors average out; systematic ones do not.

7. Keep a record of every error found

The calibration data.

What was wrong, in what category, and how it was detected. After thirty entries you know where this tool is unreliable in your context, which is information no vendor can give you.

8. Do not let confidence in the tool reduce the checking

The trajectory of most implementations.

Checking is thorough in month one, perfunctory by month four, and abandoned by month nine, at which point the exposure is at its highest. A fixed sampling rate, treated as part of the process, resists this.

9. Decide what must never be automated

The explicit boundary.

Safety statements, regulated advice, contractual commitments, anything with legal effect, and decisions about individuals. Writing this list down is more effective than expecting judgement in the moment.

Where output is used in regulated advice, safety-critical work, or decisions affecting individuals, the accountability remains with the business and the professional regardless of how the output was produced. That position is consistent across jurisdictions even where the detailed rules differ.

Conclusion

Design the checking into the process, because these tools do not fail visibly.

Size the checking effort by what a wrong answer would cost, pay particular attention to figures, dates, names and references, verify against the source rather than reading for plausibility, sample continuously because behaviour drifts after updates, look for errors that are consistently in one direction, keep a log of every error found to learn where the tool is weak in your context, hold the sampling rate steady as confidence grows, and write down what must never be automated at all.