← Engineering notebook
Evaluations & production

False defects: the real production killer

Finding defects is easy. Finding only real defects is hard.

Finding defects is easy. Finding only real defects is hard.

Busy surface texture that triggers false calls
Surface texture, dust and lighting changes all look like defects to an over-eager algorithm.

Common causes of false calls

  • Dust
  • Lighting changes
  • Surface texture
  • Process variation
  • Focus variation

The arithmetic nobody runs

A false-call rate that sounds excellent becomes a daily problem at production volume:

false-call rate:    0.2%  ("99.8% correct!")
daily volume:       100,000 parts
false rejects:      200 parts per day
manual review:      200 x 30 s  =  100 minutes of operator time, every day

And the real defect rate might be 50 parts per day - meaning 80% of everything the operator reviews is a false call. After a few weeks of that ratio, the operator stops looking carefully. Then a real defect gets waved through during review, and the system takes the blame for an escape it actually caught.

Run this arithmetic during the evaluation, not after deployment. A false-call rate is meaningless without the volume next to it.

Reduction strategies, in the order I try them

  1. Fix the imaging first - more contrast margin is worth more than any downstream cleverness. A defect at 10x noise tolerates variation that kills a defect at 3x.
  2. Add a confirming view - a second lighting angle or focus position that real defects survive and false triggers do not. Dust looks different from two angles; a void looks the same.
  3. Classify, do not just detect - separate "something is there" from "what is it". Measuring size, shape, and contrast of each candidate lets you reject the texture-shaped ones explicitly.
  4. Tighten the mechanics - a surprising share of false calls are position and focus variation wearing a defect costume.
  5. Only then tune thresholds - threshold tuning as a first resort just moves the problem between false calls and escapes.

Why it matters

Many projects spend more effort reducing false calls than they ever spent finding actual defects. A system that cries wolf gets switched off by operators, which is the worst possible outcome: now you have an expensive system and zero coverage.

A system nobody trusts is worse than no system at all.