Every sample lies - not because anyone is misleading, but because evaluation samples rarely represent reality.
Common issues with sample sets
- Sample size too small.
- Missing normal production variation.
- Supplier differences not represented.
- Only a handful of defect examples.
My evaluation process
- Prove feasibility - can it work at all?
- Gather variation - get samples across suppliers, lines, and time.
- Stress-test assumptions - actively try to break it.
- Deploy carefully - ramp slowly and watch for surprises.
The lesson
A successful evaluation proves where the solution stops working, not just where it works.