Can AI Improve Manufacturing Quality?

The problem, in numbers

Many visual quality checks in manufacturing still come down to a person looking at a part. It is flexible and cheap to set up, but it is not reliable. A 2015 review of human factors in inspection cites Drury and colleagues' view that inspectors may misclassify 20 to 30% of items, well above an older estimate of 3 to 10% for simple inspection tasks.^{Source 1} The earlier studies it tabulates, mostly outside factory lines, are sobering: inspection effectiveness of 68% for aircraft visual inspection, 53% for subsea structures and pipelines, and anywhere from 45% to 100% for soldering defects.^{Source 1} Those numbers come from different industries and decades, but they point the same way. A human inspector is a good instrument with a wide error band.

Manual inspection is also still common. In a survey of 110 manufacturers published in 2020 by the robotics association A3 and AI vision vendor Landing AI, 40% said their inspection was completely or mostly manual, and false calls were the most cited inspection problem, named by 56%.^{Source 2} When the check is a sample rather than every unit, the defects that slip through are simply never looked at.

What improves, and why

A vision model looks at every part the same way on the first shift and the last. It does not get bored, it does not relax its standard on a Friday afternoon, and it can check every unit at line speed rather than a sample pulled every hour. That is the core of the quality case: not a sharper eye, but a steadier one applied to everything.

The second gain is earlier detection. A camera at the station where a defect is created can flag it before more value is added to a part that will be scrapped anyway. The third is a record. Every image and every decision is stored, which turns quality from a tally of rejects into data that engineers can trace back to a machine, a batch or a setting.

What the evidence actually says

On benchmarks, the technology is close to solved. A 2024 peer-reviewed survey of industrial anomaly detection says most deep-learning methods are more than 97% accurate, and on the widely used MVTec AD dataset one well-known approach, PatchCore, reports an image AUROC of 0.992, with several rivals slightly higher.^{Source 3} The same authors warn that most of these datasets were not generated on real production lines and that genuine defect samples are scarce in practice.^{Source 3} One peer-reviewed study shows the pattern: 99.86% test accuracy on 715 images of cast pump impellers, from a system that was not yet on an assembly line and still relied on manual image uploads.^{Source 4}

The strongest factory evidence we found comes from a controlled experiment at a Siemens electronics plant in Switzerland. Forty-eight factory workers made 9,600 judgements on circuit-board images with AI support. Workers whose AI showed where it saw a problem found 93.0% of defects, against 82.0% for those given a black-box AI, and balanced accuracy was 96.3% against 88.6%.^{Source 5} That is a real result from real workers. It also has limits: there was no group working without AI, so it compares two ways of using AI rather than AI against none, and the authors note that defects were more common in the test than in normal work.^{Source 5}

What is missing is the outcome that matters most: independent, controlled evidence that AI inspection reduces the defects reaching customers, or scrap and warranty cost, across many sites. Vendor case studies make those claims often. We did not find peer-reviewed work that measures them. That gap does not mean the gains are imaginary. It means the burden of proof sits with the site doing the deploying, not with the brochure.

Where it falls short

• Defects are rare by design. A good line produces few bad parts, so there is little to train on. In the vendor-backed survey, 62% named scarce defect data as the top obstacle, and 41% said integration with existing infrastructure is complex.^{Source 2}

• Benchmarks flatter. Most public datasets were not generated on a real production line, and the survey's authors warn their defects may not match those that occur in actual production.^{Source 3}

• Performance is local. NIST's industrial AI programme notes that a tool's performance means little outside its effect on the system and the people using it.^{Source 6} A model validated on one line is a hypothesis on the next.

• Explanations can mislead. The heatmaps that helped Siemens inspectors can be unreliable, and the authors tested only one form of explanation in one setting.^{Source 5}

Where governance decides the outcome

A quality model makes release decisions, and some of those parts end up in brakes, medical devices or structures. Someone has to own the reject threshold, the trade-off between false rejects and escapes, and the decision to revalidate when the product, supplier or camera changes. Vendor accuracy claims need testing on your own parts before they are trusted. In Australia, the 2024 Responsible AI Index, a cross-industry survey, found a 49-point gap between organisations' average agreement that their AI is run responsibly (78%) and the share actually implementing the matching practices (29%), and nearly 7 in 10 had not assessed and did not plan to assess vendor claims about black-box AI performance.^{Source 7} NIST's diagnosis is similar: without standard ways to evaluate industrial AI, the result is hesitation, mistrust and misapplication.^{Source 6}

How to think about adoption

Start where the case is strongest: high-volume parts with visible surface defects, a stable product and a known cost per escape. Measure your current human catch rate before you buy anything, because without a baseline no improvement can be proven. Plan for the data problem early by collecting and labelling defect images from your own line. Run the model in shadow mode beside existing inspection, long enough to compare the two on the same parts. Keep people in the loop for uncertain calls, and show them what the model saw. Then judge the system on the outcomes that matter (escapes, scrap and returns) rather than on the accuracy figure in the sales deck.