Foreign material in food: you can't write rules for nature

No two food products look exactly alike. To find the defect, stop asking "what does a defect look like?" and start asking "what does a good product look like?"

Pour a handful of cashews onto a table. No two are the same. One curls more, one is shorter, one has a slightly chipped tip, and the color runs from pale ivory to deep cream. All of them are good. Then one has a small dark spot. That is the one your customer will see when they open the bag, and the one they will complain about.

That is the core problem of visual inspection in food, and of finding foreign material in food: when every product is different, how do you find the one that shouldn't be? In this post we look at why rule-based machine vision struggles with food, how a system that learns the good product flips the question, and where visual inspection should stop.

Where rules break down

Rule-based machine vision works well when every part is identical, but it struggles with food. In food, "correct" is not a single value but a wide range. Set the threshold tight and you reject good product. Set it loose and real defects get through.

Classic machine vision describes the defect as a rule: pixels below a color threshold, an expected object size, an edge shape. Anything outside the criteria is rejected.

For a screw, a cap, or a gasket, "correct" is clear. Try to write rules for food and you hit two options:

• Keep the threshold tight, and naturally darker or oddly shaped good products get rejected too. That is yield loss.
• Keep the threshold loose, and real defects fall inside the range and pass. That is customer complaints.

Every new product, harvest, or supplier lot can mean rebalancing the rules, an endless tuning job for the engineer.

How is foreign material in food detected with AI?

AI-based detection of foreign material in food works by showing the system good product, not by describing defects. The system learns how varied good product can be, compares every item on the belt against that reference, and marks anything outside normal with a heatmap and a confidence score. Even a never-seen object stands out.

This is anomaly detection, and it matters for food in two ways. You don't need a defect list, since nobody can predict every foreign object a line might see. And natural variation stops being a problem, because the system has learned it as "good."

ISEE Vision's AI visual inspection system for food and packaging lines, best known for label and fill level inspection, is added to the line as an independent station. It uses a light-isolated enclosure with controlled lighting, runs at line speeds up to 250 m/min, and checks every product rather than a sample. Images are processed on the device, with no cloud dependency, so data stays on site.

Your team decides which deviations raise an alert. Sensitivity and the acceptance limit are a threshold set per product, so the call is tied to a defined limit rather than an operator's judgment in the moment. For examples of the same logic in other industries, see our post on anomaly detection and its industrial applications.

Two examples, two different questions

Visual inspection in food answers two separate questions: is the product itself sound, and is there something in it that shouldn't be there? The same reference logic applies to both. The difference is what the system learns as "good."

A dark spot on a cashew: "Is this product sound?"

In a cashew recording from our sample work, every nut sat on the belt in a different shape and at a different angle. Despite that variation, the system marked the deviation on the nut with the dark spot in its heatmap. The question was about the product itself: is this nut inside the learned range of good cashews?

Foreign matter in a water carboy: "Is there something here that shouldn't be?"

The water carboy is a different question. The product, the water, always looks the same; what you are looking for is an object that got into it. In one carboy recording, several foreign objects sat at the bottom, and the system marked each of them separately. Nobody had to define in advance what the foreign matter would be. It was enough for the system to learn that, in a good carboy, nothing is there.

ISEE Vision Anomaly Detection analyzes the carboy's interior volume and tells particles apart by size and characteristics. So different particles aren't lumped under one "defect" label.

What can a camera see of foreign material in food, and what can't it?

Camera-based foreign body detection finds what is visible in the image: a stain or defect on the product surface, an object inside a transparent container, or a fault on the outside of the package. It cannot see a metal or glass fragment buried inside an opaque product or a sealed pack. That stays the job of metal detection or X-ray.

A system sold as "AI sees everything" loses trust at the worst moment. Visual inspection doesn't replace the existing controls in your HACCP plan. It adds to them for visible defects and for objects in transparent media.

The packaging side, meaning labels, caps, barcodes, and fill level, is a topic of its own. We cover it in 8 questions about packaging inspection on filling lines.

What to get right at installation

Half of a good result in food comes from the camera and lighting. Products need to flow in a single layer without overlapping. Lighting has to avoid glare on shiny surfaces and, in transparent packaging, make the foreign object stand out.

That is why the enclosure is engineered for each line, with camera count, lens, lighting angle, and working distance chosen for it. Camera position and the distance to the reject mechanism are best decided together, on site.

Why isn't the human eye enough for manual sorting?

An experienced sorter catches a lot, but even under the best conditions some defects get through, especially with small, fast-moving products that look alike. The problem is the nature of the task, not the sorter's skill.

Many food plants still sort by hand. One of the clearest studies of visual inspection limits comes from defense manufacturing, not food. At Sandia National Laboratories, 82 inspectors examined 140 precision-manufactured nuclear-weapons parts for eight defect types (See, Human Factors, 2015). They correctly rejected 85% of the defective items. The author notes this is "not vastly superior to the industry average of 80%," which means roughly one defect in five gets through typical visual inspection. In this study the high hit rate also came with 35% of acceptable parts rejected, a scrap rate the author says is not typically observed. In food the lesson is direct: a missed defect becomes a complaint, and a tighter threshold turns good product into waste.

And in food, a missed defect can turn into a recall and a question of consumer trust, not just a return.

Who makes the call?

When the system flags a product, it shows the heatmap, the confidence score, and the image of the defect. You configure what happens next: alert only, alert plus line stop, or full automation with a sorting signal. Outputs include a siren, tower light, on-screen notification, email to the responsible staff, and a PLC or relay signal to the reject mechanism.

Whether a flagged product is scrapped, reworked, or accepted is your quality team's decision. Foreign material and a small color deviation are not handled the same way, and your quality policy sets that line.

Records: what do you show the auditor?

Under HACCP, BRCGS, or IFS audits, auditors often ask for evidence that the check was done.

Every detection is stored time-stamped, with the original image, the heatmap, and the confidence score. Daily, weekly, and monthly reports are produced automatically, with distribution by defect type, product code, and shift. These records help document your control activity during audits. The system itself is not a certificate and doesn't guarantee compliance, but it makes it easier to answer "what did you see in this lot?" with images.

Frequently asked questions

Does AI replace metal detection or X-ray?

No. A camera only finds what is visible. Metal detection or X-ray stays in place for metal or glass inside opaque products and sealed packs.

Do we have to retrain the system when the season or product changes?

No. A new product is taught on site with good samples and saved as a profile under its product code. When a saved product comes back, the operator selects it on the control panel. No mechanical adjustment or reinstallation is needed.

Does the system stop the line?

Only if you want it to. A relay or PLC line stop is one option; alert only and automatic sorting are the others.

Do the records guarantee a HACCP or BRCGS audit?

No. Records help document your control activity, but compliance is judged on your quality system as a whole.

Key takeaways

It is hard to write rules for a natural product, but easy to show what a good one looks like. That is why learning the good product works for foreign material in food.

• With rule-based systems, natural variation in food turns into either yield loss or missed defects. A system that learns good product treats that variation as normal.
• The same logic finds both a defect in the product itself and a foreign object in a transparent container.
• Visual inspection doesn't replace metal detection or X-ray for metal and glass inside opaque products. It complements them.
• The system checks and records every product. The disposition stays with your quality team.

You can also explore all of ISEE Vision's automated visual inspection solutions. We can start with a case of good product from your line and a few defective samples you already have. Get in touch, bring your samples, and in a free trial we'll see together what the system flags on your product.

Contact: info@isee-vision.com · +90 216 455 95 44