A camera and a vision model can review every solder joint on every unit, which a human inspector cannot sustain at production pace. Learn how to capture each joint as an attachment, let a model flag suspect joints, and have the operator confirm or reject each flag from a TofuPilot operator prompt, keeping AI as a flagging aid next to IPC-A-610 criteria.
What a Vision Model Can and Cannot Judge
A camera sees the outside of the joint. Everything a model concludes comes from that surface.
| Defect | Visible to a camera | Notes |
|---|---|---|
| Solder bridge between pads | Yes | Clear contrast, good model target |
| Missing or insufficient solder | Yes | Fillet shape and coverage |
| Stray strands outside the joint | Yes | Needs enough resolution on the wire exit |
| Burnt or melted insulation | Yes | Colour change |
| Dull or grainy surface (possible cold joint) | Partly | Depends on lighting, alloy and flux residue |
| Crack under the fillet | No | Use an electrical measurement under flex |
| Poor wetting of strands inside the joint | No | Use a 4-wire voltage drop |
| Voids inside the joint | No | X-ray or cross-section |
The electrical methods for the hidden defects are covered in Screen Solder Joints Without Destroying Them and Catch Intermittent Contacts in Production.
Choose the Model Type
| Approach | Trained on | Output | Strength | Weakness |
|---|---|---|---|---|
| Supervised classifier | Labelled images of good joints and of each defect type | A class per joint | Names the defect | Needs many labelled examples of each defect, which are rare on a good line |
| Anomaly detector | Images of good joints only | A score: distance from "normal" | Starts with good units only | Flags any change, including harmless ones (lighting, flux residue, wire colour) |
Anomaly detection is the usual starting point because defects are rare at the beginning. The confirmations collected by the procedure below become the labelled set for a classifier later.
Three limits apply to both approaches:
- False positives. A threshold low enough to catch real defects also flags good joints. The operator review step absorbs them.
- False negatives. A defect the model was never shown, or one that looks normal on the surface, passes. AI does not replace the human inspection against IPC-A-610 criteria; it directs attention.
- Sensitivity to setup. A moved camera, a changed light or a new board revision shifts every score. Lock the optical setup and re-validate the model after any change.
Prerequisites
- The TofuPilot CLI installed on the test station
- A camera on a fixed mount, a fixed light source (a ring light or a diffuse dome), and a fixture that places each board in the same position
- An anomaly model exported to ONNX that takes one joint crop and returns one score; the preprocessing in this guide (RGB, 224 × 224, values in 0 to 1) is an example and must match how your model was trained
- The joint positions in the image, as rectangles measured once on a reference board
opencv-python-headless>=4.8onnxruntime>=1.17numpy>=1.24Step 1: Write the Camera Plug
Plug methods exchange JSON-serializable values with the phase, so the plug writes the image to disk and returns the path, not the pixel array.
plugs/camera.py26 lines
import osimport cv2class Camera: def __init__(self, index=0, width=3840, height=2160): self._cap = cv2.VideoCapture(index) self._cap.set(cv2.CAP_PROP_FRAME_WIDTH, width) self._cap.set(cv2.CAP_PROP_FRAME_HEIGHT, height) if not self._cap.isOpened(): raise RuntimeError(f"Camera {index} could not be opened") def capture(self, out_dir, name): # Drop frames buffered by the driver before the board was in place. for _ in range(5): self._cap.read() ok, frame = self._cap.read() if not ok: raise RuntimeError("Camera returned no frame") path = os.path.join(out_dir, name) cv2.imwrite(path, frame) return path def __del__(self): self._cap.release()Step 2: Write the Inspector Plug
The inspector crops each joint, scores it, saves the crop and returns one entry per joint.
plugs/inspector.py42 lines
import osimport cv2import numpy as npimport onnxruntime as ortclass JointInspector: def __init__(self, model_path, threshold, rois, input_size=224): # Resolve a relative path against the procedure directory, not the working directory. if not os.path.isabs(model_path): model_path = os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", model_path) self._session = ort.InferenceSession(model_path) self._input_name = self._session.get_inputs()[0].name self._threshold = threshold self._rois = rois self._size = input_size def inspect(self, image_path, out_dir): image = cv2.imread(image_path) results = [] for roi in self._rois: x, y, w, h = roi["x"], roi["y"], roi["w"], roi["h"] crop = image[y : y + h, x : x + w] crop_path = os.path.join(out_dir, f"{roi['joint']}.png") cv2.imwrite(crop_path, crop) # Must match the preprocessing used at training time. rgb = cv2.cvtColor(crop, cv2.COLOR_BGR2RGB) tensor = cv2.resize(rgb, (self._size, self._size)).astype(np.float32) / 255.0 tensor = np.transpose(tensor, (2, 0, 1))[np.newaxis, ...] score = float(self._session.run(None, {self._input_name: tensor})[0].ravel()[0]) results.append( { "joint": roi["joint"], "score": score, "flagged": score >= self._threshold, "crop_path": crop_path, } ) return resultsStep 3: Capture and Score in a Phase
phases/inspect.py22 lines
import base64import tempfileimport cv2def capture_and_score(camera, inspector, measurements, attach, log): work_dir = tempfile.mkdtemp(prefix="joint-inspect-") image_path = camera.capture(work_dir, "board.png") attach.file(image_path, "board.png") results = inspector.inspect(image_path, work_dir) flagged = [r for r in results if r["flagged"]] measurements.max_anomaly_score = max(r["score"] for r in results) measurements.flagged_count = len(flagged) measurements.flagged = { "joints": [r["joint"] for r in flagged], "crops": {r["joint"]: r["crop_path"] for r in flagged}, "scores": {r["joint"]: round(r["score"], 4) for r in flagged}, } log.info(f"{len(flagged)} of {len(results)} joints flagged")The full board image is attached to every run. The flagged dictionary is a JSON measurement, which the next phase reads through previous results.
Step 4: Ask the Operator to Confirm Each Flag
Add a second function to the same file. It shows the flagged crops side by side and asks which ones are real defects. The checklist options are filled at run time with ui.set, and ui.wait() returns the operator's answer.
phases/inspect.py23 lines
def confirm_flags(capture_and_score, phase, ui, measurements, attach): flagged = capture_and_score.flagged if not flagged["joints"]: phase.skip("No joint flagged by the model") tiles = [] for joint in flagged["joints"]: tile = cv2.resize(cv2.imread(flagged["crops"][joint]), (320, 320)) cv2.putText(tile, joint, (8, 28), cv2.FONT_HERSHEY_SIMPLEX, 0.8, (0, 255, 255), 2) tiles.append(tile) ok, png = cv2.imencode(".png", cv2.hconcat(tiles)) ui.flagged_image = "data:image/png;base64," + base64.b64encode(png.tobytes()).decode() ui.set("confirmed", options=[{"value": j, "label": j} for j in flagged["joints"]]) answer = ui.wait() confirmed = answer["confirmed"] # Name each crop with the verdict: the attachments become a labelled dataset. for joint in flagged["joints"]: verdict = "defect" if joint in confirmed else "false_positive" attach.file(flagged["crops"][joint], f"{joint}_{verdict}.png") measurements.confirmed_defects = len(confirmed)Step 5: Declare the Procedure
procedure.yaml59 lines
name: ESC Joint Visual Inspectionversion: 1.0.0execution: workers: 1plugs: - name: Inspection Camera key: camera python: plugs.camera:Camera scope: station with: index: 0 - name: Joint Inspector key: inspector python: plugs.inspector:JointInspector scope: station with: model_path: "models/joint_anomaly.onnx" threshold: 0.6 rois: - { joint: bat_pos, x: 410, y: 220, w: 300, h: 300 } - { joint: bat_neg, x: 760, y: 220, w: 300, h: 300 } - { joint: m1_a, x: 1210, y: 640, w: 260, h: 260 }main: - name: Capture and Score key: capture_and_score python: phases.inspect:capture_and_score measurements: - name: Max Anomaly Score key: max_anomaly_score - name: Flagged Count key: flagged_count - name: Flagged Joints key: flagged - name: Confirm Flagged Joints key: confirm_flags python: phases.inspect:confirm_flags depends_on: [capture_and_score] ui: components: - key: flagged_image type: image label: "Joints flagged by the model" width: 100% fit: contain - key: confirmed type: checklist label: "Select the joints that are real defects per IPC-A-610" description: "Leave all unselected if every flag is a false positive" required: false measurements: - name: Confirmed Defects key: confirmed_defects validators: - operator: "==" expected_value: 0Design choices in this file:
- Only a confirmed defect fails the unit. The model's flag is recorded but has no validator. The operator, trained to IPC-A-610, makes the accept or reject decision.
- The threshold is an example. Set it from scores measured on known-good units and on the defects you have, then review it as confirmations accumulate.
- When nothing is flagged, the confirmation phase skips itself before recording anything, so it does not count as a failure. Its screen opens at phase start and closes on the skip.
scope: stationkeeps the camera open and the model loaded between units.
Step 6: Measure How Well the Model Performs
Every run now records how many joints the model flagged and how many the operator confirmed. In the TofuPilot dashboard:
| Question | Where to read it |
|---|---|
| Is the false-positive rate acceptable | Flagged count against confirmed defects across runs |
| Is the model drifting (light, camera, board revision) | Control chart of the maximum anomaly score |
| Which joint is most often confirmed defective | Failure Pareto on confirmed defects, and the attachments of each run |
| Is one operator rejecting flags others confirm | Confirmed defects filtered per operator |
A rise in the anomaly score across all units at once usually comes from the setup, not the soldering. A rise on one joint position is a process signal, to investigate with Why ESC Solder Joints Fail in Flight.
Operator agreement matters as much as model accuracy. If two inspectors give different verdicts on the same crop, the label is unreliable. Inspector agreement is a measurement system question, and the principles in What Is GR&R and MSA apply to it. Keep the inspection record as described in IPC-A-610 Inspection Tracking.
Common Pitfalls
Using the model as the pass/fail decision. A model has a false-negative rate, and nobody reviews what it passes. Keep the human verdict as the decision and the model as a flag.
Changing the lighting or the camera without re-validating. Every score shifts. Lock the optical setup and repeat the validation on known-good and known-defective units after any change.
Returning images from a plug. Pixel arrays are not JSON-serializable. Return a file path.
Preprocessing that differs from training. BGR instead of RGB, a different input size or normalization gives scores that look plausible and mean nothing.
Expecting the camera to find hidden defects. Cracks under the fillet, poor internal wetting and voids are invisible. Pair the inspection with a 4-wire measurement.
Discarding the operator's verdicts. The named crops are the labelled defect images the next model needs.
Key Points
- A camera and a model can review every joint, but only its visible surface.
- Start with an anomaly detector trained on good joints; collect labelled defects through operator confirmations.
- Capture the board as an attachment, score each joint crop in a plug, and record the flags as measurements.
- Let the operator confirm each flag with
ui.wait(); only confirmed defects fail the unit. - Track false positives and score drift in the dashboard, and re-validate after any change to the optical setup.
- Keep IPC-A-610 inspection and electrical measurement in place; AI directs attention, it does not replace them.
