Object positioning is a fundamental task for assembly and inspection operations in the electronic industry. Traditional methods such as template matching and feature-point detection have been applied for object positioning. They are however computationally expensive and are generally affected by environmental changes. Deep learning models based on Convolutional Neural Network (CNN) have also been used for object positioning. They are computationally very efficient but can be sensible under environmental variations such as illumination and noise. In this article, a Cosine-Convolution operation is proposed with the objective of minimizing the effects of illumination variations. The proposed Cosine-Convolution can substitute any convolutional operation in a CNN-based regressor for positioning tasks. The proposed convolution is based on cosine-measure that normalizes the convolution between an image window and a filter at local scope. The proposed Cosine-Convolutional Neural Network (Cosine-CNN) performs more accurate than the traditional CNN under environment variations, including illumination changes, noise, defocusing and template occlusion. The experimental results reveal that the proposed model can reach mean prediction errors smaller than 1 pixel and$1^\circ$for object shift and rotation, regardless of the environmental variations. The method is computationally very fast, with an average evaluation time of 1.2-ms for real-time PCB positioning.Note to Practitioners—The training of deep learning models based on Convolutional Neural Network (CNN) can be tedious. Image samples with different illumination settings must be provided to have an efficient and robust model, which may not be easy to acquire or synthesize. This study proposes a solution to the challenge of illumination variations in images for the object positioning task in Printed Circuit Boards (PCB). A new Convolutional operation called Cosine-Convolutional that normalizes the traditional Convolution is introduced to overcome the illumination variations in the input images. Experimental results demonstrate that the proposed Cosine-Convolutional operation is less sensitive against lightning variations with respect to the training images.

With the rapid development of the semiconductor industry, fault diagnosis is an important task in routine operations to determine the root cause for faults that occur. A tool in manufacturing processes is equipped with a wide variety of sensors that record different types of process data. Wafers being processed are accompanied by a considerable amount of process data as a multivariate time series. Identifying the key process parameters and the corresponding process operations where faults occur can be used to facilitate the tasks of monitoring the process, maintaining the stability of the process, and stabilizing wafer production yield. This article proposes a novel solution procedure for fault diagnosis of wafer acceptance test (WAT) and chip probing (CP) using machine learning (ML). Based on the process flow of wafers and the corresponding process data, a sampling method, called synthetic minority oversampling technique (SMOTE), is first used to augment classification models with an imbalanced process dataset, and the best-practice SMOTE ratio is sought by using four competitive classifiers. By means of principal component analysis (PCA), the original data are transformed into visualizations to explore the distributions of good and bad lots of wafers. Based on the comparison of the four classifiers used in the test, the proposed logistic regression with data augmentation (LR-SMOTE) performs best. The ten most important features identified by using the LR are collected to determine potential failure operations. The identified failures in operations returned by using the proposed solution procedure for fault diagnosis of WAT and CP were confirmed by the engineers who work in the domain. Note to Practitioners—In semiconductor manufacturing practice, during the WAT and CP, domain engineers need to trace upstream the key process parameters and the corresponding key operations in the front-end-of-line (FEOL) process if any faults are detected in the back-end-of-line (BEOL) process. In fact, the FEOL and the BEOL are located in completely separate plants with different clean room standards. Bridging the information gap between these two different processes merits thorough scrutiny. Another challenge posed by fault diagnosis of WAT and CP is that the total number of features (i.e., process parameters) involved in the FEOL can be tens of thousands, leading to an adverse effect of data sparsity on model building.

To monitor process and identify the deviation as early as possible, data-driven methods have been applied for process monitoring and fault detection in semiconductor manufacturing. Although various fault detection and classification models had been discussed in the literature, however, little research has been devoted to feature selection from trace data that is important for process monitoring of natural variation. Additionally, the high-mix production mode with different recipes leads to process dynamic of wafer-to-wafer (W2W) variation which should also be identified for safeguarding false alarms and serving as a warning indicator. Therefore, this paper proposes a data-driven framework to identify the key features with respect to the W2W variation. In particular, the self-organizing map is used to annotate the grade of wafer variation among the in-line metrology data. Subsequently, the adaptive boosting (AdaBoost) is adopted to examine the effectiveness of every feature and its processing times, respectively. To validate the proposed framework, an empirical study from a semiconductor fabrication plant is conducted. The experimental results demonstrate that the key feature identification is of critical importance to build highly capable models for process monitoring. Through the dimensionality reduction technique, it has been illustrated that a smaller set of the identified key features are able to pinpoint the W2W variation of different wafer grades more clearly than the whole set of process features.Note to Practitioners— Process monitoring has become more difficult with the shrinking linewidth in semiconductor manufacturing. The challenges of analyzing equipment sensor or raw trace data for process monitoring in high-mix manufacturing processes are to incorporate subject-matter expert knowledge for setting control limit meticulously, to detect the subtle changes by analyzing the whole trace data profile, and to identify W2W variation for reducing false alarms. This paper proposes a data-driven framework for process monitoring by adopting data-driven approaches without recourse to domain judgement. Experimental results demonstrate that the proposed data-driven framework can effectively identify the key features via sensor readings and corresponding processing times, respectively. The engineers can make use of the extracted features to perform a predictive monitoring on metrology data for detection of potential process deterioration.

Fault detection and classification (FDC) is important for semiconductor manufacturing to monitor equipment's condition and examine the potential cause of the fault. Each equipment in the semiconductor manufacturing process is often accompanied by a large amount of sensor readings, also called status variable identification (SVID). Identifying the key SVIDs accurately can make it easier for engineers to monitor the process and maintain the stability of the process and wafer productive yields. This article proposes using the random forests algorithm to analyze the importance of SVIDs of equipment sensors, automatically filters the key SVID by using k-means, and integrates various machine learning methods to verify the key SVIDs and identify key processing time and steps. Upon the key parameters are identified, the key processing time and steps are investigated subsequently. The ensemble models constructed on k-nearest neighbors (kNNs) and naïve Bayes classifiers are presented for classifying wafers as normal or abnormal. Data visualization of multidimensional key SVIDs is performed by using t-distributed stochastic neighbor embedding (t-SNE) to create a graphical aid in FDC for the process engineer. An empirical study is conducted to validate the proposed data-driven framework for fault detection and diagnostic. The experimental results demonstrate that the proposed framework can detect abnormality effectively with highly imbalanced classes and also gain insightful information about the key SVIDs and corresponding key processing time and steps.

Solar power has become an attractive alternative of electricity energy. Solar cells that form the basis of a solar power system are mainly based on multicrystalline silicon. A set of solar cells are assembled and interconnected into a large solar module to offer a large amount of electricity power for commercial applications. Many defects in a solar module cannot be visually observed with the conventional CCD imaging system. This paper aims at defect inspection of solar modules in electroluminescence (EL) images. The solar module charged with electrical current will emit infrared light whose intensity will be darker for intrinsic crystal grain boundaries and extrinsic defects including micro-cracks, breaks and finger interruptions. The EL image can distinctly highlight the invisible defects but also create a random inhomogeneous background, which makes the inspection task extremely difficult. The proposed method is based on independent component analysis (ICA), and involves a learning and a detection stage. The large solar module image is first divided into small solar cell subimages. In the training stage, a set of defect-free solar cell subimages are used to find a set of independent basis images using ICA. In the inspection stage, each solar cell subimage under inspection is reconstructed as a linear combination of the learned basis images. The coefficients of the linear combination are used as the feature vector for classification. Also, the reconstruction error between the test image and its reconstructed image from the ICA basis images is also evaluated for detecting the presence of defects. Experimental results have shown that the image reconstruction with basis images distinctly outperforms the ICA feature extraction approach. It can achieve a mean recognition rate of 93.4% for a set of 80 test samples.

Template matching has been widely used in image processing for visual inspection of complicated, patterned surfaces. The currently existing methods such as golden template matching and normalized cross correlation are very sensitive to the displacement even the object under test is carefully aligned with respect to the template. This paper proposes a dissimilarity measure based on the optical-flow technique for surface defect detection, and aims at light-emitting diode (LED) wafer die inspection. The dissimilarity measure of each pixel derived from the optical flow field does not represent the true translation distance, but is reliable enough to indicate the degree of difference between an image pair. It is well tolerated to misalignment and random product variation. The integral image technique is applied to replace the sum operations in optical flow computation, and speeds up the intensive computation. We also point out the pitfall of the Lucas-Kanade optical flow when it is applied for defect detection, and propose a swapping process to tackle the problem. The experiment on LED wafer dies has shown that the proposed method can achieve a 100% recognition rate based on a test set of 357 die images.

In this paper, we propose a Hough transform-based method to identify low-contrast defects in unevenly illuminated images, and especially focus on the inspection of mura defects in liquid crystal display (LCD) panels. The proposed method works on 1-D gray-level profiles in the horizontal and vertical directions of the surface image. A point distinctly deviated from the ideal line of a profile can be identified as a defect one. A 1-D gray-level profile in the unevenly illuminated image results in a nonstationary line signal. The most commonly used technique for straight line detection in a noisy image is Hough transform (HT). The standard HT requires a sufficient number of points lie exactly on the same straight line at a given parameter resolution so that the accumulator will show a distinct peak in the parameter space. It fails to detect a line in a nonstationary signal. In the proposed HT scheme, the points that contribute to the vote do not have to lie on a line. Instead, a distance tolerance to the line sought is first given. Any point with the distance to the line falls within the tolerance will be accumulated by taking the distance as the voting weight. A fast search procedure to tighten the possible ranges of line parameters is also proposed for mura detection in LCD images.

This paper presents an automated visual inspection scheme for multicrystalline solar wafers using the mean-shift technique. The surface quality of a solar wafer critically determines the conversion efficiency of the solar cell. A multicrystalline solar wafer contains random grain structures and results in a heterogeneous texture in the sensed image, which makes the defect detection task extremely difficult. Mean-shift technique that moves each data point to the mode of the data based on a kernel density estimator is applied for detecting subtle defects in a complicated background. Since the grain edges enclosed in a small spatial window in the solar wafer show more consistent edge directions and a defect region presents a high variation of edge directions, the entropy of gradient directions in a small neighborhood window is initially calculated to convert the gray-level image into an entropy image. The mean-shift smoothing procedure is then performed on the entropy image to remove noise and defect-free grain edges. The preserved edge points in the filtered image can then be easily identified as defective ones by a simple adaptive threshold. Experimental results have shown the proposed method performs effectively for detecting fingerprint and contamination defects in solar wafer surfaces.

It is important in image restoration to remove noise while preserving meaningful details such as edges and fine features. The existing edge-preserving smoothing methods may inevitably take fine detail as noise or vice versa. In this paper, we propose a new edge-preserving smoothing technique based on a modified anisotropic diffusion. The proposed method can simultaneously preserve edges and fine details while filtering out noise in the diffusion process. Since the fine detail in the neighborhood of a small image window generally have a gray-level variance larger than that of the noisy background, the proposed diffusion model incorporates both local gradient and gray-level variance to preserve edges and fine details while effectively removing noise. Experimental results have shown that the proposed anisotropic diffusion scheme can effectively smooth noisy background, yet well preserve edge and fine details in the restored image. The proposed method has the best restoration result compared with other edge-preserving methods.

In this paper, an anisotropic diffusion model with a generalized diffusion coefficient function is presented for defect detection in low-contrast surface images and, especially, aims at material surfaces found in liquid crystal display (LCD) manufacturing. A defect embedded in a low-contrast surface image is extremely difficult to detect because the intensity difference between unevenly-illuminated background and defective regions are hardly observable. The proposed anisotropic diffusion model provides a generalized diffusion mechanism that can flexibly change the curve of the diffusion coefficient function. It adaptively carries out a smoothing process for faultless areas and performs a sharpening process for defect areas in an image. An entropy criterion is proposed as the performance measure of the diffused image and then a stochastic evolutionary computation algorithm, particle swarm optimization (PSO), is applied to automatically determine the best parameter values of the generalized diffusion coefficient function. Experimental results have shown that the proposed method can effectively and efficiently detect small defects in low-contrast surface images.

In video surveillance, detection of moving objects from an image sequence is very important for target tracking, activity recognition, and behavior understanding. Background subtraction is a very popular approach for foreground segmentation in a still scene image. In order to compensate for illumination changes, a background model updating process is generally adopted, and leads to extra computation time. In this paper, we propose a fast background subtraction scheme using independent component analysis (ICA) and, particularly, aims at indoor surveillance for possible applications in home-care and health-care monitoring, where moving and motionless persons must be reliably detected. The proposed method is as computationally fast as the simple image difference method, and yet is highly tolerable to changes in room lighting. The proposed background subtraction scheme involves two stages, one for training and the other for detection. In the training stage, an ICA model that directly measures the statistical independency based on the estimations of joint and marginal probability density functions from relative frequency distributions is first proposed. The proposed ICA model can well separate two highly-correlated images. In the detection stage, the trained de-mixing vector is used to separate the foreground in a scene image with respect to the reference background image. Two sets of indoor examples that involve switching on/off room lights and opening/closing a door are demonstrated in the experiments. The performance of the proposed ICA model for background subtraction is also compared with that of the well-known FastICA algorithm.

In this paper, we propose an eigenvalue-based similarity measure between two gray-level images and, in particular, aim at the application in defect detection. The pair-wise gray levels at coincident pixel locations in two compared images are used as the coordinates to plot the correspondence map. If two compared images are identical, the plot in the correspondence map is a diagonal straight line. Otherwise, it results in a non-linear shape in the correspondence map. The smaller eigenvalue of the covariance matrix of the data points in the correspondence map is used as the similarity measure. It will be approximately zero for two resembled images, and distinctly large for dissimilar images. Experimental results from a number of assembled PCBs (printed circuit boards) have shown the effectiveness of the proposed similarity measure for detecting local defects in complicated images.

The normalized cross correlation (NCC) has been used extensively in machine vision for industrial inspection, but the traditional NCC suffers from false alarms for a complicated image that contains partial uniform regions. In this paper, we study the use of NCCs for defect detection in complicated images. The performance of NCCs in monochrome and color images, and the effect of image smoothing are empirically evaluated. The proposed NCC in a smoothed color image can effectively alleviate false alarms in defect detection applications.