Anomaly detection in the industrial manufacturing process is important for controlling product quality. In real-world scenarios, anomalies manifest in a number of diverse and unforeseen ways and can be categorized into two main types: structural anomalies, such as scratches and breakages, and logical anomalies, such as missing components and dislocations. Current mainstream methods tend to focus on knowledge generalization but are inadequate for detailed and logical detection tasks. To address this issue, this study proposes Biomimetic Visual Perception Network (BVPN), a biomimetic design-based novel paradigm that simulates biological vision and human logical discrimination with detailed distillation dependencies and semantic component information. BVPN introduces three convolution-based networks to extract information from different regions, imitating biological central, para-central, and peripheral vision, and facilitates the integration of disparate information through the process of knowledge distillation. On the basis, BVPN develops two modules based on the student-teacher network: Focus Module, for local detailed detection, and Peripheral Module, for global context detection. Moreover, BVPN proposes the Logical Module, a clustering-based segmentation branch, assessing logical anomalies by the number and color information of segmentation components. Through a fused structure highly aligning with the perception mode of human system, BVPN achieves excellent performance in the detection of structural and logical anomalies. Extensive experiments with mainstream anomaly detection datasets and real-world inkjet printing dataset demonstrate that BVPN outperforms state-of-the-art competitors in accuracy.

Visual inspection of surface defects in industrial products is crucial for quality control but remains challenging due to unpredictable and multiscale defects. Unsupervised anomaly detection methods based on feature distribution have rapidly developed, requiring only normal samples and no additional manual labeling. However, these methods still suffer from feature bias and image misalignment, leading to overdetection of multiscale defects, making them impractical for deployment in factory settings. To address these issues, this study proposes a hierarchical patch feature distribution modeling (HPDM) network. Specifically, the new patch feature alignment learning (PFAL) module mitigates image misalignment by enhancing feature dimension similarity via the PFAL loss function. Additionally, the proposed discrete feature memory (DFM) module employs linear mapping to increase the density of the normal feature distribution, addressing feature bias from pretrained networks. Furthermore, the pyramid-shaped Gaussian distribution (PGD) module fits multivariate Gaussian models at different feature scales for defect detection across various scales. On the MVTec2D, MVTec3D, and VISION datasets, the HPDM network achieves pixel-level AUROC scores of 98.83%, 98.30%, and 71.20%, respectively, surpassing those of the existing methods and demonstrating state-of-the-art anomaly detection performance. The experimental results on a real printed circuit board assembly (PCBA) circuit dataset demonstrate a detection speed of 30 frames per second (FPS), meeting the requirements for factory online detection. Note to Practitioners—Previous unsupervised anomaly detection methods have struggled to detect both large and small surface defects simultaneously. However, the proposed multiscale patch feature modeling network effectively identifies various multiscale defects on product surfaces, including PCBA, thin-film transistor liquid crystal displays, and tiles. Additionally, HPDM requires only a small number of normal samples to learn a robust network, which is crucial for industrial applications where identifying and labeling samples is challenging. Furthermore, by leveraging model pruning and quantization techniques, HPDM can be applied to online visual inspection.

Robotic ultrasound (US)-guided surgery is one of the most common clinical workflows for many diseases. Intraoperative tissue deformation is a key challenge faced by robots in realizing autonomous surgery. This study develops a US-guided robotic autonomous operation scheme based on deformation tracking and prediction, which can overcome the adverse effects of intraoperative deformation. First, a deformation tracking method based on real-time nonrigid image registration is proposed to track tissue and path deformation in real time. Simultaneously, we propose a deformation prediction model that is updated by the forgetting factor recursive least squares for predicting the next deformation according to historical tracked data. Then, a time-varying vector field navigation law is incorporated to guide the robot following the dynamically deformed surgical path, therefore compensating for deformation. To verify the performance of the proposed method, a dual-arm robotic US surgical navigation system prototype is constructed, andin vitroexperiments in a deformable environment are conducted. The results demonstrated that our method can guide the robot to compensate for tissue deformation in real-time with a dynamic path following error within 1 mm.

Uncontrolled intersections, as a typical urban traffic environment, are challenging scenarios for autonomous driving due to the potential conflicts and lack of co-ordination between traffic participants. Even if both vehicles are rational players, the existence of multiple Nash equilibira in this dynamic game may result in no consensus between them during the interaction. This may lead to unsmooth rides with late emergency brakes. This paper proposes a conflict-aware game theoretic decision-making framework for autonomous vehicles at uncontrolled intersections considering the Nash equilibrium selection. Combining dynamic programming and iterative best response, a dynamic closed-loop Nash equilibrium calculating method is proposed. The issue of multiple equilibria is analyzed. An equilibrium selection mechanism is designed based on sequentially-constructed Conflict-Aware Bimatrix Games (CAB-Games) and the idea of risk dominant equilibria. The proposed policy provides a relatively smooth drive without being too aggressive or conservative. It is demonstrated in simulations and driving simulator experiments that the policy can interact well with other autonomous driving policies as well as human drivers.

Spotting blemished areas automatically on a textured surface is a particular challenge, as both nominal and defective surface samples are inconsistent in large-scale industrial manufacturing. The most efficient solution uses the memory bank extracted from the nominal samples to detect outliers. We approach our strategy, the multi-category decomposition editing network (MCDEN), from a similar viewpoint. Notably, we do not use defect-free samples. Instead, we use virtual results to construct a defect library. MCDEN decomposes abnormalities to basic elements from the library while editing outlier features to reconstruct the texture normality, offering a rational segmentation map through decomposition and reconstruction. Based on this strategy, MCDEN is more interpretable than most neural network methods since interpretability is particularly important in industrial production to ensure stability; however, the existing deep learning methods are similar to a black box structure, which makes MCDEN more appropriate in industry. Experiments on texture surface samples from the MVTec anomaly detection (MVTAD) dataset confirm the efficacy of MCDEN with a pixel-level area under the receiver operator characteristic curve (AUC) score of 96.6%. In other experiments collected from semi-manufactured inkjet printing organic electroluminescence display (OLED) panels, MCDEN demonstrated competitive results with a 99.2% detection rate and rapid real-time detection capabilityNote to Practitioners—MCDEN detects defects based on the assumption that texture defects are decomposed into five basic transformation combinations. The method does not need to collect additional defect images for model training. It only needs to collect about 2000 positive defect-free images to generate negative samples through the random generation of defects and complete the training. All training images and detection images are based on a single-channel gray image. The MCDEN method can be applied to the detection of object defects on textured surfaces with periodic features, such as steel, leather, and screen. Based on the known texture features, the trained model has high detection accuracy and real-time detection for specific types of textured surfaces.

Image-guided depth completion (IGDC) is a multimodal computer vision task for acquiring high-precision dense depth maps. It reasonably predicts values around accurate sparse depth measurements by relying on the details of simultaneous dense RGB images. To achieve this goal, spatial propagation networks (SPNs) elaborate context-aware meta cells that connect pixels to their neighbors and fuse sparse and dense modalities by linear propagation. However, static affinity matrices and fixed neighborhood connections limit the representation of the networks. In this paper, our proposed dynamic SPN (DySPN) uses a nonlinear propagation model (NLPM), which processes the propagation more finely by adjusting the affinity weights, diffusion paths, and the number of neighbors. Specifically, we first generate adaptive weighting (AW) matrices by decoupling the neighborhood into parts with respect to different distances. Independent attention maps are recursively applied to refine the weight value. Furthermore, a dynamic path (DP) strategy is adopted to unfreeze the links of the neighborhood for learning variable connections. The solution space of the paths is also constrained by a propagation decay loss to keep the results stable. Finally, we introduce a diffusion suppression (DS) operation, which preserves the edge of dense depth maps by manipulating the AW and DP strategies to decelerate and terminate the propagation. In our experiment, the proposed method requires fewer iterations and neighbors than other SPNs while yielding better results. DySPN outperforms state-of-the-art (SoTA) methods on the KITtI DC, NYU Depth v2, and VOID datasets. Our code is available at: https://github.com/Kyakaka/DySPN.

Localization in a large-scale three-dimensional scene is a key challenge faced by climbing robots on large workpieces. This article proposes a global localization method for climbing robots based on tether displacement sensor, visual-inertial odometry (VIO), and computer-aided design (CAD) model of workpieces. Tether displacement sensor measures the distance between robot and tether anchor with little drift, which enables robot to acquire global pose. Adsorption constraints on robot motion are extracted using CAD model to reduce the drift of VIO. The approach realizes global localization with high accuracy for the robots when climbing on large workpieces without other external locating equipment. The performance is verified with a prototype of climbing robot testing on real large workpieces. In all experiments, our method with adsorption constraints outperforms existing VIO. The largest drift of merged trajectory is as low as 0.51% in global localization on a wind turbine blade with length of 8 m.

In this article, we propose an RFID-based simultaneous localization and mapping (RF-SLAM) method that allows us, for the first time, to estimate the robot's position and the tags’ 3D position in the warehouse environment simultaneously without any reference tags and external sensors, using only COTS RFID device. RF-SLAM is designed to transform the RFID measurement into the relative tag position constraint and use a corresponding graph based model to solve the SLAM problem. Specifically, a multiantenna-based relative localization method using phase measurement and odometer data in a short time is proposed as the front end. The back end is a novel graph model based on the relative tags position constraint and odometer constraint. Experiments in different types of warehouses show the localization accuracy of robot and tags’ 3D position is about 5 cm and 10 cm, respectively. The experimental results in a more challenging and actual environment are still competitive.

Defect detection on textured surfaces remains a challenging task due to the wide range of textures and defects. Current unsupervised learning-based texture defect detection methods based on texture background reconstruction cannot detect texture defects with high precision because it is difficult to guarantee a high-precision reconstruction of the texture background while suppressing the defect foreground. In this study, we propose a novel semantic information decomposition network (SIDN) for accurate texture defect segmentation. The SIDN is trained on artificial defective images produced by a defect generation module (DGM). First, the SIDN uses a feature extraction module (FEM) to extract latent features with both texture semantic information and defect semantic information. Then, a novel feature separation extraction module (FSEM) for decomposing the texture semantic information and defect semantic information from the feature map generated by the FEM is proposed, preventing the coupling of the texture and defect semantic information from affecting the final segmentation accuracy. Next, a novel global semantic relation module (GSRM) is proposed to determine the relevance of the global semantic information to comprehensively consider the context and improve the feature representation. Finally, a segmentation module (SM) that directly segments the textures and defects instead of reconstructing the texture background is proposed. The final detection result is obtained by calculating a weighted average of the texture and defect segmentation results. The extensive experimental tests with the most popular and most challenging texture defect dataset demonstrate that the SIDN achieves accurate segmentation of various texture defects without using real defect samples.

A novel radio frequency identification (RFID)-based mobile robot global localization method combining two kinds of RFID signal information, i.e., phase difference and readability, is proposed. Specifically, a phase difference model and a classification logic strategy based on readability are built and integrated into a particle filter localization algorithm. Compared with existing RFID localization methods, the proposed localization method can achieve competitive localization performance in an environment with a relatively sparse reference tag distribution and without the need for offline phase drift calibration. A series of real experimental tests were performed, and the results show that the proposed method can localize a mobile robot with centimeter-level position accuracy and satisfactory attitude angle accuracy when the distance between adjacent reference tags is approximately 60 cm, even if all RFID devices are commercial off-the-shelf (COTS). The proposed method provides a promising option for mobile robot localization applications, such as path tracking of mobile robots.Note to Practitioners—Mobile robot localization is a key technology for its location-based services. Considering that radio frequency identification (RFID) is entirely unaffected by light interference and has a globally unique ID, RFID has been regarded as a localization sensor with broad application prospects. This article proposes an RFID-based mobile robot global localization method combining phase difference and readability, by which the mobile robot can be accurately localized in an environment with a relatively sparse reference tag distribution and without the need for offline phase drift calibration. The experimental results indicate that the proposed method can localize the mobile robot with good performance, including centimeter-level position accuracy and satisfactory attitude angle accuracy. The proposed method can effectively contribute to many practical applications, such as the path tracking of a mobile robot.

This article proposes a standalone radio frequency identification (RFID)-based mobile robot navigation method, in which a mobile robot equipped with reader antennas can be continuously guided to a static object marked with a single passive UHF RFID tag. An observation model based on the RFID phase difference is built and integrated into a particle filter, by which the instantaneous relative position between the mobile robot and the tagged object can be detected in real time. Based on the position information extracted from the RFID system, the mobile robot adjusts its pose to move toward the RFID-tagged object. Compared with the existing RFID-based mobile robot navigation methods, the proposed method requires no external sensors other than the RFID and requires only a single passive tag. Experiments using commercial off-the-shelf (COTS) RFID devices are performed, and the results indicate that the mobile robot can satisfactorily realize navigation task with a distance accuracy of 4.04 cm and a bearing accuracy of 2.23°. The proposed method is well applicable for the navigation scenes in which the absolute position of the tagged target object is not known beforehand.Note to Practitioners—UHF radio frequency identification (RFID) has been widely applied as an asset management ID sensor in many fields. RFID-based mobile robot navigation technology can further increase its application value as a location sensor. This article proposes a standalone RFID-based mobile robot navigation method, in which the reference tag and the external sensors other than RFID are both not required. In the proposed method, only a single passive tag is attached to the static target object. Experimental results indicate that the proposed method can enable navigation task with good performance in situations in which the absolute navigation goal position is not known in advance.

Texture recognition remains a challenging visual task due to the complex appearance variations caused by scale changes in the real world. In most existing texture recognition methods, textures are represented at a single scale; thus, multi-scale texture information is not fully utilized, resulting in insufficient representation and inaccurate recognition. In this study, with the goal of addressing the challenge of scale changes, we propose a novel multi-scale boosting feature encoding network (MSBFEN) for accurate texture recognition. MSBFEN first extracts multi-scale features with multi-scale texture structure information under the guidance of texture priors using a novel prior-guided feature extraction (PFE) method. Then, a multi-scale texture encoding (MSTE) method is devised to capture discriminative multi-scale texture representations by encoding the extracted features. Finally, to fully utilize the multi-scale texture representations for accurate texture recognition, a novel multi-scale boosting learning (MSBL) method is proposed. In MSBL, the learning procedure for multi-scale texture recognition is boosted in a hierarchical, progressively reinforced manner, significantly addressing the challenge of scale changes and greatly enhancing the recognition accuracy. In addition, a novel outlier-aware texture encoding (OTE) method is proposed for robust texture encoding at each scale of MSTE. OTE can resist the influence of background interference and can further enhance the robustness of MSBFEN. In extensive experiments conducted on six challenging texture recognition datasets, namely, KTH-TIPS2b, FMD, DTD, MINC, GTOS and GTOS-mobile, MSBFEN achieves accuracies of 86.2%, 86.4%, 77.8%, 85.3%, 86.4% and 87.57%, respectively, representing state-of-the-art texture recognition performance.

Establishing a unified model for the defect inspection of different texture surfaces remains a challenge in the industrial automation field because these surfaces can vary in regular and irregular ways. Current unsupervised learning methods are trained on defect-free samples only and cannot directly address anomalies during testing, which precludes these methods from simultaneously inspecting for various texture defects. In this article, we propose a novel unsupervised anomaly feature-editing-based adversarial network (AFEAN) to accurately inspect various texture defects. To impart the AFEAN with the ability to address anomalies, a paired input, consisting of a defect-free image and an artificially defective image, is utilized for training. First, the AFEAN employs a feature extraction module (FEM) to extract latent features for the paired input. Subsequently, a novel anomaly feature detection module (AFDM) is proposed to detect anomaly features of the artificially defective image in the latent space. In the proposed AFDM, a novel central-constraint-based clustering method is proposed to detect anomaly features by learning the distribution of the latent features. Next, a novel global context feature editing module (GCFEM) is proposed to convert the detected anomaly features to normal features to suppress the reconstruction of defects. Finally, a feature decoding module (FDM) utilizes the edited features to reconstruct the texture background. Through the AFDM and GCFEM, the AFEAN achieves the ability to address anomaly features, effectively suppressing the reconstruction of defects on the texture background. In addition, to further improve the texture reconstruction accuracy, a pixel-level discrimination module (PDM) is employed to reconstruct texture details. In the testing phase, the defects are segmented by the residual image between the input image and the reconstructed texture background. The extensive experimental results demonstrate that the AFEAN achieves the state-of-the-art inspection accuracy.

Most face recognition methods employ single-bit binary descriptors for face representation. The information from these methods is lost in the process of quantization from real-valued descriptors to binary descriptors, which greatly limits their robustness for face recognition. In this study, we propose a novel weighted feature histogram (WFH) method of multi-scale local patches using multi-bit binary descriptors for face recognition. First, to obtain multi-scale information of the face image, the local patches are extracted using a multi-scale local patch generation (MSLPG) method. Second, with the goal of reducing the quantization information loss of binary descriptors, a novel multi-bit local binary descriptor learning (MBLBDL) method is proposed to extract multi-bit local binary descriptors (MBLBDs). In MBLBDL, a learned mapping matrix and novel multi-bit coding rules are employed to project pixel difference vectors (PDVs) into the MBLBDs in each local patch. Finally, a novel robust weight learning (RWL) method is proposed to learn a set of robust weights for each patch to integrate the MBLBDs into the final face representation. In RWL, a codebook is first constructed by clustering MBLBDs on each local patch to extract a feature histogram. Then, considering that different parts of the face have different degrees of robustness to local changes, a set of weights is learned to concatenate the feature histograms of all local patches into the final representation of a face image. In addition, to further improve the performance for heterogeneous face recognition, a coupled WFH (C-WFH) method is proposed. C-WFH maintains the similarity of the corresponding MBLBDs and feature histograms for a pair of heterogeneous face images by means of a novel coupled feature learning (CFL) method to reduce the modality gap. A series of experiments are conducted on widely used face datasets to analyze the performance of WFH and C-WFH. Extensive experimental results show that WFH and C-WFH outperform state-of-the-art face recognition methods.

In order to improve the robustness and optimize trajectory of projective homography-based uncalibrated visual servoing (PHUVS) proposed in our previous work, an analytical expression of optimal trajectory for camera in projective homography space is proposed in this article, which is totally free of camera parameters and is corresponding to camera's shortest path in the 3-D space with straight path in translation and minimal geodesic in rotation. The projective homography is computed without scale ambiguity in both planning and tracking stages. The PHUVS controller is modified correspondingly to track the planned trajectory in projective homography space while maintaining superior characteristic of PHUVS under uncalibrated scenario. The simulations and experiments' results reveal the effectiveness and necessity of the proposed trajectory optimization method in the existence of large initial errors. Note to Practitioners-The state-of-the-art visual-guided robotic technology in industry usually requires system calibration, which is often costly, vulnerable, and challenging for ordinary workers. In our previous work, we offered an uncalibrated visual servo method based on projective homography named projective homography-based uncalibrated visual servoing (PHUVS), which is suitable for plug and play application for eye-in-hand robot visual servo tasks. However, PHUVS suffers from some defects, including undesirable 3-D space motion and local convergence. In this article, we proposed the trajectory planning method along with a modified PHUVS controller to improve the original one from the disadvantages mentioned earlier. This planning method is also calibration-free. With pure image information, a straight-line path in translational motion along with minimal geodesic in rotary motion can be achieved. This approach is capable of extending the range of applications for uncalibrated visual servo technology in robotic tasks, such as assembling, painting, and robotic machining.

A novel ultrahigh-frequency (UHF) radio frequency identification (RFID) localization method is proposed in this paper, by which the location of a static passive tag can be easily obtained using a mobile RFID antenna. An unwrapped phase-position model with three parameters is built, and the location of the tag can be pinpointed through an ordinary nonlinear least-squares algorithm. The main advantage of this method is that it is cheap in computation cost compared with the existing grid-based methods. The experimental tests confirm that the proposed method can localize the RFID tags with a competitive computational efficiency and accuracy performance, i.e., millisecond-level computing time and centimeter-level location accuracy. The proposed UHF RFID localization method is well suited to the pervasive location-aware applications, searching RFID-tagged item in the intelligent warehouse by the mobile robot with an onboard RFID system, for example.

Free-form surface matching that aligns measured points with a design model is a common problem in manufacturing automation. In this paper, an iterative variance-minimization matching (VMM) method is proposed to address measured points that have measuring defects, such as uneven/open point distributions and measuring noise. The basic idea is that the objective function is defined as the variance of the closest distance from each measured point to the design model, and the measuring defects are considered by incorporating an average distance item into the objective function. Using the defined average distance item, a strategy for analyzing the effect of measuring defects on VMM and existing methods is presented. It is shown that the VMM method does not easily become trapped in a local optimum when measuring defects exist. To consider convergence speed and convergence stability, a new distance based on the first-order point-to-point distance and point-to-tangent distance is developed and used in the objective function. To demonstrate the availability of the proposed method, quadratic convergence and positive definiteness are theoretically analyzed. The proposed method is efficient and insensitive to measuring defects and is useful for shape matching tasks involving free-form surface features. Note to Practitioners-This paper is motivated by the problem of matching measured points with a design model to automate manufacturing processes such as geometric inspection, workpiece localization, and allowance distribution. Measured points are obtained by applying a scanning device where measuring defects usually appear. Existing matching methods suffer from the drawback that the measured points may incline toward dense data and become trapped in a local optimum, due to measuring defects. To address this practical issue, this paper proposes a new method called variance-minimization matching (VMM), in which the objective function is optimized to weaken the effect of measuring defects. By examining the differences between VMM and existing methods, it is found that VMM can achieve quadratic convergence speed. Most importantly, the method is insensitive to uneven/open point distributions. In summary: 1) this method allows us to improve the matching accuracy in the presence of measuring defects; 2) there is no need to obtain a high-quality scan of the entire workpiece, potentially reducing scanning difficulty and improving scanning efficiency; and 3) the requirement of uniform sampling for measured points is reduced.

In the first part of this paper, a free-form surface matching method called variance-minimization matching (VMM) was proposed to address uneven/open point distributions and measuring noise. The convergence property and sensitivity to measuring defects were theoretically studied. In the second part of this paper, a series of experiments are presented to verify the feasibility of the proposed method in free-form surface matching. The experiments are divided into four sets: a measuring defects experiment, a noise experiment, a convergence experiment, and an artificial experiment. In the first set of experiments, the existing methods are prone to becoming trapped in a local optimum affected by uneven/open point distributions, which shows that measured points incline toward dense areas. However, in VMM, there is little inclination regardless of the increase in the number of measuring defects. In the second set of experiments, sensitivity to varying noise is tested. The results show that VMM helps prevent unstable sliding in the presence of Gaussian noise. In the third set of experiments, we compare convergence speed and convergence stability under different initial positions. It is verified that VMM exhibits the quadratic convergence. Finally, a set of artificial experiments is implemented, revealing that the proposed method is appropriate for use in automated manufacturing processes such as geometric inspection and allowance distribution. Note to Practitioners-Measuring defects usually occur when using a scanning device to obtain the measured points of a workpiece. Weakening the effect of measuring defects on matching results is critical to promoting manufacturing automation. This paper proposes a new method called variance-minimization matching (VMM) that considers measuring defects. In the first part of this paper, the modeling and theoretical analysis of VMM were introduced. In the second part of this paper, simulated experiments are performed to verify the feasibility of VMM in addressing uneven/open point distributions, measuring noise, and large initial positions. Next, artificial experiments employing VMM in geometric inspection and allowance distribution are presented. The proposed method also applies to other automated manufacturing processes, such as workpiece localization, deformation analysis, and complex parts repair.

Visual inspection of texture surface defects is still a challenging task in the industrial automation field due to the tremendous changes in the appearance of various surface textures. Current visual inspection methods cannot simultaneously and efficiently inspect various types of texture defects due to either the low discriminative capabilities of handcrafted features or their time-consuming sliding-window strategy. In this paper, we present a novel unsupervised multiscale feature-clustering-based fully convolutional autoencoder (MS-FCAE) method that efficiently and accurately inspects various types of texture defects based on a small number of defect-free texture samples. The proposed MS-FCAE method utilizes multiple FCAE subnetworks at different scale levels to reconstruct several textured background images. The residual images are obtained by subtracting these texture backgrounds from the input image individually; then, they are fused into one defect image. To maximize the efficiency, each FCAE subnetwork utilizes fully convolutional neural networks to extract the original feature maps directly from the input images. Meanwhile, each FCAE subnetwork performs feature clustering to improve the discriminant power of the encoded feature maps. The proposed MS-FCAE method is evaluated on several texture surface inspection data sets both qualitatively and quantitatively. This method achieves a Precision of 92.0% while requiring only 82 ms for input images of $1920\times 1080$ pixels. The extensive experimental results demonstrate that MS-FCAE achieves highly efficient and state-of-the-art inspection accuracy. Note to Practitioners-Most conventional visual inspection methods can address only one specific type of texture defect, while multiscale feature-clustering-based fully convolutional autoencoder (MS-FCAE) can simultaneously and accurately inspect various types of texture surface defects, such as those of thin-film transistor liquid crystal displays, wood, fabrics, and ceramic tiles. Furthermore, MS-FCAE requires only a small number of surface texture samples to learn a robust network model, and its training requires no defect samples. This is extremely important for industrial applications because identifying and labeling defect samples is difficult. Moreover, MS-FCAE can be applied to online visual inspection utilizing a graphics processing unit-based parallel processing strategy.

Object detection remains a challenging task in computer vision due to the tremendous extent of changes in the appearances of objects caused by clustered backgrounds, occlusion, truncation, and scale change. Current deep neural network (DNN)-based object detection methods cannot simultaneously achieve a high accuracy and a high efficiency. To overcome this limitation, in this paper, we propose a novel multi-scale attention (MSA) DNN for accurate object detection with high efficiency. The proposed MSA-DNN method utilizes a novel multi-scale feature fusion module (MSFFM) to construct high-level semantic features. Subsequently, a novel MSA module (MSAM) based on the fused layers of the MSFFM is introduced to exploit the global semantic information of image-level labels to guide detection. On the one hand, MSAM can capture global semantic information to further enhance the semantic feature representation of the fused layers constructed by the MSFFM, thereby improving the detection accuracy. On the other hand, the MSA maps generated by MSAM can be employed to rapidly and coarsely locate objects at different scales. In addition, an attention-based hard negative mining strategy is introduced to filter out negative samples to reduce the search space, dramatically alleviating the severe class imbalance problem. Extensive experimental results on the challenging PASCAL VOC 2007, PASCAL VOC 2012, and MS COCO datasets demonstrate that MSA-DNN achieves a state-of-the-art detection accuracy while maintaining a high efficiency. Furthermore, MSA-DNN significantly improves the small-object detection accuracy.

Almost all conventional template-matching methods employ low-level image features to measure the similarity between a template image and a scene image using similarity measures such as pixel intensity and pixel gradient. Although these methods have been widely used in many applications, they cannot simultaneously address all types of robustness challenges. In this study, with the goal of simultaneously addressing the various challenges, we present a robust semantic template-matching approach (RSTM). Inspired by the local binary descriptor, we propose a novel superpixel region binary descriptor (SRBD) to construct a multilevel semantic fusion feature vector for RSTM. SRBD uses a new kernel-distance-based simple linear iterative clustering (KD-SLIC) method to extract the stable superpixels from the template image; Then, based on the average intensity difference between each superpixel region and its neighbors, the dominant gradient orientation of each superpixel can be obtained, and the semantic features of each superpixel can be described as the dominant orientation difference vector, which is coded as the rotation-invariant SRBD. In the off-line matching phase, the fusion semantic feature vector of RSTM combines the multilevel SRBD features with different numbers of superpixels. In the online matching phase, to cope with rotation invariance, a marginal probability model is proposed and applied to locate the positions of template images in the scene image. Moreover, to accelerate computation, an image pyramid is employed. We conduct a series of experiments on a large dataset randomly selected from the MS COCO dataset to fully analyze the robustness of this approach. The experimental results show that RSTM simultaneously addresses rotation changes, scale changes, noise, occlusions, blur, nonlinear illumination changes and deformation with high time efficiency while also outperforming previous stateof- the-art template-matching methods.

An uncalibrated visual servo method based on projective homography, denoted as Projective Homography based Uncalibrated Visual Servoing (PHUVS), is proposed in this paper, in which a novel task function based on the element of projective homography is devised to realize visual servo without a prior knowledge of the camera intrinsic parameters and hand-eye relationships. The main advantage of this method is that it is not only suitable for totally uncalibrated scenarios but also cheap in computation costs when compared with classical image-based uncalibrated visual servoing methods. Numerical experiments are performed and the results confirm that the new approach is capable of both static positioning and dynamic tracking tasks, and presents competitive computational efficiency and accuracy performance.

The visual inspection of Mura defects is still a challenging task in the quality control of panel displays because of the intrinsically nonuniform brightness and blurry contours of these defects. The current methods cannot detect all Mura defect types simultaneously, especially small defects. In this paper, we introduce an accurate Mura defect visual inspection (AMVI) method for the fast simultaneous inspection of various Mura defect types. The method consists of two parts: an outlier-prejudging-based image background construction (OPBC) algorithm is proposed to quickly reduce the influence of image backgrounds with uneven brightness and to coarsely estimate the candidate regions of Mura defects. Then, a novel region-gradient-based level set (RGLS) algorithm is applied only to these candidate regions to quickly and accurately segment the contours of the Mura defects. To demonstrate the performance of AMVI, several experiments are conducted to compare AMVI with other popular visual inspection methods are conducted. The experimental results show that AMVI tends to achieve better inspection performance and can quickly and accurately inspect a greater number of Mura defect types, especially for small and large Mura defects with uneven backlight.

Features are crucial for extensive facial landmark localization (EFLL), while deep convolutional neural network (DCNN) features lead to breakthroughs in diverse visual recognition tasks. However, there is little study of DCNN features for EFLL, mainly because less labeled data with extensive facial landmarks are available. In this letter, we employ transfer learning to overcome this limitation, and utilize DCNN for EFLL. We concentrate the power of DCNN on feature learning within a cascaded-regression framework (CRF). We present three transfer methods, which show the capacity of DCNN as a generic feature extractor, and the benefit of fine-tuning. The proposed specific fine-tuning method for cascaded regression, named cascade transfer, achieves competitive accuracy with state-of-the-art methods on the 300-W challenge dataset.

The generalized Hough transform (GHT) is widely used for detecting or locating objects under similarity transformation. However, a weakness of the traditional GHT is its large storage requirement and time-consuming computational complexity due to the 4-D parameter space voting strategy. In this paper, a polygon-invariant GHT (PI-GHT) algorithm, as a novel scale- and rotation-invariant template matching method, is presented for high-speed object vision-based positioning. To demonstrate the performance of PI-GHT, several experiments were carried out to compare this novel algorithm with the other five popular matching methods. Experimental results show that the computational effort required by PI-GHT is smaller than that of the common methods due to the similarity transformations applied to the scale- and rotation-invariant triangle features. Moreover, the proposed PI-GHT maintains inherent robustness against partial occlusion, noise, and nonlinear illumination changes, because the local triangle features are based on the gradient directions of edge points. Consequently, PI-GHT is implemented in packaging equipment for radio frequency identification devices at an average time of 4.13 ms and 97.06% matching rate, to solder paste printing at average time nearly 5 ms with 99.87%. PI-GHT is applied to LED manufacturing equipment to locate multiobjects at least five times improvement in speed with a 96% matching rate.

Visually-guided robot grinding is a novel and promising automation technique for blade manufacturing. One common problem encountered in robot grinding is hand-eye calibration, which establishes the pose relationship between the end effector (hand) and the scanning sensor (eye). This paper proposes a new calibration approach for robot belt grinding. The main contribution of this paper is its consideration of both joint parameter errors and pose parameter errors in a hand-eye calibration equation. The objective function of the hand-eye calibration is built and solved, from which 30 compensated values (corresponding to 24 joint parameters and six pose parameters) are easily calculated in a closed solution. The proposed approach is economic and simple because only a criterion sphere is used to calculate the calibration parameters, avoiding the need for an expensive and complicated tracking process using a laser tracker. The effectiveness of this method is verified using a calibration experiment and a blade grinding experiment. The code used in this approach is attached in the Appendix.

Impulse noise detection is a critical issue when removing impulse noise and impulse/Gaussian mixed noise. In this paper, we propose a new detection mechanism for universal noise and a universal noise-filtering framework based on the nonlocal means (NL-means). The operation is carried out in two stages, i.e., detection followed by filtering. For detection, first, we propose the robust outlyingness ratio (ROR) for measuring how impulselike each pixel is, and then all the pixels are divided into four clusters according to the ROR values. Second, different decision rules are used to detect the impulse noise based on the absolute deviation to the median in each cluster. In order to make the detection results more accurate and more robust, the from-coarse-to-fine strategy and the iterative framework are used. In addition, the detection procedure consists of two stages, i.e., the coarse and fine detection stages. For filtering, the NL-means are extended to the impulse noise by introducing a reference image. Then, a universal denoising framework is proposed by combining the new detection mechanism with the NL-means (ROR-NLM). Finally, extensive simulation results show that the proposed noise detector is superior to most existing detectors, and the ROR-NLM produces excellent results and outperforms most existing filters for different noise models. Unlike most of the other impulse noise filters, the proposed ROR-NLM also achieves high peak signal-to-noise ratio and great image quality by efficiently removing impulse/Gaussian mixed noise.

A novel filter based on mathematical morphology for high probability impulse noise removal is presented. First, an impulse noise detector using mathematical residues is proposed to identify pixels that are contaminated by the salt or pepper noise. Then the image is restored using specialized open-close sequence algorithms that apply only to the noisy pixels. Finally, black and white blocks that degrade the quality of the image will be recovered by a block smart erase method. Experimental results demonstrate that the proposed filter outperforms a number of existing algorithms and is particularly effective for the very highly corrupted images

To realize the fast and accurate detection of the edges from the blurry images, the fast multilevel fuzzy edge detection (FMFED) algorithm is proposed. The FMFED algorithm first enhances the image contrast by means of the fast multilevel fuzzy enhancement (FMFE) algorithm using the simple transformation function based on two image thresholds. Second, the edges are extracted from the enhanced image by the two-stage edge detection operator that identifies the edge candidates based on the local characteristics of the image and then determines the true edge pixels using the edge detection operator based on the extremum of the gradient values. Experimental results demonstrate that the FMFED algorithm can extract the thin edges and remove the false edges from the image, which leads to its better performance than the Sobel operator, Canny operator, traditional fuzzy edge detection algorithm, and other multilevel fuzzy edge detection algorithms