When applying template matching to the robot arm grasping of rotated objects with high aspect ratios, the accuracy of the matching process is often degraded by the occurrence of high similarity scores with pixels or patches located in neighboring objects. Accordingly, we propose a learning-based template matching (LbTM) algorithm in which the accuracy of the matching results is improved by clearing the matching scores of these confused pixels or patches to zero using a spatial clustering process. This algorithm consists of two modules: First, the translation matching module uses the learning-based pairwise similarity matrix. Having determined the center coordinate of the target object, the second module is applied to estimate the target rotation angle by using a Siamese network. The effectiveness of the proposed algorithm is evaluated for 600 template-rotated target pairs. It is shown that the area under curve (AUC) performance of the proposed algorithm (0.678) is higher than that of three other template matching algorithms (DDIS, CoTM, and QATM). Moreover, our algorithm achieves a minimum success rate of 80% in practical grasping trials performed even using high-aspect objects with various rotational angles and positions.

To increase an industrial production rate in a robotic operation, automated assembly tasks on production lines require the operation being performed while a target is still moving. Due to complicated factory environments and missions, unexpected vision occlusion, switching grippers, and grasping the target in motion are demanded. Therefore, a vision guided robotic system consisting of a joint space adaptive controller and a visual target motion estimator is proposed. With the visual target motion estimator, current target velocities and positions are estimated. Regarding the current target velocities and positions, desired robotic joint velocities and positions are computed even if vision is occluded. Moreover, by tracking the desired joint velocities and positions with the adaptive controller, a robotic end-effector is controlled to catch the target even if target and gripper parameters are uncertain. In experiments, grasping the moving target at different speeds with the different grippers for different lengths of occlusion time by using our proposed system were presented for performance evaluations according to its efficiency and robustness.

An automated assembly task on production lines requires robotic manipulation of a target when the target is moving. Due to complicated environments and multi-tasking, vision occlusion and uncertainty of dynamics of a gripper/target may happen. Therefore, we developed a vision guided robotic system composed of a target motion estimation-based visual tracking controller and an adaptive controller. The target motion estimation-based visual tracking controller enables the system to estimate target positions and velocities and compute desired robotic positions and velocities based on previous estimates even if the target is not viewed occasionally. Moreover, a robotic end-effector is controlled with the adaptive controller to pursue and catch the target even if the target and gripper parameters are not exactly known. In experiments, interceptions of the target moving at different speeds by using different grippers based on visual feedback were presented to demonstrate good performance on efficiency and robustness of the proposed method.

Automatically synthesizing the facial sketches of a facial image is highly challenging since facial images typically exhibit a wide range of poses, expressions and scales, and have differing degrees of illumination and/or occlusion. When the facial sketches are to be synthesized in the unique sketching style of a particular artist, the problem becomes even more complex. This study develops an automatic facial sketch synthesis system based on a novel direct combined model (DCM) algorithm carrying three major advantages: First, DCM approach takes account of both the local details of each facial feature and the global geometric structure of the face, and thus the synthesized sketches more accurately mimic the caricatures drawn by the artist. Second, although the training database contains only full-frontal facial images with a neutral expression, sketches with a wide variety of facial poses, gaze directions and facial expressions can be successfully synthesized. Third, previous synthesizing proposals are heavily reliant on the quality of the texture reconstruction results, which in turn are highly sensitive to occlusion and lighting effects in the input image. DCM approach accurately produces lifelike synthesized facial sketches without the need to restore the texture information lost as a result of such unfavorable conditions.

Existing tone reproduction schemes are generally based on a single image and are, therefore, unable to accurately recover the local details and colors of scene since the limited available information. Accordingly, the proposed tone reproduction system utilizes two images with different exposures (one low and one high) to capture the local detail and color information of low- and high-luminance regions of scene, respectively. The adaptive local region of each pixel is developed in order to appropriately reveal the details and maintain the overall impression of scene. Our system implements the local tone mapping and color mapping based on the adaptive local region by taking the lowly-exposed image as the basis and referencing the information of highly-exposed image. The local tone mapping compresses the luminance range in the image and enhances the local contrast to reveal the details, while the local color mapping maps the precise color information from the highly-exposed image to the lowly-exposed image. Finally, a fusion process is proposed to mix the local tone mapping and local color mapping results to produce the output image. A multiresolution approach is also developed to reduce time cost. The experimental results confirm that the system generates realistic reproductions of HDR scenes.

Automatically locating multiple feature points (i.e., the shape) in a facial image and then synthesizing the corresponding facial sketch are highly challenging since facial images typically exhibit a wide range of poses, expressions, and scales, and have differing degrees of illumination and/or occlusion. When the facial sketches are to be synthesized in the unique sketching style of a particular artist, the problem becomes even more complex. To resolve these problems, this paper develops an automatic facial sketch synthesis system based on a novel direct combined model (DCM) algorithm. The proposed system executes three cascaded procedures, namely, 1) synthesis of the facial shape from the input texture information (i.e., the facial image); 2) synthesis of the exaggerated facial shape from the synthesized facial shape; and 3) synthesis of a sketch from the original input image and the synthesized exaggerated shape. Previous proposals for reconstructing facial shapes and synthesizing the corresponding facial sketches are heavily reliant on the quality of the texture reconstruction results, which, in turn, are highly sensitive to occlusion and lighting effects in the input image. However, the DCM approach proposed in this paper accurately reconstructs the facial shape and then produces lifelike synthesized facial sketches without the need to recover occluded feature points or to restore the texture information lost as a result of unfavorable lighting conditions. Moreover, the DCM approach is capable of synthesizing facial sketches from input images with a wide variety of facial poses, gaze directions, and facial expressions even when such images are not included within the original training data set.

This study presents a novel learning-based pedestrian detection system capable of automatically detecting individuals of different sizes and orientations against a wide variety of backgrounds, including crowds, even when the individual is partially occluded. To render the detection performance robust toward the effects of geometric and rotational variations in the original image, the feature extraction process is performed using both rectangular- and circular-type blocks of various sizes and aspect ratios. The extracted blocks are rotated in accordance with their dominant orientation(s) such that all the blocks extracted from the input images are rotationally invariant. The pixels within the cells in each block are then voted into rectangular- and circular-type 9-bin histograms of oriented gradients (HOGs) in accordance with their gradient magnitudes and corresponding multivariate Gaussian-weighted windows. Finally, four cell-based histograms are concatenated using a tri-linear interpolation technique to form one 36-dimensional normalized HOG feature vector for each block. The experimental results show that the use of the Gaussian-weighted window approach and tri-linear interpolation technique in constructing the HOG feature vectors improves the detection performance from 91% to 94.5%. In the proposed scheme, the detection process is performed using a cascaded detector structure in which the weak classifiers and corresponding weights of each stage are established using the AdaBoost self-learning algorithm. The experimental results reveal that the cascaded structure not only provides a better detection performance than many of the schemes presented in the literature, but also achieves a significant reduction in the computational time required to classify each input image.

This study presents a rapid image completion system comprising a training (or analysis) process and an image completion (or synthesis) process. The proposed system adopts a multiresolution approach, which not only improves the convergence rate of the synthesis process, but also provides the ability to deal with large replaced regions. In the training process, a down-sampling approach is applied to create a patch-based texture eigenspace based on multiresolution background region information. In the image completion process, an up-sampling approach is applied to synthesize the replaced foreground regions. To ensure the continuity of the geometric texture structure between the original background scene regions and the replaced foreground regions, directional and nondirectional image completion approaches are developed to reconstruct the global geometric structure and to enhance the local detailed features of the replaced foreground regions in the lower and higher resolution level images, respectively. Moreover, the synthesis priority order of the individual patches and the appropriate choice of completion scheme (i.e., directional or nondirectional) are both determined in accordance with a Hessian matrix decision value (HMDV) parameter. Finally, a texture refinement process is performed to optimize the resolution of the synthesized result.

This paper develops a novel statistical approach for automatic vehicle detection based on local features that are located within three significant subregions of the image. In the detection process, each subregion is projected onto its associated eigenspace and independent basis space to generate a principal components analysis (PCA) weight vector and an independent component analysis (ICA) coefficient vector, respectively. A likelihood evaluation process is then performed based on the estimated joint probability of the projection weight vectors and the coefficient vectors of the subregions with position information. The use of subregion position information minimizes the risk of false acceptances, whereas the use of PCA to model the low-frequency components of the eigenspace and ICA to model the high-frequency components of the residual space improves the tolerance of the detection process toward variations in the illumination conditions and vehicle pose. The use of local features not only renders the system more robust toward partial occlusions but also reduces the computational overhead. The computational costs are further reduced by eliminating the requirement for an ICA residual image reconstruction process and by computing the likelihood probability using a weighted Gaussian mixture model, whose parameters and weights are iteratively estimated using an expectation-maximization algorithm.

We have developed a computer vision system, including both facial feature extraction and recognition, that automatically discriminates among subtly different facial expressions. Expression classification is based on Facial Action Coding System (FACS) action units (AUs), and discrimination is performed using Hidden Markov Models (HMMs). Three methods are developed to extract facial expression information for automatic recognition. The first method is facial feature point tracking using a coarse-to-fine pyramid method. This method is sensitive to subtle feature motion and is capable of handling large displacements with sub-pixel accuracy. The second method is dense flow tracking together with principal component analysis (PCA) where the entire facial motion information per frame is compressed to a low-dimensional weight vector. The third method is high gradient component (i.e., furrow) analysis in the spatio-temporal domain, which exploits the transient variation associated with the facial expression. Upon extraction of the facial information, non-rigid facial expression is separated from the rigid head motion component, and the face images are automatically aligned and normalized using an affine transformation. This system also provides expression intensity estimation, which has significant effect on the actual meaning of the expression.