The use of neural networks for processing X-ray detector data has become a key development trend in high-energy physics and medical imaging. As detector arrays grow, handling the resulting massive data volumes poses significant challenges in terms of system bandwidth and power consumption. To address the bandwidth bottleneck and high analog-to-digital conversion (ADC) power consumption in conventional X-ray imaging systems, we propose an in-sensor computing architecture, SpikeX, for accelerating spiking neural networks (SNN). First, a photon-counting analog front end is directly integrated with pixel-level SNN neurons, enabling the first convolutional layer to be computed using spike signals. This eliminates the need for analog-to-digital conversion, significantly reducing power and bandwidth requirements. Next, an event-driven on-chip SNN accelerator performs efficient temporal inference, enabling an end-to-end pipeline from signal acquisition to image classification. Furthermore, detector non-idealities can be absorbed by the SNN model during training. To systematically investigate these effects, we develop a signal processing algorithm, enabling effective correction of detector non-idealities. To validate the proposed architecture, we implemented the SpikeX using a 180 nm CMOS process, featuring a$28\times 28$pixel array and 32 Leaky Integrate-and-Fire (LIF) processing units. Simulation results show that, at a clock frequency of 50 MHz, the architecture achieves an energy efficiency of 1.88 TOPS/W, significantly outperforming comparable state-of-the-art designs.

In an automatic control system, multiple peg-in-hole (MPIH) assembly of a relay and its socket is often required. However, the existing hybrid position/force control (HPFC) methods cannot meet the requirement for MPIH assembly because of its great complexity. A reinforcement learning (RL)–based HPFC method is presented for the assembly of a relay and its socket, whose controller's parameters are adjusted to adapt to the complexity. First, the socket is modeled as a spring based on its internal structural characteristics. The force in assembly simulation is computed with this model by combining it with their relative positions. Second, the proposed RL-based HPFC method segregates the assembly motion control into a force control branch and a position control branch. In the force control branch, a proportional-derivative controller regulates radial motion to obtain compliance. In the position control branch, a proportional controller manages the axial motion, which is further constrained by the radial force to ensure safety. Finally, a parameter adjustment method for the controller based on RL is further developed. The sliding window state collection is used for training. A series of comparative experiments in both simulation and real assembly environments are conducted. The experimental results validate the effectiveness of the proposed method.

The generalization model for multicategory peg-in-hole assembly (MPHA) skills is hard to acquire. An adaptive meta policy learning (AMPL) algorithm with virtual model set is proposed to deal with the difficulties of low learning efficiency and low adaptability for the multicategory assembly skill learning. First, the AMPL framework incorporates meta-reinforcement learning and can obtain generalization model of multicategory assembly skills. It has higher learning efficiency compared to single-category skill learning algorithms. Second, the AMPL algorithm introduces a similarity function constructed from the demonstration learning algorithm in the state value function. It has stronger adaptability compared to the multicategory skills learning algorithms. Finally, a simulation environment set for MPHA is constructed, consisting of the mathematical force contact models and the physical simulation models. The simulations and experiments are well conducted with the proposed algorithm. The results demonstrate the efficacy of the proposed algorithm.

Vision measurement is important for intelligent systems to obtain the precise structural and spatial information of objects. Beyond the object-specific vision measurement developed for fixed object type, it is appealing to explore the object-agnostic vision measurement, which can be efficiently reconfigured and adapted to various novel objects. This article proposes a framework to mimic the human's versatile visual measurement behavior: extract a set of contour primitives of interest (CPIs) from an image, then utilize the CPIs to calculate the key geometric information. First, a deep convolutional neural network (CNN) CPieNet+ is proposed under the one-shot learning scheme, aiming to extract the pixel-level object CPI from a raw query image, given an annotated support image. The fine-grained CPI prototypes are formed by sampling multiple points on the feature map of the support image. To leverage the explicit geometric knowledge in the CNN inference, the annotation map is encoded as a shape descriptor to guide the feature channel attention, and the geometric attribute awareness is realized by supervising the model to predict the direction and size of CPI. Second, the measurement behavior tree (BT) is designed to model the hierarchical geometric calculation procedure, which is flexibly configurable for different measurement requirements and is interpretable for nonexpert users. After the execution of the measurement BT, the pixel-level CPIs are converted to the required key geometric data. The effectiveness of the proposed methods is validated by a series of experiments.

Although many existing vision measurement systems have achieved high performances, they are object-specific and have limitations in flexibility. Toward intelligent vision measurement that can be conveniently reused for novel objects, this article focuses on the image geometric feature extraction with one-shot learning ability. We propose a contour primitive of interest (CPI) extraction network with dual metric (CPieNet-DM), which can obtain a designated CPI in a query image of a novel object under the guidance of only one annotated support image. First, the dual-metric learning mechanism is proposed, which not only utilizes inter-image similarity as guidance but also leverages the intra-image coherency of CPI pixels to facilitate the inference. Second, a neural network is designed to infer the CPI map based on the dual metric, which also predicts the CPI's geometric parameters. Moreover, the dual context aggregator is plugged in to provide the awareness of both images’ contexts. Third, the network training is jointly supervised by the multiple tasks of dual-metric learning, geometric parameters regression, and CPI extraction. The online hard example mining is utilized to improve the training outcome. The effectiveness of the proposed methods is validated with a series of experiments.

The force-based control algorithm of robotic multiple peg-in-hole assembly is a challenge. For the difficulty of low adaptability of model-based control algorithms and low learning efficiency of model-free control algorithms, a goal-based hierarchical policy learning (HPL) algorithm that combines conventional control algorithm and demonstration learning (DL) algorithm is proposed to learn the assembly skill. First, the goal-based HPL algorithm adds goal as a new variable to the action value function. Multiple states reached in each episode are randomly selected as subgoals to improve the distribution of positive rewards. Second, an initial policy that combines conventional control algorithm and DL algorithm is designed. The combined coefficient of these two algorithms is learned by HPL algorithm. Finally, a conical surface is used to compute the forces and moments of simplified assembly simulation model. Our algorithm is well implemented in both simulation and real-world environments. The experimental results verify the effectiveness of the proposed method.

Most recent semisupervised video object segmentation (VOS) methods rely on fine-tuning deep convolutional neural networks online using the given mask of the first frame or predicted masks of subsequent frames. However, the online fine-tuning process is usually time-consuming, limiting the practical use of such methods. We propose a directional deep embedding and appearance learning (DDEAL) method, which is free of the online fine-tuning process, for fast VOS. First, a global directional matching module (GDMM), which can be efficiently implemented by parallel convolutional operations, is proposed to learn a semantic pixel-wise embedding as an internal guidance. Second, an effective directional appearance model-based statistics is proposed to represent the target and background on a spherical embedding space for VOS. Equipped with the GDMM and the directional appearance model learning module, DDEAL learns static cues from the labeled first frame and dynamically updates cues of the subsequent frames for object segmentation. Our method exhibits the state-of-the-art VOS performance without using online fine-tuning. Specifically, it achieves a J & F mean score of 74.8% on DAVIS 2017 data set and an overall score G of 71.3% on the large-scale YouTube-VOS data set, while retaining a speed of 25 fps with a single NVIDIA TITAN Xp GPU. Furthermore, our faster version runs 31 fps with only a little accuracy loss.

Image contour based vision measurement is widely applied in robot manipulation and industrial automation. It is appealing to realize object-agnostic vision system, which can be conveniently reused for various types of objects. We propose the contour primitive of interest extraction network (CPieNet) based on the one-shot learning framework. First, CPieNet is featured by that its contour primitive of interest (CPI) output, a designated regular contour part lying on a specified object, provides the essential geometric information for vision measurement. Second, CPieNet has the one-shot learning ability, utilizing a support sample to assist the perception of the novel object. To realize lower-cost training, we generate support-query sample pairs from unpaired online public images, which cover a wide range of object categories. To obtain single-pixel wide contour for precise measurement, the Gabor-filters based non-maximum suppression is designed to thin the raw contour. For the novel CPI extraction task, we built the Object Contour Primitives dataset using online public images, and the Robotic Object Contour Measurement dataset using a camera mounted on a robot. The effectiveness of the proposed methods is validated by a series of experiments.

Multiple peg-in-hole insertion control is one of the challenging tasks in precision assembly for its complex contact dynamics. In this article, an insertion policy learning method is proposed for multiple peg-in-hole precision assembly. The insertion policy learning process is separated into two phases: initial policy learning and residual policy learning. In initial policy learning, a state-to-action policy mapping model based on the Gaussian mixture model (GMM) is established. And Gaussian mixture regression (GMR) is used to generalize the policy reuse. In residual policy learning, a reinforcement learning method named normalized advantage function (NAF) is employed to refine the insertion policy via agent's exploration in the insertion environment. Moreover, an adaptive action exploration (AAE) strategy is designed to improve the performance of exploration, and the prioritized experience replay strategy is introduced to make the residual policy learning from historical experience more efficient. Besides, the hierarchical reward function is designed considering the contact dynamics as well as the efficiency and safety of precision insertion. Finally, comprehensive experiments are conducted to validate the effectiveness of the proposed insertion policy learning method.

This article uses a multiple compliant degree-of-freedoms (DOFs) mechanism (i.e., a spring) to facilitate compliant insertion in precision assembly and proposes an efficient insertion strategy accordingly. The addition of a spring increases the insertion compliance while resulting in the object being not directly controllable. The proposed strategy handles both vertical insertion and inclined insertion with an unknown posture according to force feedback. We investigate horizontal compliance when the spring is compressed or stretched and introduce the withdrawal process for exceptional situations by taking advantage of the insertion compliance. The inclined insertion is a process of inserting while estimating the hole posture and a radial compensation strategy is presented while not affecting the axial length of the spring. Efficient insertion planning is discussed in the presence of uncertainty caused by the spring for both insertion types. Experiments are carried out to demonstrate the validation of the proposed method.

This article investigates simultaneous precision assembly of multiple objects, which plays a significant role in forming combined and complicated structures. Taking circular assembly as an example, we present a general alignment and insertion strategy, considering the difficulties of joint alignment as well as complicated interaction. In alignment, this article uses movable microscopic cameras to realize a large view area required by the spatial alignment and an optimization approach to achieve the even display of multiple objects. In insertion, we propose a wriggling insertion method to distinguish interaction forces, iteratively counting according to the relative movements of adjacent objects, and also an algorithm to plan the insertion: the compensational motion regarding the mutual effects of all objects and the toward-center motion based on radial forces. Experiments are carried out to demonstrate the validation of the proposed method.

Simultaneously inserting multiple objects is an essential topic in precision assembly to compose complicated shapes. This task involves the acquisition difficulty of unobservable interaction states, which makes it hard to plan the insertion. To solve it, this article investigates the circular assembly of multiple objects and proposes a strategy to control the simultaneous insertion in belief space. We first present the insertion state transition and observation models, in which the stochastic parts are modeled as Gaussian noise, and then estimate the belief state using an extended Kalman filter. An optimization approach is discussed, for the compensational movement planning, to decrease the estimated radial interaction forces and the toward-center movement is thus determined, considering the optimized compensational movement and the belief state. Experiments are carried out to demonstrate the validation of the proposed method.

The tracking-by-detection tracking framework usually consists of two stages: drawing samples around the target object and classifying each sample as either the target object or background. Current popular trackers under this framework typically draw many samples from the raw image and feed them into the deep neural networks, resulting in high computational burden and low tracking speed. In this article, we propose an adversarial feature sampling learning (AFSL) method to address this problem. A convolutional neural network is designed, which takes only one cropped image around the target object as input, and samples are collected from the feature maps with spatial bilinear resampling. To enrich the appearance variations of positive samples in the feature space, which has limited spatial resolution, we fuse the high-level features and low-level features to better describe the target by using a generative adversarial network. Extensive experiments on benchmark data sets demonstrate that the proposed ASFL achieves leading tracking accuracy while significantly accelerating the speed of tracking-by-detection trackers.

This article proposes an efficient insertion strategy for inclined precision assembly. According to the characteristics of the inclined insertion, the radial force between objects is separated into three parts: the force due to insertion, the force retained from previous compensation, and the force caused by estimation error of object inclination. We employ Gaussian distribution to model the probability of the radial force in insertion and predict the future contact based on the current force and the compensation movement. Object inclination is online estimated by peeling off the forces not related with inclination and investigating the relationship between inclination difference and radial force error. The insertion of the lower object is originated from the parameter iteratively updated by past performance, the current assessment depicting the current state, the confidence on future execution, and the coordinated motion of the upper object is then planned based on the inclined insertion, the estimated inclination, and the force to be compensated. Experiments are carried out to demonstrate the validation of the proposed method.

This article investigates the coordinated motion planning of independent manipulators in the domain of precision manipulation, where forces are considered in the level of tens or hundreds of milliNewtons. Independent manipulating is crucial in randomly combining robot arms to temporarily fulfill certain tasks. We study the cases where a part of task information is known to each manipulator and yet no communication exists between them. In the coordinated structure, the leading manipulator plans its movement according to the desired state and its evaluation of the object force. The follower needs to compensate for the offset errors, estimate the intention of the leader, evaluate its confidence on the estimation, and plan its movement accordingly. The handling of special states, i.e., zero force feedback, is also discussed. Experiments are carried out to demonstrate the validation of the proposed method.

The inherent hysteresis property of piezoelectric actuator (PEA) brings challenges to its modeling and control. This paper proposes a model learning method that is suitable for both forward and inverse PEA models. The hysteresis property is learned based on least squares support vector machines (LS-SVMs). A larger dataset is used for training LS-SVM to guarantee a good generalization performance. Support vectors pruning is utilized to reduce the model complexity. The rate-dependent property of PEA is identified as a linear dynamic submodel. Moreover, a pointing control system with two dualPEA-axis steering mirrors is developed, which can regulate the 4-degree-of-freedom pose of a laser beam. The coordinated control of four PEAs is realized based on the Jacobian matrix. The learned inverse PEA models are used for the feedforward compensation of each PEA's nonlinearity. A series of experiments were conducted to evaluate the proposed method's effectiveness.

As the failure of power line insulators leads to the failure of power transmission systems, an insulator inspection system based on an aerial platform is widely used. Insulator defect detection is performed against complex backgrounds in aerial images, presenting an interesting but challenging problem. Traditional methods, based on handcrafted features or shallow learning techniques, can only localize insulators and detect faults under specific detection conditions, such as when sufficient prior knowledge is available, with low background interference, at certain object scales, or under specific illumination conditions. This paper discusses the automatic detection of insulator defects using aerial images, accurately localizing insulator defects appearing in input images captured from real inspection environments. We propose a novel deep convolutional neural network (CNN) cascading architecture for performing localization and detecting defects in insulators. The cascading network uses a CNN based on a region proposal network to transform defect inspection into a two-level object detection problem. To address the scarcity of defect images in a real inspection environment, a data augmentation method is also proposed that includes four operations: 1) affine transformation; 2) insulator segmentation and background fusion; 3) Gaussian blur; and 4) brightness transformation. Defect detection precision and recall of the proposed method are 0.91 and 0.96 using a standard insulator dataset, and insulator defects under various conditions can be successfully detected. Experimental results demonstrate that this method meets the robustness and accuracy requirements for insulator defect detection.

The real-time and robust surgical instrument segmentation is an important issue for endoscopic vision. We propose an instrument segmentation method fusing the convolutional neural networks (CNN) prediction and the kinematic pose information. First, the CNN model ToolNet-C is designed, which cascades a convolutional feature extractor trained over numerous unlabeled images and a pixel-wise segmentor trained on few labeled images. Second, the silhouette projection of the instrument body onto the endoscopic image is implemented based on the measured kinematic pose. Third, the particle filter with the shape matching likelihood and the weight suppression is proposed for data fusion, whose estimate refines the kinematic pose. The refined pose determines an accurate silhouette mask, which is the final segmentation output. The experiments are conducted with a surgical navigation system, several animal-tissue backgrounds, and a debrider instrument.

This paper proposes an efficient strategy for the insertion of the partially flexible object in precision assembly. The partially flexible object refers to the component with unevenly distributed flexibilities: coupling rigidity and flexibility. This paper focuses on the insertion of one class of partially flexible objects: rigid shapes connected by a compliant mechanism. We first analyze the horizontal compliance of the compliant mechanism and build a model to relate its state and force. The insertion is separated into two stages according to the insertion type. The first stage is a compliant insertion and we estimate the insertion direction based on the built model, horizontally compensate resorting to the horizontal compliance and the updated direction, and efficiently plan the vertical insertion in an aggressive strategy regarding the uncertainties caused by the compliant mechanism and predicting the future insertion. The second stage is a hybrid insertion with both rigid and compliant gripping and its features include that the object states are not precisely measurable and the motion of a part of the object is not directly controllable. To solve it, we qualitatively and quantitatively analyze all possible configurations and, taking advantage of the insertion property, conclude one insertion posture based on which a control strategy is proposed. Additionally, a conservative insertion strategy is planned resorting to the past execution performance and the current state evaluation. Experiments are carried out to demonstrate the validation of the proposed method.

In this paper, a position measurement system, including drogue's landmark detection and position computation for autonomous aerial refueling of unmanned aerial vehicles, is proposed. A multitask parallel deep convolution neural network (MPDCNN) is designed to detect the landmarks of the drogue target. In MPDCNN, two parallel convolution networks are used, and a fusion mechanism is proposed to accomplish the effective fusion of the drogue's two salient parts' landmark detection. Considering the drogue target's geometric constraints, a position measurement method based on monocular vision is proposed. An effective fusion strategy, which fuses the measurement results of drogue's different parts, is proposed to achieve robust position measurement. The error of landmark detection with the proposed method is 3.9%, and it is obviously lower than the errors of other methods. Experimental results on the two KUKA robots platform verify the effectiveness and robustness of the proposed position measurement system for aerial refueling.

In recent years, the application of deep learning based on deep convolutional neural networks has gained great success in face detection. However, one of the remaining open challenges is the detection of small-scaled faces. The depth of the convolutional network can cause the projected feature map for small faces to be quickly shrunk, and most detection approaches with scale invariant can hardly handle less than 15×15 pixel faces. To solve this problem, we propose a different scales face detector (DSFD) based on Faster R-CNN. The new network can improve the precision of face detection while performing as real-time a Faster R-CNN. First, an efficient multitask region proposal network (RPN), combined with boosting face detection, is developed to obtain the human face ROI. Setting the ROI as a constraint, an anchor is inhomogeneously produced on the top feature map by the multitask RPN. A human face proposal is extracted through the anchor combined with facial landmarks. Then, a parallel-type Fast R-CNN network is proposed based on the proposal scale. According to the different percentages they cover on the images, the proposals are assigned to three corresponding Fast R-CNN networks. The three networks are separated through the proposal scales and differ from each other in the weight of feature map concatenation. A variety of strategies is introduced in our face detection network, including multitask learning, feature pyramid, and feature concatenation. Compared to state-of-the-art face detection methods such as UnitBox, HyperFace, FastCNN, the proposed DSFD method achieves promising performance on popular benchmarks including FDDB, AFW, PASCAL faces, and WIDER FACE.

In this paper, a novel method is proposed for curve detection in images using a feature-related searching control model. It is composed of three parts: 1) prediction; 2) searching; and 3) updating. First, curve related features are modeled to a three order array. Then, equations of the prediction, searching, and curve parameter updating are deduced. Third, an optimal model for curve parameter estimation during iterations is given. Based on the proposed model, a curve detection algorithm is designed. Experiments on thousands of images demonstrate the effectiveness and advantages of the proposed method. Comparison experiments with state-of-the-art methods show that the proposed method outperforms the existing methods on most indexes. Our method can describe the contents of original images more completely with fewer curves. The contour evaluation framework and the Berkeley segmentation dataset are used to evaluate the performances of different curve detection methods. The proposed method can also detect curves in the order relates to their importance, which has been validated in experiments.

This paper proposes an efficient strategy to simultaneously insert multiple objects, which are parallel connected by passive compliant mechanisms, in precision assembly. The distinctions of this task include: each object is held compliantly; multiple objects are parallel connected to a manipulator; not all the peg-in-hole has the same insertion condition; and high accuracy is required for each insertion. This configuration can provide sufficient compliance and improve insertion efficiency for massive precision assembly. We model the relationship between the state and force of a single compliant mechanism, and analyze the horizontal compliance of parallel mechanisms. Based on the model, with a fitting and optimization method the states of all but one compliant mechanisms are acquired from microscopic views and the remaining states are optimized with resultant forces provided by a force sensor. To efficiently plan the parallel insertion, we propose a strategy to horizontally compensate according to the resultant force and the horizontal compliance, and to vertically insert based on the insertion ratio expectation, the horizontal offsets of each individual insertion, and the horizontal force. Experiments are carried out to demonstrate the validation of the proposed method.

Facial expression recognition is a challenging task in computer vision field using only single facial image. As we know, human faces are convex spheres. The self-occlusion phenomenon generated from face pose will seriously affect the accuracy of expression recognition. In order to solve this problem, we propose a novel facial expression recognition method for different pose faces based on special landmark detection (FER-MPI-SFL). Our method is based on two shared networks. The outputs of the first Network are 29 special landmarks and 1 face box, which are the inputs of the second network and used to estimate face pose. The methods of RoIAlign and feature map concatenation are introduced in the second network to recognize the facial expression. The weight allocation of feature maps concatenation is guided by the result of pose estimation. In addition, an improved center loss is proposed to make the distances between the features of different expressions larger and easier to be classified in the feature space. As a result, superior performance to other state-of-the-art methods is achieved in facial expression databases CK+, MMI, Oulu-CASIA VIS and a new created database CASIA-MFE which contains more faces with different poses.

This paper proposes an efficient and accurate method for collision detection between convex prisms in precision alignment and a detach controller based on the contact location for the collision occurrence. We project the objects onto an appropriate plane and detect the collision status considering the relationship between their planar contours. Efficient methods are presented for the overlap checking of several elementary contours, and the way to obtain the vertices of the projected shapes is also introduced. The detection is then accelerated by classifying into the corresponding case based on the relative configuration and using the efficient planar checking to replace the cubic calculation. A detach controller is presented to immediately separate the objects according to the contact location once collision occurs. The computational efficiency and comparison are demonstrated in simulations, and experiments are carried out to validate the detection and the controller.

This paper proposes a suite of methods to realize high precision pose measurement in 3-D Cartesian space based on a multicamera microscopic vision system. Since it is inefficient to develop a specific image algorithm for each kind of object and the imaging condition might be unsatisfactory, we propose a method of contour primitives of interest extraction, which allows flexible reconfiguration for novel object image and owns robustness under different imaging conditions. The object is detected in grayscale image based on a template of contour primitives. Edges are extracted according to derivatives along the normal vectors of these contour primitives. The positions and directional derivatives of these edges are used for feature extraction and autofocus, respectively. The point features and line features extracted from multiview images are utilized to measure 3-D vectors and orientations, respectively, based on image Jacobian matrices. Cameras' linear motions are considered in the imaging model, so that the measurement range is expanded beyond the limitation of microscopes' shallow depths of field. The affine epipolar constraint and focused planes intersection constraint between cameras are applied to improve the real time performances of image feature extraction and multicamera autofocus, respectively. A series of experiments are conducted to verify the effectiveness of the proposed methods. The root mean square errors of pose measurement are evaluated as 3 μm in position and 0.05° in orientation, while the measurement range is about 5000 μm in position and 20° in orientation.

This paper describes a high-frame-rate (HFR) vision system that can detect multiple objects in an image of 512 × 512 pixels at 12,000 frames per seconds (fps). An optimized algorithm is proposed based on conventional Histograms of Oriented Gradient (HOG) descriptor and Support Vector Machine (SVM) classifier algorithms for hardware implementation. By implementing the proposed algorithm on a field-programmable gate array (FPGA) of a high-speed vision platform, multi-object in an image can be detected at 12,000 fps under complex background. In hardware implementation, 64 pixels were processed in parallel with 80 MHz camera clock. Source image and detection results can be transferred to personal computer (PC) in real-time for recording or post-processing. Our developed HFR multi-object detection system was verified by performing several evaluations.

A new image-based visual servoing method based on sensitive features is presented to separately realize the position control and orientation control. Line features are used for the orientation control because of their sensitivities to rotational motions. Point features and area size features are employed to realize the position control since area size features are very sensitive to the objects' depths. The translations resulting from rotational motions are introduced into the position control as the compensation in order to eliminate the influence of the camera's motions on the point features. The depths for all active features are estimated via interaction matrices, features variations, and the executed camera motions. The proposed method can keep the tracked objects in the camera's field of view in the visual servoing process. In addition, the determination methods of the interaction matrices for point, line, and area size features are proposed. Comparing to the traditional method, the proposed determination method of the interaction matrix for line is independent from the parameters of the plane containing the line. Experimental results verify the effectiveness of the proposed methods.

An automatic system is developed to realize high precision assembly of two components in the size of mm level with an interference fit in 3-dimensional (3-D) space with 6-degree-of- freedoms (DOF), which consists of a manipulator, an adjusting platform, a sensing system and a computer. The manipulator is employed to align component B to the component A in position. The adjusting platform aligns the component A to component B in orientations and inserts A into B. The sensing system includes three microscopes and a force sensor. The three microscopes are mounted approximately orthogonal to observe components from different directions in the aligning stage. The force sensor is introduced to detect the contact force in assembly process. In the aligning stage, a pose control method based on image Jacobian matrix is proposed. In the insertion stage, a position control method based on the contact force is proposed. The calibration of image Jacobian matrix is also presented. Experimental results demonstrate the effectiveness of the proposed system and methods.

This correspondence paper proposes a fast orientation estimation approach of natural images without the help of semantic information. Different from traditional low-level features, our low-level features are extracted inspired by the biological simple cells of the visual cortex. Two approximated receptive fields to mimic the biological cells are presented, and a local rotation operator is introduced to determine the optimal output and local orientation corresponding to an image position, which serve as the low-level feature employed in this paper. To generate the low-level features, a bisection method is applied to the first derivative of the model of receptive fields. Moreover, the feature screener is introduced to eliminate too much useless low-level features, which will speed up the processing time. After all the valuable low-level features are combined, the overall image orientation is estimated. The proposed approach possesses several features suitable for real-time applications. First, it avoids the tedious training procedure of some conventional methods. Second, no specific reference such as the horizon is assumed and no a priori knowledge of image is required. The proposed approach achieves a real-time orientation estimation of natural images using only low-level features with a satisfactory resolution. The effectiveness of our proposed approach is verified on real images with complex scenes and strong noises.

This paper proposes two methods for collision detection between cylindrical components when mutual blocking occurs in the view of cameras. In reconstruction approach, 3D models are built according to key features captured by cameras and constraint optimization is employed to rapidly find possible intersections. To satisfy the computational efficiency required by real time operation, we present another way, projection method, to convert two planar views to contours on a projection plane and to detect high dimension collisions by studying the projection's relationships in low dimension. Nine cases are totally categorized and eleven parameters are constructed for detection on the basis of relative postures and positions. Simulations and experiments are carried out to demonstrate their validity.

In this paper, we propose a robust state-based structured support vector machine (SVM) tracking algorithm combined with incremental principal component analysis (PCA). Different from the current structured SVM for tracking, our method directly learns and predicts the object's states and not the 2-D translation transformation during tracking. We define the object's virtual state to combine the state-based structured SVM and incremental PCA. The virtual state is considered as the most confident state of the object in every frame. The incremental PCA is used to update the virtual feature vector corresponding to the virtual state and the principal subspace of the object's feature vectors. In order to improve the accuracy of the prediction, all the feature vectors are projected onto the principal subspace in the learning and prediction process of the state-based structured SVM. Experimental results on several challenging video sequences validate the effectiveness and robustness of our approach.

The microscope has characteristics of a planar vision with small view field and small view depth. For micro operation systems with multiple manipulators, the handling of irregular objects may lead to a nonorthogonal microscopic system, which needs to focus on clear viewing interested features, and it may also hardly locate the exact position and posture of the robot arms. In view of these, this paper proposes an active calibration method to compute image Jacobian matrix, which maps from the relative motion of the manipulators to the image coordination changes in the microscopes. We also investigate the applications in micro operator positioning, tracking for distributed systems, and movement optimization in micro-assembly. Experiments are carried out on a micro-assembly platform equipped with three microscopes and six robot arms, and the results validate the effectiveness of the proposed method.

Difficulties arise in the micro-assembly of many irregular objects and in the insertion with contact between components of soft materials. To handle these problems, we design a micro-operational platform with multiple manipulators to facilitate a sequence of assembly. Six robot arms and three microscopes are incorporated, together with macro and micro motion systems. We also propose a hybrid control strategy to achieve high precision and protect objects. This hybrid scheme includes vision based positioning controllers for alignment, which employ incremental PI controllers and image Jacobian matrix, force based controllers for insertion, and a decision mechanism determining the assembly state. Experiments demonstrate the effectiveness of the proposed platform and control methods.

A DataSpace Support Platform (DSSP) is a self-sustained and self-managed system which needs to support uncertainty among its mediated schemas and its schema mappings. Some approaches for managing such uncertainty by assigning probabilities and reliability degrees to schema mappings have been proposed. Unfortunately, the number of mappings self-generated by a DSSP is usually too large and among those possible mappings, some might be totally correct and others partially correct. Therefore, providing probabilities or reliability degrees to the mappings is necessary but not sufficient to resolve uncertainty among them. This paper proposes a stepper-based approach called pos-mapping to managing reliable mappings using possibility theory. Instead of choosing a threshold for managing the reliable mappings, pos-mapping approach orders and divides the set of reliable mappings into subsets of possibility distributions and assigns to each of these subsets a recursive possibility degree function. The recursiveness of the possibility degree function leads to an incremental management of the possibility distributions. Experimental results show that our system is more efficient than the existing systems and the accuracy of the results increases with the number of reliable schemas in the DSSP.

An analytic flying model that can well represent the physical behavior is derived, where the ball's self-rotational velocity changes along with the flying velocity. Based on the least square method, a rebound model that represents the relation between the velocities before and after rebound is established. The initial trajectory is fitted to three second order polynomials of the flying time with the measured positions of the ball. The initial velocities of the ball in the analytic flying model, including the flying velocity and the self-rotational velocity, are computed from the polynomials. The ball's landing position and velocity is predicted with the model. The velocities after rebound are determined with the rebound model. By taking the velocities after rebound as new initial ones, the flying trajectory after rebound is described with the model again. In other words, the ball's trajectory is predicted. Experimental results verify the effectiveness of the proposed method.

A visual control system is designed for a table tennis robot with five degrees of Freedom (DOFs). It consists of four parts such as ball sensing, trajectory predicting, motion planning, and motion control. A highvelocity stereo vision system with parallel architecture is developed to sense the motions of table tennis ball. The striking parameters including position, velocity, and time are predicted according to the predicted trajectory of the ball based on several measured positions. The motion computer receives the predicted striking parameters and performs motion planning for the robot. A motion control card embedded in the motion computer receives the planning results and controls the motions of the robot via the servo drivers for X and Y axes. A microprocessor is designed to produce pulses to control the motions of the rest three axes via the drivers. Experiments are well conducted to verify the effectiveness of the developed robot and control system.

Color constancy is an important perceptual ability of humans to recover the color of objects invariant of light information. It is also necessary for a robust machine vision system. Until now, a number of color constancy algorithms have been proposed in the literature. In particular, the edge-based color constancy uses the edge of an image to estimate light color. It is shown to be a rich framework that can represent many existing illumination estimation solutions with various parameter settings. However, color constancy is an ill-posed problem; every algorithm is always given out under some assumptions and can only produce the best performance when these assumptions are satisfied. In this article, we have investigated a combination strategy relying on the Extreme Learning Machine (ELM) technique that integrates the output of edge-based color constancy with multiple parameters. Experiments on real image data sets show that the proposed method works better than most single-color constancy methods and even some current state-of-the-art color constancy combination strategies.

The aim of color constancy is to remove the effect of the color of the light source. As color constancy is inherently an ill-posed problem, most of the existing color constancy algorithms are based on specific imaging assumptions such as the grey-world and white patch assumptions.

The aim of color constancy is to remove the effect of the color of the light source. Since color constancy is inherently an ill-posed problem, different assumptions have been proposed. Because existing color constancy algorithms are based on specific assumptions, none of them can be considered as universal. Therefore, how to select a proper algorithm for a given imaging configuration is an important question.

Salient Region Extraction provides an alternative methodology to image description in many applications such as adaptive content delivery and image retrieval. In this paper, we propose a robust approach to extracting the salient region based on bottom-up visual attention. The main contributions are twofold: 1) Instead of the feature parallel integration, the proposed saliencies are derived by serial processing between texture and color feature. 2) A constructive approach is proposed for rendering an image by a non-linear intensity mapping, which can efficiently eliminate high contrast noise regions in the image. And then the salient map can be robustly generated for a variety of nature images. Finally, the salient region extracted by our algorithm is used for image semantic retrieval. Experiments show that the proposed algorithm can characterize the human perception well and achieve satisfied retrieval performance.

Retrieving videos using key words requires obtaining the semantic features of the videos. Most work reported in the literature focuses on annotating a video shot with a fixed number of key words, no matter how much information is contained in the video shot. In this paper, we propose a new approach to automatically annotate a video shot with an adaptive number of annotation key words according to the richness of the video content. A Semantic Candidate Set (SCS) with fixed size is discovered using visual features. Then the final annotation set, which has an unfixed number of key words, is obtained from the SCS by using Bayesian Inference, which combines static and dynamic inference to remove the irrelevant candidate key words. We have applied our approach to video retrieval. The experiments demonstrate that video retrieval using our annotation approach outperforms retrieval using a fixed number of annotation words.

This paper deals with maneuver issues of a multilink biomimetic robotic fish, particularly focusing on turning control in free swimming. The characteristic parameters determining turning performance involve magnitude, position, and time of the deflections applied to the links, which are discussed via a series of simulation calculations and actual experiments.

In this paper, a new active visual system is developed, which is based on bionic vision and is insensitive to the property of the cameras. The system consists of a mechanical platform and two cameras. The mechanical platform has two degrees of freedom of motion in pitch and yaw, which is equivalent to the neck of a humanoid robot. The cameras are mounted on the platform. The directions of the optical axes of the two cameras can be simultaneously adjusted in opposite directions. With these motions, the object's images can be located at the centers of the image planes of the two cameras. The object's position is determined with the geometry information of the visual system. A more general model for active visual positioning using two cameras without a neck is also investigated. The position of an object can be computed via the active motions. The presented model is less sensitive to the intrinsic parameters of cameras, which promises more flexibility in many applications such as visual tracking with changeable focusing. Experimental results verify the effectiveness of the proposed methods.

In this paper, a new visual positioning method based on corresponding points at two adjacent views is developed for mobile robots. A camera is mounted on a wheeled mobile robot with nonholonomic constraints. The camera whose intrinsic parameters are well calibrated can rotate around an axis perpendicular to the ground plane. The relative orientation and scaled position offsets of the mobile robot are computed from the corresponding points in the common part of the views despite their unknown positions in Cartesian space. The relative orientation is used in a simple visual dead reckoning method to modify the odometry information. Then, based on the relative orientation and modified odometry information, equations are given to determine the position and orientation of the mobile robot. Experiments are performed to verify the effectiveness of the proposed methods

In this paper, a new method is proposed to detect the pose of an object with two cameras. First, the intrinsic parameters of the cameras are self-calibrated with two pairs of parallel lines that are orthogonal. Then, the poses of the cameras relative to the parallel lines are deduced, and the rotational transformation between the two cameras is calculated. With the intrinsic parameters and the relative pose of the two cameras, a method is proposed to obtain the poses of a line, plane, and rigid object. Furthermore, a new visual-control method is developed using a pose detection rather than a three-dimensional reconstruction. Experiments are conducted to verify the effectiveness of the proposed method.

A method of image process and features extraction for structured light image of welding seam with arc and splash disturbance is proposed. The seam area is detected by search with large step. The adaptive thresholds of image enhancement are determined in the frequency domain of the gray image. Then, the target image is pre-processed using image enhancement and binarization. After thinning the seam with its both edges, main characteristic line is obtained using Hotelling transform and Hough transform. Finally, the feature points in the seam are found according to its second derivative. Experimental results show its effectiveness, good performance in real time and adaptability to different seams.

Dead reckoning is an important method for mobile robot. If its accuracy can be improved, navigation tasks will be simplified. The mobile robot concerned in this paper is with five wheels, in which there are dual driving wheels and a castor. A measurement wheel with encoder is fixed beside each driving wheel. There are two encoders fixed on the castor. One measures the castor's movement, and another indicates the angle of yawing. A new kind of kinematics equations derived from the robot's turning radius and angle of movement trajectory is presented. We can get a group of turning radius and angle from the data of four encoders, in which there are four different values for radius and six different values for angle. The estimated values of radius and angle can be gotten by fusion using fuzzy algorithm. The improved dead reckoning method has the advantages of simplicity, less computation cost, cheapness and good performance in uneven road. The simulation shows its effectiveness.