This article addresses two key issues in RGB-D salient object detection based on the convolutional neural network (CNN). 1) How to bridge the gap between the "data-hungry" nature of CNNs and the insufficient labeled training data in the depth modality? 2) How to take full advantages of the complementary information among two modalities. To solve the first problem, we model the depth-induced saliency detection as a CNN-based cross-modal transfer learning problem. Instead of directly adopting the RGB CNN as initialization, we additionally train a modality classification network (MCNet) to encourage discriminative modality-specific representations in minimizing the modality classification loss. To solve the second problem, we propose a densely cross-level feedback topology, in which the cross-modal complements are combined in each level and then densely fed back to all shallower layers for sufficient cross-level interactions. Compared to traditional two-stream frameworks, the proposed one can better explore, select, and fuse cross-modal cross-level complements. Experiments show the significant and consistent improvements of the proposed CNN framework over other state-of-the-art methods.

In the context of wearable gaze tracking techniques, the problems of two-dimensional (2-D) and three-dimensional (3-D) gaze estimation can be viewed as inferring 2-D epipolar lines and 3-D visual axes from eye monitoring cameras. To this end, in this article, a simple local polynomial model is proposed to back-project a pupil center onto its corresponding visual axis. Based on this approximation, a homographylike relation is derived in a local manner, and via the Leave-One-Out cross-validation criterion, training gaze samples at one certain depth is leveraged to partition entire input space into multiple overlapping subregions. Then, the gaze data at another depth are utilized to recover the epipolar point, i.e., the image eyeball center. Thus, given a pupil image, the corresponding epipolar line can be determined by the resolved homographylike mapping and the epipolar point. By using the same partition structure, 3-D gaze prediction model can be inferred by solving a nonlinear optimization problem, which aims to minimize the angular disparities between training visual directions and prediction ones. Meanwhile, it is necessary to form a good starting point and suitable constraints for the optimization problem. Otherwise, it may end up with trivial solutions, i.e., faraway eye positions. To facilitate the practical implementation of our proposed method, we also analyze how the spatial distribution of calibration points impacts the model learning accuracy. The experiment results justify the effectiveness of our proposed gaze estimation method for both the normal vision and eyewear users.

Infrared imaging spectrometer (IRIS) often suffers from overlapped bands and random noises, which limit the precision of subsequent processing in robot vision sensing. To address this problem, we propose a novel Gabor transform-based infrared spectrum restoration method by successfully exploring the intrinsic structure of the clean IR spectrum from the degraded one. At first, a total variation (TV) regularized Gabor coefficients adjustment descriptor is designed and incorporated into the spectrum restoration model. Then, the proposed model is inferred via an efficient optimization approach based on split Bregman iteration method. Comprehensive experiments illustrate the significant and consistent improvements of the developed model over state-of-the-art approaches. The restored high-resolution spectrum can be utilized for detecting the different materials in the robot visual tracking systems.

In the context of mobile gaze tracking techniques, a 3D gaze point can be calculated as the middle point between two 3D visual axes. To infer gaze directions and eyeball positions, a nonlinear optimization problem is typically formulated to minimize the angular disparities between the training gaze directions and prediction ones. Nonetheless, the experimental results reported by some previous works show that this kind of approaches are very likely to yield large prediction errors hence considered less useful for human-machine interactions. In this study, we aim to address this widespread issue in three aspects. At first, instead of using a global regression model, a simple local polynomial model is proposed to back-project a pupil center onto its corresponding visual axis. Based on the Leave-One-Out cross-validation criterion, the partition structure is automatically learned in the process of resolving a homography-like relationship. Secondly, a good starting point for nonlinear-optimization is obtained by the image eyeball center, which can be estimated by systematic parallax errors. Meanwhile, it is necessary to add the suitable constraints for 3D eye positions. Otherwise, the optimization may end up with trivial solutions, i.e., faraway eye positions. Thirdly, we explore a strategy for designing the spatial distribution of calibration points in a principled manner. The experiment results demonstrate that an encouraging gaze estimation accuracy can be achieved by our proposed framework for both the normal vision and eyewear users.

The gaze estimation in the mobile scenario often suffers from the extrapolation and parallax errors. In this paper, we propose a novel calibration framework to achieve the precise gaze estimation for head-mounted gaze trackers. Our proposed framework consists of two steps to learn a point-to-point and a point-to-line relations, respectively. The aim of step I is to infer the relation between pupil centers and spatially constrained points of regard. By adopting the “CalibMe” gaze data acquisition method, a sparse Gaussian Process using pseudo-inputs is used to capture the smooth residual field unmodeled by the polynomial function. Meanwhile, a distraction detection criterion is introduced to identify the moment when user's attention is taken away from the calibration point thereby removing outliers. By combining with the point-to-point relation inferred in step I, the observed parallax errors are leveraged in step II to obtain a point-to-line relation, i.e., each pupil center will correspond to an epipolar line. Thus, the real image gaze point projected from different depths is predicted as the intersection of two epipolar lines inferred from binocular data. The simulation and experimental results show the effectiveness of our proposed calibration framework for head-mounted gaze trackers.

Convolutional neural networks have achieved wide success in RGB saliency detection. Recently, the advent of RGB-D sensors such as Kinect provide additional geometric saliency cues. However, the key challenge for RGB-D salient object detection that how to fuse RGB and depth information sufficiently is still under-studied. Traditional works mainly follow the two-stream architecture and combine RGB and depth features/decisions in an early or late point. The multi-modal fusion stage is performed by directly concatenating the features from two modalities without selection. In this work, we address this question by proposing a novel network with a distinguished insight: A selection module is significantly helpful for more informative and sufficient cross-modal cross-level combination. To this end, we introduce a top-down RGB-D fusion network which integrates an attention-aware cross-modal cross-level fusion block in each level to select discriminative features from each level and each modality. Extensive experiments on public datasets show that the proposed network is able to solve the key problems in RGB-D fusion and achieves state-of-the-art performance on RGB-D salient object detection.

In this paper, we aim to address two challenges in the implementation of mobile gaze tracking systems, i.e., the parallax error and the inflexible calibration procedure. Our proposed method mainly involves two steps and all the calibration process can be completed without needs to receive user's commands. At first, instead of fixating at calibration points successively, users are required to fixate at one calibration point while smoothly and persistently varying the head position. In this case, the eye movements will compensate for head movements due to the activation of the smooth pursuit system. Then the PCA analysis is applied for distinguishing the smooth pursuits from other kinds of eye movements to get reliable training data. Meanwhile, a online sparse Gaussian Process using the FITC approximation is proposed to model the relationship between image pupil centers and gaze points. The next step aims to compensate the parallax error by recovering the epipolar geometry of gaze tracking systems and eyeballs. Users are asked to fixate at the points with different distances and detected parallax errors can be applied to estimate the epipolar geometry of mobile gaze trackers by solving a nonlinear optimization problem. Thus the real image gaze point with different depths can be straightforwardly estimated as the intersection point of two epipolar lines derived from binocular data. The simulation and experimental results demonstrate the effectiveness of our proposed method.

Fusing RGB and depth data is compelling in boosting performance for various robotic and computer vision tasks. Typically, the streams of RGB and depth information are merged into a single fusion point in an early or late stage to generate combined features or decisions. The single fusion point also means single fusion path, which is congested and inflexible to fuse all the information from different modalities. As a result, the fusion process is brute-force and consequently insufficient. To address this problem, we propose a multi-scale multi-path multi-modal fusion network (M3Net), in which the fusion path is scattered to diversify the contributions of each modality from global and local perspectives. Specially, the CNN streams of each modality are fused with a global understanding path and meanwhile a local capturing path. By filtering and regulating information flow in a multi-path way, the M3Net is equipped with more adaptive and flexible fusion mechanism, thus easing the gradient-based learning process, improving the directness and transparency of the fusion process and simultaneously facilitating the fusion process with multi-scale perspectives. Comprehensive experiments demonstrate the significant and consistent improvements of the proposed approach over state-of-the-art methods.