Online unsupervised video object segmentation (UVOS) uses the previous frames as its input to automatically separate the primary object(s) from a streaming video without using any further manual annotation. A major challenge is that the model has no access to the future and must rely solely on the history, i.e., the segmentation mask is predicted from the current frame as soon as it is captured. In this work, a novel contrastive motion clustering algorithm with an optical flow as its input is proposed for the online UVOS by exploiting the common fate principle that visual elements tend to be perceived as a group if they possess the same motion pattern. We build a simple and effective auto-encoder to iteratively summarize non-learnable prototypical bases for the motion pattern, while the bases in turn help learn the representation of the embedding network. Further, a contrastive learning strategy based on a boundary prior is developed to improve foreground and background feature discrimination in the representation learning stage. The proposed algorithm can be optimized on arbitrarily-scale data (i.e., frame, clip, dataset) and performed in an online fashion. Experiments on$\textit {DAVIS}_{\textit {16}}$, FBMS, and SegTrackV2 datasets show that the accuracy of our method surpasses the previous state-of-the-art (SoTA) online UVOS method by a margin of 0.8%, 2.9%, and 1.1%, respectively. Furthermore, by using an online deep subspace clustering to tackle the motion grouping, our method is able to achieve higher accuracy at$3\times $faster inference time compared to SoTA online UVOS method, and making a good trade-off between effectiveness and efficiency. Our code is available athttps://github.com/xilin1991/CluterNet.

Monocular depth estimation has drawn widespread attention from the vision community due to its broad applications. In this paper, we propose a novel physics (geometry)-driven deep learning framework for monocular depth estimation by assuming that 3D scenes are constituted by piece-wise planes. Particularly, we introduce a new normal-distance head that outputs pixel-level surface normal and plane-to-origin distance for deriving depth at each position. Meanwhile, the normal and distance are regularized by a developed plane-aware consistency constraint. We further integrate an additional depth head to improve the robustness of the proposed framework. To fully exploit the strengths of these two heads, we develop an effective contrastive iterative refinement module that refines depth in a complementary manner according to the depth uncertainty. Extensive experiments indicate that the proposed method exceeds previous state-of-the-art competitors on the NYU-Depth-v2, KITTI and SUN RGB-D datasets. Notably, it ranks 1st among all submissions on the KITTI depth prediction online benchmark at the submission time. The source code is available at https://github.com/ShuweiShao/NDDepth.

Monocular depth estimation is an ill-posed task in computer vision, which holds great significance in the fields such as artificial intelligence, virtual reality, augmented reality, path planning, unmanned driving, and navigation guidance. The primary objective of monocular depth estimation is to predict the depth value of each pixel or infer depth information, given just a single red-green-blue (RGB) image as input. Traditional monocular depth estimation methods rely on limited depth cues, such as strict scene conditions. With the significant advancements in computer vision and artificial intelligence, monocular depth estimation using deep learning has been extensively researched and has yielded substantial results. This paper presents a comprehensive survey of monocular depth estimation. Firstly, we give an overall introduction to monocular depth estimation and explain it from traditional and deep learning-based methods, respectively. To specify, supervised, self-supervised and semi-supervised models are described in detail in deep learning-based methods. Additionally, we introduce publicly available benchmark datasets and evaluation metrics commonly used in this field. Finally, we discuss the current challenges and promising prospects for the development of monocular depth estimation.

Monocular depth estimation (MDE) is a fundamental topic of geometric computer vision and a core technique for many downstream applications. Recently, several methods reframe the MDE as a classification-regression problem where a linear combination of probabilistic distribution and bin centers is used to predict depth. In this paper, we propose a novel concept of iterative elastic bins (IEBins) for the classification-regression-based MDE. The proposed IEBins aims to search for high-quality depth by progressively optimizing the search range, which involves multiple stages and each stage performs a finer-grained depth search in the target bin on top of its previous stage. To alleviate the possible error accumulation during the iterative process, we utilize a novel elastic target bin to replace the original target bin, the width of which is adjusted elastically based on the depth uncertainty. Furthermore, we develop a dedicated framework composed of a feature extractor and an iterative optimizer that has powerful temporal context modeling capabilities benefiting from the GRU-based architecture. Extensive experiments on the KITTI, NYU-Depth-v2 and SUN RGB-D datasets demonstrate that the proposed method surpasses prior state-of-the-art competitors. The source code is publicly available at https://github.com/ShuweiShao/IEBins.

Differently exposed low dynamic range (LDR) images are often captured sequentially using a smart phone or a digital camera with movements. Optical flow thus plays an important role in ghost removal for high dynamic range (HDR) imaging. The optical flow estimation is based on the theory of photometric consistency, which assumes that the corresponding pixels between two images have the same intensity. However, the assumption is no longer valid for the differently exposed LDR images since a pixel’s intensity changes significantly inter images. To address the problem, an unsupervised optical flow estimation framework, is presented in this study. Intensity mapping functions (IMFs) are first adopted to alleviate the intensity changes between the LDR images. Then a novel IMF-based unsupervised learning objective is proposed to circumvent the need for ground truth optical flows when training the deep network. Experimental results and ablation studies on publicly available datasets show that our framework outperforms the state-of-the-art unsupervised optical flow methods, demonstrating the effectiveness of the IMF and the learning objective. Our code is available athttps://github.com/liuziyang123/LDRFlow.

Monocular depth estimation plays a fundamental role in computer vision. Due to the costly acquisition of depth ground truth, self-supervised methods that leverage adjacent frames to establish a supervision signal have emerged as the most promising paradigms. In this work, we propose two novel ideas to improve self-supervised monocular depth estimation: 1) self-reference distillation and 2) disparity offset refinement. Specifically, we use a parameter-optimized model as the teacher updated as the training epochs to provide additional supervision during the training process. The teacher model has the same structure as the student model, with weights inherited from the historical student model. In addition, a multiview check is introduced to filter out the outliers produced by the teacher model. Furthermore, we leverage the contextual consistency between high-level and low-level features to obtain multiscale disparity offsets, which are used to refine the disparity output incrementally by aligning disparity information at different scales. The experimental results on the KITTI and Make3D datasets show that our method outperforms previous state-of-the-art competitors.

Depth estimation attracts widespread attention in the computer vision community. However, it is still quite difficult to recover an accurate depth map using only one RGB image. We observe a phenomenon that existing methods tend to fail in different cases, caused by differences in network architecture, loss function and so on. In this work, we investigate into the phenomenon and propose to integrate the strengths of multiple weak depth predictor to build a comprehensive and accurate depth predictor, which is critical for many real-world applications, e.g., 3D reconstruction. Specifically, we construct multiple base (weak) depth predictors by utilizing different Transformer-based and convolutional neural network (CNN)-based architectures. Transformer establishes long-range correlation while CNN preserves local information ignored by Transformer due to the spatial inductive bias. Therefore, the coupling of Transformer and CNN contributes to the generation of complementary depth estimates, which are essential to achieve a comprehensive depth predictor. Then, we design mixers to learn from multiple weak predictions and adaptively fuse them into a strong depth estimate. The resultant model, which we refer to as Transformer-assisted depth ensembles (TEDepth). On the standard NYU-Depth-v2 and KITTI datasets, we thoroughly explore how the neural ensembles affect the depth estimation and demonstrate that our TEDepth achieves better results than previous state-of-the-art approaches. To validate the generalizability across cameras, we directly apply the models trained on NYU-Depth-v2 to the SUN RGB-D dataset without any fine-tuning, and the superior results emphasize its strong generalizability.

Existing methods detect the keypoints in a non-differentiable way, therefore they can not directly optimize the position of keypoints through back-propagation. To address this issue, we present a partially differentiable keypoint detection module, which outputs accurate sub-pixel keypoints. The reprojection loss is then proposed to directly optimize these sub-pixel keypoints, and the dispersity peak loss is presented for accurate keypoints regularization. We also extract the descriptors in a sub-pixel way, and they are trained with the stable neural reprojection error loss. Moreover, a lightweight network is designed for keypoint detection and descriptor extraction, which can run at 95 frames per second for 640x480 images on a commercial GPU. On homography estimation, camera pose estimation, and visual (re-)localization tasks, the proposed method achieves equivalent performance with the state-of-the-art approaches, while greatly reduces the inference time.

The generative adversarial network (GAN) is successfully applied to study the perceptual single image super-resolution (SISR). However, since the GAN is data-driven, it has a fundamental limitation on restoring real high frequency information for an unknown instance (or image) during test. On the other hand, the conventional model-based methods have a superiority to achieve instance adaptation as they operate by considering the statistics of each instance (or image) only. Motivated by this, we propose a novel model-based algorithm, which can extract the detail layer of an image efficiently. The detail layer represents the high frequency information of image and it is constituted of image edges and fine textures. It is seamlessly incorporated into the GAN and serves as a prior knowledge to assist the GAN in generating more realistic details. The proposed method, named DSRGAN, takes advantages from both the model-based conventional algorithm and the data-driven deep learning network. Experimental results demonstrate that the DSRGAN outperforms the state-of-the-art SISR methods on perceptual metrics, meanwhile achieving comparable results in terms of fidelity metrics. Following the DSRGAN, it is feasible to incorporate other conventional image processing algorithms into a deep learning network to form a model-based deep SISR.

Unsupervised video object segmentation (UVOS) aims at automatically separating the primary foreground object(s) from the background in a video sequence. Existing UVOS methods either lack robustness when there are visually similar surroundings (appearance-based) or suffer from deterioration in the quality of their predictions because of dynamic background and inaccurate flow (flow-based). To overcome the limitations, we propose an implicit motion-compensated network (IMCNet) combining complementary cues (i.e., appearance and motion) with aligned motion information from the adjacent frames to the current frame at the feature level without estimating optical flows. The proposed IMCNet consists of an affinity computing module (ACM), an attention propagation module (APM), and a motion compensation module (MCM). The light-weight ACM extracts commonality between neighboring input frames based on appearance features. The APM then transmits global correlation in a top-down manner. Through coarse-to-fine iterative inspiring, the APM will refine object regions from multiple resolutions so as to efficiently avoid losing details. Finally, the MCM aligns motion information from temporally adjacent frames to the current frame which achieves implicit motion compensation at the feature level. We perform extensive experiments on$\textit {DAVIS}_{\textit {16}}$and$\textit {YouTube-Objects}$. Our network achieves favorable performance while running at a faster speed compared to the state-of-the-art methods. Our code is available athttps://github.com/xilin1991/IMCNet.

Spatially varying exposure (SVE) is a promising choice for high-dynamic-range (HDR) imaging (HDRI). The SVE-based HDRI, which is called single-shot HDRI, is an efficient solution to avoid ghosting artifacts. However, it is very challenging to restore a full-resolution HDR image from a real-world image with SVE because: a) only one-third of pixels with varying exposures are captured by camera in a Bayer pattern, b) some of the captured pixels are over- and under-exposed. For the former challenge, a spatially varying convolution (SVC) is designed to process the Bayer images carried with varying exposures. For the latter one, an exposure-guidance method is proposed against the interference from over- and under-exposed pixels. Finally, a joint demosaicing and HDRI deep learning framework is formalized to include the two novel components and to realize an end-to-end single-shot HDRI. Experiments indicate that the proposed end-to-end framework avoids the problem of cumulative errors and surpasses the related state-of-the-art methods. Related codes and datasets will be provided athttps://github.com/yilun-xu/SVEHDRI/.

Keypoints matching is a pivotal component for many image-relevant applications such as image stitching, visual simultaneous localization and mapping (SLAM), and so on. Both handcrafted-based and recently emerged deep learning-based keypoints matching methods merely rely on keypoints and local features, while losing sight of other available sensors such as inertial measurement unit (IMU) in the above applications. In this paper, we demonstrate that the motion estimation from IMU integration can be used to exploit the spatial distribution prior of keypoints between images. To this end, a probabilistic perspective of attention formulation is proposed to integrate the spatial distribution prior into the attentional graph neural network naturally. With the assistance of spatial distribution prior, the effort of the network for modeling the hidden features can be reduced. Furthermore, we present a projection loss for the proposed keypoints matching network, which gives a smooth edge between matching and un-matching keypoints. Image matching experiments on visual SLAM datasets indicate the effectiveness and efficiency of the presented method.

Ghosting artifacts are believed to be the Achilles’ heel for high dynamic range (HDR) imaging (HDRI) via differently exposed images sequentially captured by a digital device. Spatially varying exposure (SVE)-based HDRI is an efficient solution to prevent the ghosting artifacts from appearing in a HDR image. However, it is challenging to restore a high-quality HDR image with the full resolution from a single raw Bayer image for the SVE-based HDRI. In this paper, a novel deep convolution neural network (DCNN) is proposed to address such a challenging problem. The proposed DCNN includes two distinctive components, a spatially varying convolution and an exposedness-aware compensation branch. The evaluations indicate that the quality of our results significantly surpasses several related algorithms. Related materials will be provided at https://github.com/yilun-xu/SVEHDRI/.

Self-supervised learning algorithms that compute depth map from monocular videos have achieved remarkable performance on urban scenes and have been applied extensively. These techniques still face significant challenges, however, when applied directly to endoscopic videos because of the brightness variations from frame to frame and inadequate representation learning during the training phase. Inspired by the optical flow for motion alignment between adjacent frames, we design a AFNet with structural stability loss and residual-based smoothness loss to learn the appearance flow across adjacent frames, which handles the brightness inconsistency issue efficaciously. In addition, we propose a novel self-attention mechanism named feature scaling module to alleviate the inadequate representation learning problem. In a comparison study to the current state-of-the-art self-supervised methods explored for urban videos on the SCARED dataset, the developed model surpasses existing methods by a large margin.

With the improvement of sensor technology and significant algorithmic advances, the accuracy of remote heart rate monitoring technology has been significantly improved. Despite of the significant algorithmic advances, the performance of rPPG algorithm can degrade in the long-term, high-intensity continuous work occurred in evenings or insufficient light environments. One of the main challenges is that the lost facial details and low contrast cause the failure of detection and tracking. Also, insufficient lighting in video capturing hurts the quality of physiological signal. In this paper, we collect a largescale dataset that was designed for remote heart rate estimation recorded with various illumination variations to evaluate the performance of the rPPG algorithm (Green, ICA, and POS). We also propose a low-light enhancement solution (technical solution) for remote heart rate estimation under the low-light condition. Using collected dataset, we found 1) face detection algorithm cannot detect faces in video captured in low light conditions; 2) A decrease in the amplitude of the pulsatile signal will lead to the noise signal to be in the dominant position; and 3) the chrominance-based method suffers from the limitation in the assumption about skin-tone will not hold, and Green and ICA method receive less influence than POS in dark illuminance environment. The proposed solution for rPPG process is effective to detect and improve the signal-to-noise ratio and precision of the pulsatile signal.

It is recognized that existing multi-scale exposure fusion algorithms can be improved using edge-preserving smoothing techniques. However, the complexity of edge-preserving smoothing-based multi-scale exposure fusion is an issue for mobile devices. In this paper, a simpler multi-scale exposure fusion algorithm is designed in YUV color space. The proposed algorithm can preserve details in the brightest and darkest regions of a high dynamic range (HDR) scene and the edge-preserving smoothing-based multi-scale exposure fusion algorithm while avoiding color distortion from appearing in the fused image. The complexity of the proposed algorithm is about half of the edge-preserving smoothing-based multi-scale exposure fusion algorithm. The proposed algorithm is thus friendlier to the smartphones than the edge-preserving smoothing-based multi-scale exposure fusion algorithm. In addition, a simple detail-enhancement component is proposed to enhance fine details of fused images. The experimental results show that the proposed component can be adopted to produce an enhanced image with visibly enhanced fine details and a higher MEF-SSIM value. This is impossible for existing detail enhancement components. Clearly, the component is attractive for PC-based applications.

Edge-preserving smoothing based multi-scale exposure fusion is a state-of-the-art method to fuse differently exposed images of a high dynamic range (HDR) scene. However, its complexity could be an issue. In this paper, a novel multiscale exposure fusion algorithm is proposed by adopting an approximation method at the highest layer of the pyramid. Experimental results show that the proposed algorithm can be applied to fuse images with comparable or even better quality with the edge-preserving smoothing based multi-scale fusion algorithms. It simplifies the complexity of the edge-preserving smoothing based multi-scale exposure fusion algorithms significantly.

Classical convolutional neural networks used in computer vision tasks perform excellently in accuracy, but they are unsatisfactory in computational cost especially with the networks going deeper and the image size going larger. Special models based on visual attention have showed their advantages in dealing with spatial information for saving computational cost at inference time. These models are designed to imitate human visual attention mechanism, but they are not able to achieve realize adaptive receptive scope for different object size. In this paper, a recurrent location and scope selection approach is proposed to improve the attention efficiency, which is more similar to human visual mechanism. We evaluate our model on the basic visual recognition task, where it outperforms the baselines and could provide approximated bounding boxes in a weakly supervised way.

Multiscale exposure fusion is a fast approach to fuse several differently exposed images captured at the same high dynamic range (HDR) scene into a high-quality low-dynamic range (LDR) image. The fused image is expected to include all details of the input images. However the details in the brightest and darkest regions are usually not well preserved. Adding details that are extracted from the input images to the fused image is an efficient approach to overcome the problem. In this paper a new gradient domain weighted least square based image smoothing algorithm is proposed to extract the details in the brightest and darkest regions of the HDR scene. The extracted details are then added to an image that is produced using an edge-preserving smoothing pyramid based multiscale exposure fusion algorithm. Experimental results show that the proposed detail enhanced exposure fusion algorithm can preserve details in saturated regions especially the brightest regions better than the state-of-the-art multiscale exposure fusion algorithms.

Symbolic execution is a classic program analysis technique which uses symbolic inputs to explore feasible program paths. It has been widely used in bug detection and test case generation. However, there is only limited success in applying this technique to detect errors in floating-point programs. The ubiquitous, yet complicated to solve, floating-point constraints make it challenging to apply symbolic execution to floating-point exception detection. This paper proposes to accelerate symbolic execution for floating-point exception detection, using value-range analysis. Our insight is that floating-point exceptions rarely happen in real-world programs, and plenty of floating-point constraints can be effectively solved by a much more efficient value-range analysis. The value-range analysis maintains an over-approximated value range for each program variable, which can efficiently filter out those constraints for checking the operations that are guaranteed to be safe. Hence, we perform value-range analysis together with classic symbolic execution for fast floating-point exception detection. Moreover, the value ranges can be used for mathematical function modeling, and further eliminate false positives. Our experimental results show that 15% of constraints can be solved by our value-range analysis. We can also find more bugs and report fewer false positives than the classic symbolic execution technique.

Multi-scale exposure fusion is a fast approach to fuse several differently exposed images captured at the same high dynamic range (HDR) scene into a high quality low dynamic range (LDR) image. The fused image is expected to include all details of the input images, however, the details in the brightest and darkest regions are usually not preserved well. Adding details that are extracted from the input images to the fused image is an efficient approach to overcome the problem. In this paper, a fast selectively detail enhancement algorithm is proposed to extract the details in the brightest and darkest regions of the HDR scene and add the extracted details to the fused image. Experimental results show that the proposed algorithm can enhance the details of the fused image much faster than the existing algorithms with comparable or even better visual quality.

Many existing symbolic execution engines for bug detection often ignore floating-point types and operations. That will result in imprecise reasoning about the feasibility of program paths, which in turn leads to false positives and negatives. Recently, there are quite some progress in satisfiability modulo theories (SMT) solving, and some tools are able to support floating-point arithmetic. Nevertheless, naturally extending a symbolic execution engine and directly replacing the back-end with the new SMT solver will not make a good static analyzer for floating-point programs.In this paper, we extend an existing symbolic execution engine for C program bug finding, so that it can deal with floating-point arithmetic and mathematical functions. For the mathematical functions, we employ an abstract model to keep a balance between overhead and precision. We also introduce a strategy, Lazy-verification, to reduce the number of SMT solver calls. We implemented our approach as a tool called Canalyze-fp. Experiments with self-developed benchmarks and non-trivial open source programs show that the proposed approach can effectively avoid the false positives and negatives, without introducing too much overhead.

In the last few decades, selective visual attention has been extensively studied for its promising contributions to computer vision applications. Many different models have been proposed to compute visual saliency, which can be coarsely formulated as computational or psychophysical. Most existing methods are based on bottom-up mechanism, an automatic human behavior to guide gaze allocation. And low level features such as color, intensity and orientation are commonly adopted to compute saliency map. In this work, we propose a saliency computation method that integrates high-level information of object with low-level features. The result map is more suitable for most top-down tasks in the field of mobile robot requiring object information.

As a preprocessing step of many applications, such as object recognition, image retrieval and scene analysis, saliency detection plays an important role and remains a challenging and significant problem in computer vision. Most existing bottom-up methods utilize local or global contrast information to compute the saliency maps, whereas a few methods generate saliency maps with the use of background cues. This work presents a saliency detection method by applying independent component analysis with reference (ICA-R) algorithm to the background cues, which improves the performance of the final saliency maps. First, we segment the input image into superpixels. Second, we take superpixels on each side of image as reference signals to do ICA-R learning, respectively. Then, four saliency maps generated from the learning algorithm are integrated into one background saliency map. Finally, a graph-based manifold ranking algorithm is done to generate the final saliency maps. By doing experiments on a large publicly available database, we demonstrate that the proposed ICA-R saliency detection algorithm performs better than the state-of-the-art methods.

This paper presents a geometric calibration approach of a permanent magnet (PM) spherical actuator to improve its positioning accuracy. The proposed actuator consists of a ball-shaped rotor with multiple PM poles and a spherical-shell-shaped stator with circumferential air-core coils. Due to manufacturing and assembly restrictions, the actual geometric parameters of the spherical actuator differ from their nominal values. Hence, the identification of such errors is significant for high accuracy motion control. The calibration model is formulated based on the differential form of torque equation. To identify the position vector errors in the magnetization axes of PM poles and coils axes, an iterative least-squares algorithm is employed. The proposed calibration method can also be applied to other PM spherical actuators. To verify the robustness and effectiveness of the proposed calibration algorithm, simulations are conducted on the spherical actuator. The results have shown that the positioning accuracy of the spherical actuator is greatly improved after calibration.

It is difficult to classify object or scene images with high accuracy when the dataset is relatively large. Spatial Pyramid Matching (SPM) was proposed to deal with this problem, but there are some shortages. As an improvement for SPM, we proposed three pieces of meliorations: first, use approximate nearest neighbor method instead of k-means for clustering; second, regulate the size of codebook referring to quantity and pixels of the images, by calculating sub-codebook for every category and eliminating the codes which are nearer to the registered ones than the threshold; third, rescale the histogram features, and classify the scene with hierarchical strategy. Experiments prove that our approach make better performance than other state-of-the-art classification methods using just one matching kernel.

The research area of bio-inspired control methods for multi-legged robots and reptile robots has made significant development with the use of central pattern generators (CPGs) in recent years. However, there are still many problems to be solved to learn clearly the structure of CPG to adapt it to different applications. In this article, we use a method to configure CPG which makes the CPG have adaptive frequency according to the existing researches. Thus, the frequency of CPG can be changed automatically via the feedback of external limit cycle driving signals. Also, we try to explain the CPG of adaptive frequency from a dynamical way. Then we construct a new CPG method for locomotion control of quadruped robots. Finally we make simulations to verify the CPG locomotion control method on a quadruped model in the software of Adams.

In this paper we present an approach to indoor localization and 3D scene reconstruction using the Microsoft Kinect sensor. The proposed system can simultaneously estimates the position and orientation of a hand-held Kinect and generates a dense 3D model of the indoor environment. Furthermore, the robustness and processing time for four different feature descriptors (SURF, ORB, Shi-Tomasi and FAST) are evaluated. The experiment results demonstrate that our system can robustly deal with complicated data in common indoor scenarios while running in semi-real-time.

To solve the problem that the final velocity curve will be unsmoothed if the PVT(Position Velocity Time) nodes velocities are determined improperly when using PVT control algorithm, based on analyzing the general principle of spline interpolation and PVT motion control, this paper presents a triple spline interpolation based PVT algorithm, which ensures the smoothness of the velocity curve. The velocity of each node is firstly calculated via spline interpolation method, and then the final trace control curve is achieved by using PVT method to these points. The proposed algorithm is applied to the kinematics control of a cockroach robot. The structure and forward/inverse kinematics model of the cockroach robot is expounded. Finally, the reliability of the proposed algorithm is verified by the contrast simulation and experiment of a single leg of the cockroach robot with one of the existing algorithms. The results show that the proposed approach can be readily used for motion controls.