Personal robots and driverless cars need to be able to operate in novel environments and thus quickly and efficiently learn to recognise new object classes. We address this problem by considering the task of video object segmentation. Previous accurate methods for this task finetune a model using the first annotated frame, and/or use additional inputs such as optical flow and complex post-processing. In contrast, we develop a fast, causal algorithm that requires no finetuning, auxiliary inputs or post-processing, and segments a variable number of objects in a single forward-pass. We represent an object with clusters, or "visual words", in the embedding space, which correspond to object parts in the image space. This allows us to robustly match to the reference objects throughout the video, because although the global appearance of an object changes as it undergoes occlusions and deformations, the appearance of more local parts may stay consistent. We learn these visual words in an unsupervised manner, using meta-learning to ensure that our training objective matches our inference procedure. We achieve comparable accuracy to finetuning based methods (whilst being 1 to 2 orders of magnitude faster), and state-of-the-art in terms of speed/accuracy trade-offs on four video segmentation datasets. Code is available at https://github.com/harkiratbehl/MetaVOS.

Telerobotic manipulation allows patients living with upper limb impairments to interact with a variety of environments and accomplish through teleoperation daily activities such as playing, feeding, self-care, and leisure, that would otherwise be difficult to perform. In this paper, we propose a nonlinear mapping between the patient's range of motion and the workspace of an environment being manipulated. The objective is to identify the patient's workspace and span it to that of the environment or an object, thus optimizing the scaling factor while soliciting the entire patient's range of motion. The boundaries of each workspace are obtained from scattered measurements of the master and slave robots end-effector position. The nonlinear mapping is then achieved through thin plate spline interpolation that describes deformation between two surfaces by scattered point-to-point preponderances. Experimental results reported in three different scenarios confirm the suitability of the nonlinear transformation to map diverse workspace volumes.

Scene parsing has attracted a lot of attention in computer vision. While parametric models have proven effective for this task, they cannot easily incorporate new training data. By contrast, nonparametric approaches, which bypass any learning phase and directly transfer the labels from the training data to the query images, can readily exploit new labeled samples as they become available. Unfortunately, because of the computational cost of their label transfer procedures, state-of-the-art nonparametric methods typically filter out most training images to only keep a few relevant ones to label the query. As such, these methods throw away many images that still contain valuable information and generally obtain an unbalanced set of labeled samples. In this paper, we introduce a nonparametric approach to scene parsing that follows a sample-andfilter strategy. More specifically, we propose to sample labeled superpixels according to an image similarity score, which allows us to obtain a balanced set of samples. We then formulate label transfer as an efficient filtering procedure, which lets us exploit more labeled samples than existing techniques. Our experiments evidence the benefits of our approach over state-of-the-art nonparametric methods on two benchmark datasets.

Exploiting multiple modalities for semantic scene parsing has been shown to improve accuracy over the single modality scenario. Existing methods, however, assume that corresponding regions in two modalities have the same label. In this paper, we address the problem of data misalignment and label inconsistencies, e.g., due to moving objects, in semantic labeling, which violate the assumption of existing techniques. To this end, we formulate multimodal semantic labeling as inference in a CRF, and introduce latent nodes to explicitly model inconsistencies between two domains. These latent nodes allow us not only to leverage information from both domains to improve their labeling, but also to cut the edges between inconsistent regions. To eliminate the need for hand tuning the parameters of our model, we propose to learn intra-domain and inter-domain potential functions from training data. We demonstrate the benefits of our approach on two publicly available datasets containing 2D imagery and 3D point clouds. Thanks to our latent nodes and our learning strategy, our method outperforms the state-of-the-art in both cases.

In this paper, we introduce a multi-modal graphical model to address the problems of semantic segmentation using 2D-3D data exhibiting extensive many-to-one correspondences. Existing methods often impose a hard correspondence between the 2D and 3D data, where the 2D and 3D corresponding regions are forced to receive identical labels. This results in performance degradation due to misalignments, 3D-2D projection errors and occlusions. We address this issue by defining a graph over the entire set of data that models soft correspondences between the two modalities. This graph encourages each region in a modality to leverage the information from its corresponding regions in the other modality to better estimate its class label. We evaluate our method on a publicly available dataset and beat the state-of-the-art. Additionally, to demonstrate the ability of our model to support multiple correspondences for objects in 3D and 2D domains, we introduce a new multi-modal dataset, which is composed of panoramic images and LIDAR data, and features a rich set of many-to-one correspondences.

The focus of this work is addressing the challenges of performing object recognition in real world scenes as captured by a commercial, state-of-the-art, surveying vehicle equipped with a 360° panoramic camera in conjunction with a 3D laser scanner (LIDAR). Even with state-of-the-art surveying equipment, there is colour saturation and very dark regions in images, as well as some degree of time-varying misalignment between the point cloud data and imagery due to, for instance, imperfect tracking of sensor pose. Moreover, there are frequent occlusions due to both static and moving objects. These issues are inherently difficult to avoid and therefore need to be dealt with in a more robust fashion. This is where the contribution of the paper is; that is, the development of a consensus method that can intelligently incorporate feature responses from multiple views and reject those that are not very descriptive. It is shown that the overall performance in a ten class problem is increased from 70.5% for a simple 2D-3D classification system, to 77.5%. Subsequently, an enhanced CRF which has become robust using the misclassifications of training data and equipped with the probabilities of the adjacent points, was applied to the system and further improved its performance to 82.9%. The experiments were performed on a challenging dataset captured both in summer and winter.

A big challenge in sensorless image-based ultrasound tracking is in the out-of-plane motion estimation. The correlation value of a specific model of speckle known as fully developed speckle (FDS) can be used to estimate the out-of-plane displacement. In real tissue, this kind of pattern is rare and the deviation of speckle pattern from the ideal FDS model diminishes the accuracy of the out-of-plane motion estimation. In this paper a new method for estimation of the out-of-plane motion is proposed. Firstly a closed-form mathematical derivation is provided for the correlation of two RF echo signal patches at different positions. A linear regression model of the ultrasound beam profile is proposed to account for the spatial variability of the ultrasound beam and enhance the accuracy of out-of-plane motion estimation in real tissue. The statistical model of speckle used here is based on the Rician-Inverse Gaussian (RiIG) stochastic process of the speckle formation, which can be considered as a generalized form of the K-distribution with richer parametrization. In this work, for the first time the second-order statistics of the RIG model is used for speckle tracking. This statistical model allows for derivation of a closed-form formulation for the correlation coefficient based on the statistical parameters of every patch. Since the effect of coherency is considered in the RiIG model, it increases the reliability of the out-of-plane motion estimation. The flexibility of the proposed method enables almost any patch through the whole image to be used for the purpose of displacement estimation. The method has been evaluated both on ex vivo and in vivo tissues in various experiments including out-of-plane rotation (tilt, yaw) and free-hand imaging. The overall outcome demonstrates the potential of the proposed method for in vivo tissues.

In this paper, a new enhanced CRF for discriminating between different materials in natural scenes using terrestrial multi spectral imaging is established. Most of the existing formulations of the CRF often suffer from over smoothing and loss of small detail, thereby deteriorating the information from the underlying unary classifier in areas with a high spatial frequency. This work specifically addresses this issue by incorporating a new pairwise potential that is better at taking local context into account. Certain materials are very unlikely to appear next to each other in the scene and such configurations are penalised by employing the confusion matrix of the unary classifier. Similarly, horizontal as well as vertical configurations, which may be more or less likely for certain combinations of materials, are regarded in this formulation. Furthermore, the proposed pairwise potential also considers the length of boundaries between regions to account for the segmentation granularity issues and also uses class probabilities of the neighbouring regions to make up for the uncertainty of the unary classifier results. Seven band terrestrial multi spectral imaging were used due to its potential in distinguishing between different materials and objects. The proposed approach was evaluated using cross-validation, resulting in an average accuracy of 88.9% which is about 17% more than the accuracy of a standard CRF, which demonstrates the superiority of our approach in preserving local details.