By moving a depth sensor around a room, we compute a 3D CAD model of the environment, capturing the room shape and contents such as chairs, desks, sofas, and tables. Rather than reconstructing geometry, we match, place, and align each object in the scene to thousands of CAD models of objects. In addition to the fully automatic system, the key technical contribution is a novel approach for aligning CAD models to 3D scans, based on deep reinforcement learning. This approach, which we call Learning-based ICP, outperforms prior ICP methods in the literature, by learning the best points to match and conditioning on object viewpoint. LICP learns to align using only synthetic data and does not require ground truth annotation of object pose or keypoint pair matching in real scene scans. While LICP is trained on synthetic data and without 3D real scene annotations, it outperforms both learned local deep feature matching and geometric based alignment methods in real scenes. The proposed method is evaluated on real scenes datasets of SceneNN and ScanNet as well as synthetic scenes of SUNCG. High quality results are demonstrated on a range of real world scenes, with robustness to clutter, viewpoint, and occlusion.

Given a single photo of a room and a large database of furniture CAD models, our goal is to reconstruct a scene that is as similar as possible to the scene depicted in the photograph, and composed of objects drawn from the database. We present a completely automatic system to address this IM2CAD problem that produces high quality results on challenging imagery from interior home design and remodeling websites. Our approach iteratively optimizes the placement and scale of objects in the room to best match scene renderings to the input photo, using image comparison metrics trained via deep convolutional neural nets. By operating jointly on the full scene at once, we account for inter-object occlusions. We also show the applicability of our method in standard scene understanding benchmarks where we obtain significant improvement.

We introduce Segment-Phrase Table (SPT), a large collection of bijective associations between textual phrases and their corresponding segmentations. Leveraging recent progress in object recognition and natural language semantics, we show how we can successfully build a high-quality segment-phrase table using minimal human supervision. More importantly, we demonstrate the unique value unleashed by this rich bimodal resource, for both vision as well as natural language understanding. First, we show that fine-grained textual labels facilitate contextual reasoning that helps in satisfying semantic constraints across image segments. This feature enables us to achieve state-of-the-art segmentation results on benchmark datasets. Next, we show that the association of high-quality segmentations to textual phrases aids in richer semantic understanding and reasoning of these textual phrases. Leveraging this feature, we motivate the problem of visual entailment and visual paraphrasing, and demonstrate its utility on a large dataset.

A scene category imposes tight distributions over the kind of objects that might appear in the scene, the appearance of those objects and their layout. In this paper, we propose a method to learn scene structures that can encode three main interlacing components of a scene: the scene category, the context-specific appearance of objects, and their layout. Our experimental evaluations show that our learned scene structures outperform state-of-the-art method of Deformable Part Models in detecting objects in a scene. Our scene structure provides a level of scene understanding that is amenable to deep visual inferences. The scene structures can also generate features that can later be used for scene categorization. Using these features, we also show promising results on scene categorization.

We evaluate the performance of a widely used tracking-by-detection and data association multi-target tracking pipeline applied to an activity-rich video dataset. In contrast to traditional work on multi-target pedestrian tracking where people are largely assumed to be upright, we use an activity-rich dataset that includes a wide range of body poses derived from actions such as picking up an object, riding a bike, digging with a shovel, and sitting down. For each step of the tracking pipeline, we identify key limitations and offer practical modifications that enable robust multi-target tracking over a range of activities. We show that the use of multiple posture-specific detectors and an appearance-based data association post-processing step can generate non-fragmented trajectories essential for holistic activity understanding.

In this paper, we propose a novel method that exploits correlation between audio-visual dynamics of a video to segment and localize objects that are the dominant source of audio. Our approach consists of a two-step spatiotemporal segmentation mechanism that relies on velocity and acceleration of moving objects as visual features. Each frame of the video is segmented into regions based on motion and appearance cues using the QuickShift algorithm, which are then clustered over time using K-means, so as to obtain a spatiotemporal video segmentation. The video is represented by motion features computed over individual segments. The Mel-Frequency Cepstral Coefficients (MFCC) of the audio signal, and their first order derivatives are exploited to represent audio. The proposed framework assumes there is a non-trivial correlation between these audio features and the velocity and acceleration of the moving and sounding objects. The canonical correlation analysis (CCA) is utilized to identify the moving objects which are most correlated to the audio signal. In addition to moving-sounding object identification, the same framework is also exploited to solve the problem of audio-video synchronization, and is used to aid interactive segmentation. We evaluate the performance of our proposed method on challenging videos. Our experiments demonstrate significant increase in performance over the state-of-the-art both qualitatively and quantitatively, and validate the feasibility and superiority of our approach.

Standard Evolution Strategy (ES) produces the next generation via the Gaussian mutation that is not directed toward the optimum. Additionally, self-adaptation mechanism is used in the standard ES to adapt mutation step-size. This paper presents a new evolution strategy which is called Quantum-inspired Evolution Strategy (QES). QES applies a new learning mechanism whereby the information of the mutants is used as a feedback to adapt the mutation direction and step-size simultaneously. To demonstrate the effectiveness of the proposed method, several experiments on a set of numerical optimization problems are carried out and the results are compared with the standard ES and Covariance Matrix Adaptation ES (CMA-ES) which is the state-of-the-art method for adaptive mutation. The results reveal that QES is superior to standard ES and CMA-ES in terms of convergence speed and accuracy.

Active contour models are widely used in extracting object boundaries. However, most of these methods usually fail to capture concave boundaries properly and impose high computational cost. In this paper, a new SOM-based active contour model which introduces the Conscience and Archiving mechanisms (CASOM) is proposed to extend the Batch SOM method and eliminate its deficiencies. The performance of the proposed method is evaluated by some experiments on a set of grayscale images. Experimental results are compared with those of the BSOM in terms of accuracy and convergence speed. The results reveal that compared to BSOM, the proposed method requires less computations for converging to the object boundaries and extracts the boundaries of complex objects more accurately, even in the presence of weak or broken edges.

Generalized Hough Transform (GHT) is an efficient method for detecting curves by exploiting the duality between points on a curve and parameters of that curve. However GHT has some practical limitations such as high computational cost and huge memory requirement for detecting scaled and rotated objects. In this paper a new method, namely fuzzy generalized Hough transform (FGHT), is proposed that alleviates these deficiencies by utilizing the concept of fuzzy inference system. In FGHT the R-table consists of a set of fuzzy rules which are fired by the gradient direction of edge pixels and vote for the possible location of the center. Moreover, the proposed method can identify the boundary of the rotated and scaled object via a new voting strategy. To evaluate the effectiveness of FGHT several experiments with scaled, rotated, occluded and noisy images are conducted. The results are compared with two extensions of GHT and have revealed that the proposed method can locate and detect the prototype object with least error under various conditions.

The natural immune system is composed of cells and molecules with complex interactions. Jerne modeled the interactions among immune cells and molecules by introducing the immune network. The immune system provides an effective defense mechanism against foreign substances. This system like the neural system is able to learn from experience. In this paper, the Jerne's immune network model is extended and a new classifier based on the new immune network model and Learning Vector Quantization (LVQ) is proposed. The new classification method is called Hybrid Fuzzy Neuro-Immune Network based on Multi-Epitope approach (HFNINME). The performance of the proposed method is evaluated via several benchmark classification problems and is compared with two other prominent immune-based classifiers. The experiments reveal that the proposed method yields a parsimonious classifier that can classify data more accurately and more efficiently.