We propose a new method for modeling the indoor scene from a single color image. With our system, the user only needs to drag a few semantic bounding boxes surrounding the objects of interest. Our system then automatically finds the most similar 3D models from the ShapeNet model repository and aligns them with the corresponding objects of interest. To achieve this, each 3D model is represented as a group of view-dependent representations generated from a set of synthesized views. We iteratively conduct object segmentation and 3D model retrieval, based on the observation that good segmentation of the objects of interest can significantly improve the accuracy of model retrieval and make it robust to cluttered background and occlusions, and in turn, the retrieved 3D models can be used to assist with object segmentation. Segmentation of all objects of interest is achieved simultaneously under a unified multi-labeling framework which fully utilizes the correspondences between the objects of interest and retrieved model images. Besides, we propose a new method to estimate the scene layout of the input image with the segmentation masks, which helps compose the resulting scene and further improves the modeling result remarkably. We verify the effectiveness of our approach through experimenting with a variety of indoor images and comparing against the relevant methods.

We propose a single image-based 3D model retrieval method for indoor scenes. By simulating the scene context of the input image, our method is able to handle several challenging scenarios featuring cluttered backgrounds and severe occlusions. To use our system, the user only needs to drag a few semantic bounding boxes for the query objects. The proposed approach then retrieves the most similar 3D models from the ShapeNet model repository, and aligns them with the corresponding objects automatically. This requires that the 3D models are represented by calibrated view-dependent visual elements learned from the rendered views. With the estimated occlusion relationships, the rendered model images are stacked at the corresponding locations to simulate the scene context. By conducting matching between these synthesized scenes and the input image, the most similar 3D models under the approximate poses are retrieved. Moreover, we show that the retrieving time can be significantly reduced based on a novel greedy algorithm. Experimental results demonstrate the effectiveness of our proposed method.

We present in this paper an interactive approach for semantically modeling indoor environments given only a single indoor image as input, without requiring access to the scene or using any additional measurements like the RGBD cameras. Our key insight is that, although depth estimation from a single image is notoriously difficult, we can conveniently obtain a relatively accurate normal map, which essentially conveys a great deal of scene geometry. This enables us to model each object in a data-driven manner by representing the object as a normal-based graph and retrieving a similar model from the database by graph matching. We hypothesize a set of sparse surface orientations for the image, and further refine them in an intuitive and straightforward manner. With a small amount of simple user interaction, our approach is able to generate a plausible model of the scene. To verify the effectiveness of our proposed method, we show the modeling results on a variety of indoor images.

In this paper, we present a novel approach for automatically creating the photo collage that assembles the interest regions of a given group of images naturally. Previous methods on photo collage are generally built upon a well-defined optimization framework, which computes all the geometric parameters and layer indices for input photos on the given canvas by optimizing a unified objective function. The complex nonlinear form of optimization function limits their scalability and efficiency. From the geometric point of view, we recast the generation of collage as a region partition problem such that each image is displayed in its corresponding region partitioned from the canvas. The core of this is an efficient power-diagram-based circle packing algorithm that arranges a series of circles assigned to input photos compactly in the given canvas. To favor important photos, the circles are associated with image importances determined by an image ranking process. A heuristic search process is developed to ensure that salient information of each photo is displayed in the polygonal area resulting from circle packing. With our new formulation, each factor influencing the state of a photo is optimized in an independent stage, and computation of the optimal states for neighboring photos are completely decoupled. This improves the scalability of collage results and ensures their diversity. We also devise a saliency-based image fusion scheme to generate seamless compositive collage. Our approach can generate the collages on nonrectangular canvases and supports interactive collage that allows the user to refine collage results according to his/her personal preferences. We conduct extensive experiments and show the superiority of our algorithm by comparing against previous methods.

Abstract Optimization of images with bad compositions has attracted increasing attention in recent years. Previous methods however seldomly consider image similarity when improving composition aesthetics. This may lead to significant content changes or bring large distortions, resulting in an unpleasant user experience. In this paper, we present a new algorithm for improving image composition aesthetics, while retaining faithful, as much as possible, to the original image content. Our method computes an improved image using a unified model of composition aesthetics and image similarity. The term of composition aesthetics obeys the rule of thirds and aims to enhance image composition. The similarity term in contrast penalizes image difference and distortion caused by composition adjustment. We use an edge‐based measure of structure similarity which nearly coincides with human visual perception to compare the optimized image with the original one. We describe an effective scheme to generate the optimized image with the objective model. Our algorithm is able to produce the recomposed images with minimal visual distortions in an elegant and user controllable manner. We show the superiority of our algorithm by comparing our results with those by previous methods.