Recently, various material recognition methods have been introduced that use a single color or light field camera. In prior methods, color and texture information of an object are used as key features. However, there exists fundamental limitation in using color features for material recognition in that material type can be characterized better by surface reflectance, visual appearance rather than its color and textures. In this work, we propose IR surface reflectance based material type recognition method. We use off-the-shelf ToF camera to estimate the IR reflectance of arbitrary surface. Material type recognition is performed on both color and surface IR reflectance features. Several network structures including gradual convolutional neural network are proposed and verified for our material recognition within our own 3D datasets.

Recently, action recognition methods using graph convolutional networks (GCN) have shown remarkable performance thanks to its concise but effective representation of human body motion. Prior methods construct human body motion graph building edges between neighbor or distant body joints. On the other hand, human action contains lots of temporal variations showing strong temporal correlations between joint motions. Thus the characterization of an action requires a comprehensive analysis of joint motion correlations on spatial and temporal domains. In this paper, we propose Joint-temporal Motion Graph Convolutional Networks (JT-MGCN) in which joint-temporal edges learn the correlations between different joints at different time. Experimental evaluation on large public data sets such as NTU rgb+d data set and kinetics-skeleton data set show outstanding action recognition performance.

In this paper, we propose a measurement-based modeling framework for hyper-elastic material identification and real-time haptic rendering. We build a custom data collection setup that captures shape deformation and response forces during compressive deformation of cylindrical material samples. We collected training and testing sets of data from four silicone objects having various material profiles. We design an objective function for material parameter identification by incorporating both shape deformation and reactive forces and utilize a genetic algorithm. We adopted an optimization-based Finite Element Method (FEM) for object deformation rendering. The numerical error of simulated forces was found to be perceptually negligible.

Translucency introduces great challenges to 3-D acquisition because of complicated light behaviors such as refraction and transmittance. In this paper, we describe the development of a unified 3-D data acquisition framework that reconstructs translucent objects using a single commercial time-of-flight (ToF) camera. In our capture scenario, we record a depth map and intensity image of the scene twice using a static ToF camera; first, we capture the depth map and intensity image of an arbitrary background, and then we position the translucent foreground object and record a second depth map and intensity image with both the foreground and the background. As a result of material characteristics, the translucent object yields systematic distortions in the depth map. We developed a new distance representation that interprets the depth distortion induced as a result of translucency. By analyzing ToF depth sensing principles, we constructed a distance model governed by the level of translucency, foreground depth, and background depth. Using an analysis-by-synthesis approach, we can recover the 3-D geometry of a translucent object from a pair of depth maps and their intensity images. Extensive evaluation and case studies demonstrate that our method is effective for modeling the nonlinear depth distortion due to translucency and for reconstruction of a 3-D translucent object.

Symmetry is a salient visual pattern in images. Symmetrical structure attracts human eye more than other regions. Therefore, detecting symmetry in an image is one of the crucial tasks in pattern recognition and computer vision research. Sparse key point based symmetry detection methods have been proposed which are fast and robust to noise showing superior detection performance. However, such local appearance-based methods have difficulties in capturing structure based patterns mostly supported by edges and contours. In this paper, we propose a scale invariant structure feature which describes points on extremum curvature along edges. We propose to use a histogram of curvature responses at respective scale space for description. Experimental evaluation on public shape dataset and real world images show that our structure feature works better in detecting visually salient structure based symmetry patterns.

Femur 3D reconstruction from x-ray images enables cost-effective and precise diagnosis of disease and surgery guidance. However, reconstructing accurate 3D model from sparse x-ray images are not a trivial task. In this paper, we propose a femur 3D reconstruction method from two x-ray images. We incorporate prior knowledge on the femur deformation such as twist distortion to improve reconstruction accuracy. We deform our generic 3D model optimizing global registration and twist correction parameters simultaneously. And then, non-rigid deformation is performed constructing dense 3d femur model of a patient. We have tested our method on real x-ray images and synthesized 3d femur models for quantitative and qualitative evaluations.

Dance is a dynamic art form that reflects a wide range of cultural diversity and individuality. With the advancement of motion-capture technology combined with crowd-sourcing and machine learning algorithms, we explore the complex relationship between perceived dance quality/dancer's gender and dance movements/music respectively. As a feasibility study, we construct a computational framework for an analysis-synthesis-feedback loop using a novel multimedia dance-music texture representation. Furthermore, we integrate crowd-sourcing, music and motion-capture data, and machine learning-based methods for dance segmentation, analysis and synthesis of new dancers. A quantitative validation of this framework on a motion-capture dataset of 172 dancers evaluated by more than 400 independent on-line raters demonstrates significant correlation between human perception and the algorithmically intended dance quality or gender of synthesized dancers. The technology illustrated in this work has a high potential to advance the multimedia entertainment industry via dancing with Turks.

Recently, many consumer time-of-flight depth cameras have been introduced that provide direct 3-D geometry measurement of real world objects in real-time. However, these cameras suffer from motion blur artifact when there is any movement of camera or target object in the scene causing serious geometry measurement distortions. Unlike other noises, depth camera motion blur is difficult to eliminate using any existing image processing method due to the unique principle of time-of-flight depth calculation. In this letter, we propose a novel depth motion blur detection and deblurring method that can be applied for any ToF depth sensor. Our method utilizes the relations between different phase offsets observed at multiple time slots in ToF sensor. Experimental results show that the proposed method successfully detects motion blur regions and accurately eliminates them with minimal computational cost in real time.

The framework of capturing multiview images has received a great attention from industry as well as academy due to its usability in various applications; 3D scene reconstruction, super-resolution, scene understanding, surveillance etc. This paper presents a multiview capturing system that uses a single camera with multiple mirrors. We place multiple mirrors pointing to the target scene and record an image of those mirrors. From this image of mirrors, we can extract the multiview images of the target scene, images reflected by mirrors. Based on the proposed method, we achieve some success in geometry reconstruction and view synthesis using multiview images, which shows the usability of the proposed work. The main contribution of this work is to introduce a light weight implementation for multiview capturing with a number of useful properties; the automatic synchronization across multiple views, the consistent color/white balance setting across multiple views, the capability of recording a wide range of view and the cost-effective implementation.

Time-of-Flight cameras provide real-time depth information at each pixel. These cameras, however, suffer from modular errors, recording shortened distance measurements if the objects are farther than a certain maximum range. Previous error-correction methods use either a single depth map or successively acquired multi-frequency depth maps. In general, they are prone to incorrect range extension results when the scene is discontinuous or when the camera or objects are moving. Our method based on a stereo Time-of-Flight camera deals with moving cameras and objects by acquiring a pair of depth maps simultaneously. Challenges caused by noise and different viewpoints are resolved by an iterative Markov random field optimization technique that delivers the consistent recovery of actual depth values beyond the maximum range. The experiments show that the proposed method extends the range use of Time-of-Flight cameras beyond twice.

Time-of-flight (ToF) and structured light depth cameras have got increasing interest for their ability of direct 3D geometry acquisition. Robotics, computer vision and graphics researchers have tried to employ depth camera for their applications such as robot navigation, gesture recognition, 3d reconstruction, etc. Depth cameras, however, suffer from lots of unique noises such as range ambiguity, scattering and motion blur. For instance, the depth motion blur cannot be correctly removed by any conventional deblurring method yielding serious distortions in 3D reconstruction. In this paper, we introduce various systematic and non-systematic depth errors and state of the art enhancement methods.

Photorealism has been one of essential goals for virtual reality. The state-of-the-art techniques employ various rendering algorithms to simulate physically accurate light transport for generating the photorealistic appearance of scene. However, they still require a labor-intensive tone mapping and color tunes by an experienced artist. In this paper, we propose an automatic photorealism enhancement algorithm by manipulating the color distribution of graphics to match with that of real photographs. Based on the hypothesis that photorealism is highly correlated with the frequency of color characteristics appearing in real photographs, we find principal color components. Then, we transfer the statistical characteristics of photographs onto graphics so to enhance their photorealism. Experiments and a user study have confirmed the effectiveness of proposed method.

We generalize the concept of bilateral reflection symmetry to curved glide-reflection symmetry in 2D euclidean space, such that classic reflection symmetry becomes one of its six special cases. We propose a local feature-based approach for curved glidereflection symmetry detection from real, unsegmented 2D images. Furthermore, we apply curved glide-reflection axis detection for curved reflection surface detection in 3D images. Our method discovers, groups, and connects statistically dominant local glidereflection axes in an Axis-Parameter-Space (APS) without preassumptions on the types of reflection symmetries. Quantitative evaluations and comparisons against state-of-the-art algorithms on a diverse 64-test-image set and 1,125 Swedish leaf-data images show a promising average detection rate of the proposed algorithm at 80 and 40 percent, respectively, and superior performance over existing reflection symmetry detection algorithms. Potential applications in computer vision, particularly biomedical imaging, include saliency detection from unsegmented images and quantification of deviations from normality. We make our 64-test-image set publicly available.

An approximately Euclidean representation of the visible scene can be obtained directly from a range, or `time-of-flight', camera. An uncalibrated binocular system, in contrast, gives only a projective reconstruction of the scene. This paper analyzes the geometric mapping between the two representations, without requiring an intermediate calibration of the binocular system. The mapping can be found by either of two new methods, one of which requires point-correspondences between the range and colour cameras, and one of which does not. It is shown that these methods can be used to reproject the range data into the binocular images, which makes it possible to associate high-resolution colour and texture with each point in the Euclidean representation.

We propose a bilateral symmetry Gabor Wavelet Network (BSGWN) for face modeling and online tracking. Bilateral symmetry wavelet pairs reduce the computational complexity of the BSGWN optimization and yield robust tracking performance in real world environments. We decompose face image into a set of frequency components. The face model of Gabor wavelets is built on the reconstructed image from corresponding frequency bands of wavelets used. The frequency guidance allows fast and accurate face tracking by removing potential clutters from other frequency bands. We verify the proposed algorithm on real video sequences including various challenging face conditions.

We present a bi-layer inpainting method for synthesizing novel views from a single color image and its corresponding depth map under the exemplar-based inpainting framework. Unlike conventional image inpainting, the decisions of the inpainting direction and the sample regions are important to inpaint disoccluded regions which disclose hidden background regions in the new viewpoint. The proposed algorithm first labels boundaries along the disoccluded regions whether it belongs to the foreground or background objects, and then separates their surrounding regions into the foreground and background regions using the graph cut algorithm. The disoc-cluded regions are then filled from the background boundary with best-match patches taken from the background regions. As demonstrated in the experimental results, the proposed method recovers the disoccluded region with visually plausible quality.

We present a novel and effective algorithm for affinely skewed rotation symmetry group detection from real-world images. We define a complete skewed rotation symmetry detection problem as discovering five independent properties of a skewed rotation symmetry group: 1) the center of rotation, 2) the affine deformation, 3) the type of the symmetry group, 4) the cardinality of the symmetry group, and 5) the supporting region of the symmetry group in the image. We propose a frieze-expansion (FE) method that transforms rotation symmetry group detection into a simple, 1D translation symmetry detection problem. We define and construct a pair of rotational symmetry saliency maps, complemented by a local feature method. Frequency analysis, using Discrete Fourier Transform (DFT), is applied to the frieze-expansion patterns (FEPs) to uncover the types (cyclic, dihedral, and O2), the cardinalities, and the corresponding supporting regions, concentric or otherwise, of multiple rotation symmetry groups in an image. The phase information of the FEP is used to rectify affinely skewed rotation symmetry groups. Our result advances the state of the art in symmetry detection by offering a unique combination of region-based, feature-based, and frequency-based approaches. Experimental results on 170 synthetic and natural images demonstrate superior performance of our rotation symmetry detection algorithm over existing methods.

We generalize reflection symmetry detection to a curved glide reflection symmetry detection problem. We propose a unifying, local feature based approach for curved glide reflection symmetry detection from real, unsegmented images, where the classic reflection symmetry becomes one of four special cases. Our method detects and groups statistically dominant local reflection axes in a 3D parameter space. A curved glid reflection symmetry axis is estimated by a set of contiguous local straight reflection axes. Experimental results of the proposed algorithm on 40 real world images demonstrate promising performance.

We present a novel and effective algorithm for rotation symmetry group detection from real-world images. We propose a frieze-expansion method that transforms rotation symmetry group detection into a simple translation symmetry detection problem. We define and construct a dense symmetry strength map from a given image, and search for potential rotational symmetry centers automatically. Frequency analysis, using discrete Fourier transform (DFT), is applied to the frieze-expansion patterns to uncover the types and the cardinality of multiple rotation symmetry groups in an image, concentric or otherwise. Furthermore, our detection algorithm can discriminate discrete versus continuous and cyclic versus dihedral symmetry groups, and identify the corresponding supporting regions in the image. Experimental results on over 80 synthetic and natural images demonstrate superior performance of our rotation detection algorithm in accuracy and in speed over the state of the art rotation detection algorithms.

Symmetry is one of the important cues for human and machine perception of the world. For over three decades, automatic symmetry detection from images/patterns has been a standing topic in computer vision. We present a timely, systematic, and quantitative performance evaluation of three state of the art discrete symmetry detection algorithms. This evaluation scheme includes a set of carefully chosen synthetic and real images presenting justified, unambiguous single or multiple dominant symmetries, and a pair of well-defined success rates for validation. We make our 176 test images with associated hand-labeled ground truth publicly available with this paper. In addition, we explore the potential contribution of symmetry detection for object recognition by testing the symmetry detection algorithm on three publicly available object recognition image sets (PASCAL VOC'07, MSRC and Caltech-256). Our results indicate that even after several decades of effort, symmetry detection in real-world images remains a challenging, unsolved problem in computer vision. Meanwhile, we illustrate its future potential in object recognition.

Gait is an attractive biometric for vision-based human identification. Previous work on existing public data sets has shown that shape cues yield improved recognition rates compared to pure motion cues. However, shape cues are fragile to gross appearance variations of an individual, for example, walking while carrying a ball or a backpack. We introduce a novel, spatiotemporal shape variation-based frieze pattern (SVB frieze pattern) representation for gait, which captures motion information over time. The SVB frieze pattern represents normalized frame difference over gait cycles. Rows/columns of the vertical/horizontal SVB frieze pattern contain motion variation information augmented by key frame information with body shape. A temporal symmetry map of gait patterns is also constructed and combined with vertical/horizontal SVB frieze patterns for measuring the dissimilarity between gait sequences. Experimental results show that our algorithm improves gait recognition performance on sequences with and without gross differences in silhouette shape. We demonstrate superior performance of this computational framework over previous algorithms using shape cues alone on both CMU MoBo and UoS HumanID gait databases.