In the weakly supervised temporal video grounding study, previous methods use predetermined single Gaussian proposals which lack the ability to express diverse events described by the sentence query. To enhance the expression ability of a proposal, we propose a Gaussian mixture proposal (GMP) that can depict arbitrary shapes by learning importance, centroid, and range of every Gaussian in the mixture. In learning GMP, each Gaussian is not trained in a feature space but is implemented over a temporal location. Thus the conventional feature-based learning for Gaussian mixture model is not valid for our case. In our special setting, to learn moderately coupled Gaussian mixture capturing diverse events, we newly propose a pull-push learning scheme using pulling and pushing losses, each of which plays an opposite role to the other. The effects of components in our scheme are verified in-depth with extensive ablation studies and the overall scheme achieves state-of-the-art performance. Our code is available at https://github.com/sunoh-kim/pps.

We propose a dual-domain generative model to estimate a texture map from a single image for colorizing a 3D human model. When estimating a texture map, a single image is insufficient as it reveals only one facet of a 3D object. To provide sufficient information for estimating a complete texture map, the proposed model simultaneously generates multi-view hallucinations in the image domain and an estimated texture map in the texture domain. During the generating process, each domain generator exchanges features to the other by a flow-based local attention mechanism. In this manner, the proposed model can estimate a texture map utilizing abundant multi-view image features from which multiview hallucinations are generated. As a result, the estimated texture map contains consistent colors and patterns over the entire region. Experiments show the superiority of our model for estimating a directly render-able texture map, which is applicable to 3D animation rendering. Furthermore, our model also improves an overall generation quality in the image domain for pose and viewpoint transfer tasks.

In many visual applications, we often encounter people with irregular postures, such as lying down. Many approaches adopted two-step methods to handle a person with irregular postures: 1) person detection and 2) posture prediction based on the detected person. However, it is challenging to detect irregular postures because the existing detectors were trained with datasets consisting of most upright postures. Therefore, we propose a new Irregular Human Posture (IHP) dataset to handle various postures captured from real-world surveillance cameras. The IHP dataset provides sufficient annotations to understand the posture of person, including segmentation, keypoints, and postural states. This paper also provides two baseline net-works for postural state estimation of the people trained on the IHP dataset. Moreover, we show that our baseline networks effectively detect the people with irregular postures that may be in an urgent situation in a surveillance environment.

Much progress has been made for non-rigid structure from motion (NRSfM) during the last two decades, which made it possible to provide reasonable solutions for synthetically-created benchmark data. In order to utilize these NRSfM techniques in more realistic situations, however, we are now facing two important problems that must be solved: First, general scenes contain complex deformations as well as multiple objects, which violates the usual assumptions of previous NRSfM proposals. Second, there are many unreconstructable regions in the video, either because of the discontinued tracks of 2D trajectories or those regions static towards the camera, which require careful manipulations. In this paper, we show that a consensus-based reconstruction framework can handle these issues effectively. Even though the entire scene is complex, its parts usually have simpler deformations, and even though there are some unreconstructable parts, they can be weeded out to reduce their harmful effect on the entire reconstruction. The main difficulty of this approach lies in identifying appropriate parts, however, it can be effectively avoided by sampling parts stochastically and then aggregate their reconstructions afterwards. Experimental results show that the proposed method renews the state-of-the-art for popular benchmark data under much harsher environments, i.e., narrow camera view ranges, and it can reconstruct video-based real-world data effectively for as many areas as it can without an elaborated user input.

A well-defined deformation model can be vital for non-rigid structure from motion (NRSfM). Most existing methods restrict the deformation space by assuming a fixed rank or smooth deformation, which are not exactly true in the real world, and they require the degree of deformation to be predetermined, which is impractical. Meanwhile, the errors in rotation estimation can have severe effects on the performance, i.e., these errors can make a rigid motion be misinterpreted as a deformation. In this paper, we propose an alternative to resolve these issues, motivated by an observation that non-rigid deformations, excluding rigid changes, can be concisely represented in a linear subspace without imposing any strong constraints, such as smoothness or low-rank. This observation is embedded in our new prior distribution, the Procrustean normal distribution (PND), which is a shape distribution exclusively for non-rigid deformations. Because of this unique characteristic of the PND, rigid and non-rigid changes can be strictly separated, which leads to better performance. The proposed algorithm, EM-PND, fits a PND to given 2D observations to solve NRSfM without any user-determined parameters. The experimental results show that EM-PND gives the state-of-the-art performance for the benchmark data sets, confirming the adequacy of the new deformation model.

Recently, there have been many progresses for the problem of non-rigid structure reconstruction based on 2D trajectories, but it is still challenging to deal with complex deformations or restricted view ranges. Promising alternatives are the piecewise reconstruction approaches, which divide trajectories into several local parts and stitch their individual reconstructions to produce an entire 3D structure. These methods show the state-of-the-art performance, however, most of them are specialized for relatively smooth surfaces and some are quite complicated. Meanwhile, it has been reported numerously in the field of pattern recognition that obtaining consensus from many weak hypotheses can give a strong, powerful result. Inspired by these reports, in this paper, we push the concept of part-based reconstruction to the limit: Instead of considering the parts as explicitly-divided local patches, we draw a large number of small random trajectory sets. From their individual reconstructions, we pull out a statistic of each 3D point to retrieve a strong reconstruction, of which the procedure can be expressed as a sparse l1-norm minimization problem. In order to resolve the reflection ambiguity between weak (and possibly bad) reconstructions, we propose a novel optimization framework which only involves a single eigenvalue decomposition. The proposed method can be applied to any type of data and outperforms the existing methods for the benchmark sequences, even though it is composed of a few, simple steps. Furthermore, it is easily parallelizable, which is another advantage.

In this paper we propose an efficient method for the recognition of long and complex action streams. First, we design a new motion feature flow descriptor by composing low-level local features. Then a new data embedding method is developed in order to represent the motion flow as an one-dimensional sequence, whilst preserving useful motion information for recognition. Finally attentional motion spots (AMSs) are defined to automatically detect meaningful motion changes from the embedded one-dimensional sequence. An unsupervised learning strategy based on expectation maximization and a weighted Gaussian mixture model is then applied to the AMSs for each action class, resulting in an action representation which we refer to as Action Chart. The Action Chart is then used efficiently for recognizing each action class. Through comparison with the state-of-the-art methods, experimental results show that the Action Chart gives promising recognition performance with low computational load and can be used for abstracting long video sequences.

Non-rigid structure from motion is a fundamental problem in computer vision, which is yet to be solved satisfactorily. The main difficulty of the problem lies in choosing the right constraints for the solution. In this paper, we propose new constraints that are more effective for non-rigid shape recovery. Unlike the other proposals which have mainly focused on restricting the deformation space using rank constraints, our proposal constrains the motion parameters so that the 3D shapes are most closely aligned to each other, which makes the rank constraints unnecessary. Based on these constraints, we define a new class of probability distribution called the Procrustean normal distribution and propose a new NRSfM algorithm, EM-PND. The experimental results show that the proposed method outperforms the existing methods, and it works well even if there is no temporal dependence between the observed samples.

With various sensors in a smart phone, it is now possible to obtain information about a user and her surroundings, such as the location of a smartphone and the activity of the smartphone user, and the obtained context information is being used to provide new services to the users. In this paper, we propose VibePhone, which uses a built-in vibrator and accelerometer, for recognizing the type of surfaces contacted by a smart phone, enabling the sense of touch in smartphones. For humans and animals, the sense of touch is fundamental for both recognizing and learning the properties of objects. The sense of touch is obtained from the texture of an object and humans recognize the type of an object by scrubbing the surface with fingers. Since a smartphone cannot physically scrub the contacting surface, we emulate the touch by generating vibrations using a smartphone and propose a method to recognize the type of contacting objects. The recognition of the object type by vibration alone is an extremely difficult task, even for a human. However, we demonstrate that it is possible to distinguish object types into broad categories where a phone is usually placed, e.g., sofas, plastic tables, wooden tables, hands, backpacks, and pants pockets. The proposed VibePhone system achieves an accuracy over 85% on average. We have prototyped VibePhone on an Android-based smart phone which changes its background display based on the contacting surface. We envision that the haptic perception in future smart phones will enable new experiences to the users.

We propose a novel communication method between smart devices using a built-in vibrator and accelerometer. The proposed approach is ideal for low-rate private communication and its communication medium is an object on which smart devices are placed, such as tables and desks. When more than two smart devices are placed on an object and one device wants to transmit a message to the other devices, the transmitting device generates a sequence of vibrations. The vibrations are propagated through the object on which devices are placed. The receiving devices analyze their accelerometer readings to decode incoming messages. Unlike radio-based wireless communication where eavesdropping of private communication is possible without the knowledge of the user, the proposed method can guarantee privacy as long as the object used for communication is secured. The proposed method is implemented on Android smart phones and comprehensive experiments are conducted to show its feasibility.