We propose a framework for creating articulated human avatars, editing their styles, and animating the human avatars from three different types of text instructions. The three types of instructions, identity, edit, and action, are fed into three models that generate, edit, and animate human avatars. Specifically, the proposed framework takes identity instruction and multi-view pose condition images to generate the images of a human using the avatar generation model. Then, the avatar can be edited with text instructions by changing the style of the images generated. We apply the Neural Radiance Field (NeRF) and Poisson reconstruction to extract a human mesh model from images and assign linear blend skinning (LBS) weights to the vertices. Finally, the action instructions can animate human avatars, where we use the off-the-shelf method to generate the motions from text instructions. Notably, our proposed method adapts the appearance of hundreds of different individuals to construct a conditionally editable avatar-generated model, allowing easy creation of 3D avatars using text instructions. We demonstrate high-fidelity 3D animatable avatar creation with text instructions on various datasets and highlight a superior performance of the proposed method compared to the previous studies.

High-quality 3D human body reconstruction requires high-fidelity and large-scale training data and appropriate network design that effectively exploits the high-resolution input images. To tackle these problems, we propose a simple yet effective 3D human digitization method called 2K2K, which constructs a large-scale 2K human dataset and infers 3D human models from 2K resolution images. The proposed method separately recovers the global shape of a human and its details. The low-resolution depth network predicts the global structure from a low-resolution image, and the part-wise image-to-normal network predicts the details of the 3D human body structure. The high-resolution depth network merges the global 3D shape and the detailed structures to infer the high-resolution front and back side depth maps. Finally, an off-the-shelf mesh generator reconstructs the full 3D human model, which are available at https://github.com/SangHunHan92/2K2K. In addition, we also provide 2,050 3D human models, including texture maps, 3D joints, and SMPL parameters for research purposes. In experiments, we demonstrate competitive performance over the recent works on various datasets.

We propose an unsupervised approach to predict depth maps from images captured by a wireless endoscopic capsule. Recent advances in deep learning have shown that accurate depth maps can be predicted from a single image, where the deep network is trained via unsupervised or self-supervised learning by using monocular video sequences or stereo image pairs. However, directly applying these techniques to endoscopic images does not yield satisfactory results owing to the inherent difficulties of the wireless capsule imaging such as dim lighting and low-resolution of images, which are different from normal imaging conditions. For that reason, we exploit the environmental characteristics of endoscopic images - there is no external light source except ones attached to the capsule. Based on this condition, we propose the direct attenuation model-based depth map prediction scheme to guide depth prediction and to add meaningful cues to the loss function. We experimentally verify the proposed method with various endoscopic images.

Recent advances in deep learning have significantly improved the performance of video prediction, however, top-performing algorithms start to generate blurry predictions as they attempt to predict farther future frames. In this paper, we propose a unified generative adversarial network for predicting accurate and temporally consistent future frames over time, even in a challenging environment. The key idea is to train a single generator that can predict both future and past frames while enforcing the consistency of bi-directional prediction using the retrospective cycle constraints. Moreover, we employ two discriminators not only to identify fake frames but also to distinguish fake contained image sequences from the real sequence. The latter discriminator, the sequence discriminator, plays a crucial role in predicting temporally consistent future frames. We experimentally verify the proposed framework using various real-world videos captured by car-mounted cameras, surveillance cameras, and arbitrary devices with state-of-the-art methods.

We present a robust approach for computing disparity maps with a supervised learning-based confidence prediction. This approach takes into consideration following features. First, we analyze the characteristics of various confidence measures in the random forest framework to select effective confidence measures depending on the characteristics of the training data and matching strategies, such as similarity measures and parameters. We then train a random forest using the selected confidence measures to improve the efficiency of confidence prediction and to build a better prediction model. Second, we present a confidence-based matching cost modulation scheme, based on predicted confidence values, to improve the robustness and accuracy of the (semi-) global stereo matching algorithms. Finally, we apply the proposed modulation scheme to popularly used algorithms to make them robust against unexpected difficulties that could occur in an uncontrolled environment using challenging outdoor datasets. The proposed confidence measure selection and cost modulation schemes are experimentally verified from various perspectives using the KITTI and Middlebury datasets.

Accurate stereo matching for depth extraction requires a large memory space, which restricts its use in resource-limited systems. The problem is aggravated by the recent trend of applications requiring significantly high pixel resolution and disparity levels. To alleviate the high memory requirement, we propose to represent the aggregation costs as a Gaussian mixture model (GMM) function. Only a set of GMM parameters is stored and used instead of all the costs for each pixel. We also propose GMM parameter update-based aggregation along multiple paths. To preserve the accuracy of the disparity map, we employ a depth confidence measure and propose an update rule for the slanted surface of an object. Experimental results over the KITTI dataset show that the proposed method reduces the memory requirement to less than 5% of that of semiglobal matching, while the accuracy is maintained at the level of state-of-the-art semiglobal and local methods.

In this paper, we propose a novel method to jointly solve scene layout estimation and global registration problems for accurate indoor 3D reconstruction. Given a sequence of range data, we first build a set of scene fragments using KinectFusion and register them through pose graph optimization. Afterwards, we alternate between layout estimation and layout-based global registration processes in iterative fashion to complement each other. We extract the scene layout through hierarchical agglomerative clustering and energy-based multi-model fitting in consideration of noisy measurements. Having the estimated scene layout in one hand, we register all the range data through the global iterative closest point algorithm where the positions of 3D points that belong to the layout such as walls and a ceiling are constrained to be close to the layout. We experimentally verify the proposed method with the publicly available synthetic and real-world datasets in both quantitative and qualitative ways.

The detection of dynamic points on a moving platform is an important task to avoid a potential collision. However, it is difficult to detect dynamic points using only two frames, especially when various input data such as ego-motion, disparity map, and optical flow are noisy for computing the motion of points. In this paper, we propose a supervised learning-based approach to detect dynamic points in consideration of noisy input data. First of all, to consider depth ambiguity that proportionally increases according to the distance to the ego-vehicle, we divide the XZ-plane (bird-eye view) into several subregions. Then, we train a random forest for each subregion by constructing motion vectors computed based on two motion metrics. Here, in order to reduce errors of the input data, the motion vectors are filtered based on a pairwise planarity check and then filtered motion vectors are used for training. In the experiments, the proposed method is verified by comparing the detection performance with that of previous approaches on the KITTI dataset.

We propose a new approach to associate supervised learning-based confidence prediction with the stereo matching problem. First of all, we analyze the characteristics of various confidence measures in the regression forest framework to select effective confidence measures using training data. We then train regression forests again to predict the correctness (confidence) of a match by using selected confidence measures. In addition, we present a confidence-based matching cost modulation scheme based on the predicted correctness for improving the robustness and accuracy of various stereo matching algorithms. We apply the proposed scheme to the semi-global matching algorithm to make it robust under unexpected difficulties that can occur in outdoor environments. We verify the proposed confidence measure selection and cost modulation methods through extensive experimentation with various aspects using KITTI and challenging outdoor datasets.

In this letter, we propose a robust dynamic point clustering method for detecting moving objects in stereo image sequences, which is essential for collision detection in driver assistance system. If multiple objects with similar motions are located in close proximity, dynamic points from different moving objects may be clustered together when using the position and velocity as clustering criteria. To solve this problem, we apply a geometric constraint between dynamic points using line segments. Based on this constraint, we propose a variable K-nearest neighbor clustering method and three cost functions that are defined between line segments and points. The proposed method is verified experimentally in terms of its accuracy, and comparisons are also made with conventional methods that only utilize the positions and velocities of dynamic points.