We propose a framework for creating articulated human avatars, editing their styles, and animating the human avatars from three different types of text instructions. The three types of instructions, identity, edit, and action, are fed into three models that generate, edit, and animate human avatars. Specifically, the proposed framework takes identity instruction and multi-view pose condition images to generate the images of a human using the avatar generation model. Then, the avatar can be edited with text instructions by changing the style of the images generated. We apply the Neural Radiance Field (NeRF) and Poisson reconstruction to extract a human mesh model from images and assign linear blend skinning (LBS) weights to the vertices. Finally, the action instructions can animate human avatars, where we use the off-the-shelf method to generate the motions from text instructions. Notably, our proposed method adapts the appearance of hundreds of different individuals to construct a conditionally editable avatar-generated model, allowing easy creation of 3D avatars using text instructions. We demonstrate high-fidelity 3D animatable avatar creation with text instructions on various datasets and highlight a superior performance of the proposed method compared to the previous studies.

High-quality 3D human body reconstruction requires high-fidelity and large-scale training data and appropriate network design that effectively exploits the high-resolution input images. To tackle these problems, we propose a simple yet effective 3D human digitization method called 2K2K, which constructs a large-scale 2K human dataset and infers 3D human models from 2K resolution images. The proposed method separately recovers the global shape of a human and its details. The low-resolution depth network predicts the global structure from a low-resolution image, and the part-wise image-to-normal network predicts the details of the 3D human body structure. The high-resolution depth network merges the global 3D shape and the detailed structures to infer the high-resolution front and back side depth maps. Finally, an off-the-shelf mesh generator reconstructs the full 3D human model, which are available at https://github.com/SangHunHan92/2K2K. In addition, we also provide 2,050 3D human models, including texture maps, 3D joints, and SMPL parameters for research purposes. In experiments, we demonstrate competitive performance over the recent works on various datasets.

We propose an unsupervised approach to predict depth maps from images captured by a wireless endoscopic capsule. Recent advances in deep learning have shown that accurate depth maps can be predicted from a single image, where the deep network is trained via unsupervised or self-supervised learning by using monocular video sequences or stereo image pairs. However, directly applying these techniques to endoscopic images does not yield satisfactory results owing to the inherent difficulties of the wireless capsule imaging such as dim lighting and low-resolution of images, which are different from normal imaging conditions. For that reason, we exploit the environmental characteristics of endoscopic images - there is no external light source except ones attached to the capsule. Based on this condition, we propose the direct attenuation model-based depth map prediction scheme to guide depth prediction and to add meaningful cues to the loss function. We experimentally verify the proposed method with various endoscopic images.

The detection of dynamic points on a moving platform is an important task to avoid a potential collision. However, it is difficult to detect dynamic points using only two frames, especially when various input data such as ego-motion, disparity map, and optical flow are noisy for computing the motion of points. In this paper, we propose a supervised learning-based approach to detect dynamic points in consideration of noisy input data. First of all, to consider depth ambiguity that proportionally increases according to the distance to the ego-vehicle, we divide the XZ-plane (bird-eye view) into several subregions. Then, we train a random forest for each subregion by constructing motion vectors computed based on two motion metrics. Here, in order to reduce errors of the input data, the motion vectors are filtered based on a pairwise planarity check and then filtered motion vectors are used for training. In the experiments, the proposed method is verified by comparing the detection performance with that of previous approaches on the KITTI dataset.

Multi-object tracking (MOT) becomes more challenging when objects of interest have similar appearances. In that case, the motion cues are particularly useful for discriminating multiple objects. However, for online 2D MOT in scenes acquired from moving cameras, observable motion cues are complicated by global camera movements and thus not always smooth or predictable. To deal with such unexpected camera motion for online 2D MOT, a structural motion constraint between objects has been utilized thanks to its robustness to camera motion. In this paper, we propose a new data association method that effectively exploits structural motion constraints in the presence of large camera motion. In addition, to further improve the robustness of data association against mis-detections and false positives, a novel event aggregation approach is developed to integrate structural constraints in assignment costs for online MOT. Experimental results on a large number of datasets demonstrate the effectiveness of the proposed algorithm for online 2D MOT.

A robust algorithm is proposed for tracking a target object in dynamic conditions including motion blurs, illumination changes, pose variations, and occlusions. To cope with these challenging factors, multiple trackers based on different feature representations are integrated within a probabilistic framework. Each view of the proposed multiview (multi-channel) feature learning algorithm is concerned with one particular feature representation of a target object from which a tracker is developed with different levels of reliability. With the multiple trackers, the proposed algorithm exploits tracker interaction and selection for robust tracking performance. In the tracker interaction, a transition probability matrix is used to estimate dependencies between trackers. Multiple trackers communicate with each other by sharing information of sample distributions. The tracker selection process determines the most reliable tracker with the highest probability. To account for object appearance changes, the transition probability matrix and tracker probability are updated in a recursive Bayesian framework by reflecting the tracker reliability measured by a robust tracker likelihood function that learns to account for both transient and stable appearance changes. Experimental results on benchmark datasets demonstrate that the proposed interacting multiview algorithm performs robustly and favorably against state-of-the-art methods in terms of several quantitative metrics.

Online multi-object tracking with a single moving camera is a challenging problem as the assumptions of 2D conventional motion models (e.g., first or second order models) in the image coordinate no longer hold because of global camera motion. In this paper, we consider motion context from multiple objects which describes the relative movement between objects and construct a Relative Motion Network (RMN) to factor out the effects of unexpected camera motion for robust tracking. The RMN consists of multiple relative motion models that describe spatial relations between objects, thereby facilitating robust prediction and data association for accurate tracking under arbitrary camera movements. The RMN can be incorporated into various multi-object tracking frameworks and we demonstrate its effectiveness with one tracking framework based on a Bayesian filter. Experiments on benchmark datasets show that online multi-object tracking performance can be better achieved by the proposed method.

Recently, mobile devices such as smart phones and quad-copters are being equipped with inertial measurement units (IMUs) because of advances in micro-electro-mechanical systems technology. This has increased the importance of IMU- camera fusion for vision-based applications. However, ultralow-cost IMUs take much less accurate measurements than low-cost and high-cost IMUs. This uncertainty degrades the accuracy and reliability of IMU-camera calibration, which is the most important step for IMU-camera fusion technology. In this paper, we propose three effective algorithms for robust IMU- camera calibration with uncertain measurements: boundary constraint, adaptive prediction, and angular velocity constraint. These algorithms incorporate a Bayesian filtering framework to estimate calibration parameters more efficiently. The experimental results on both simulation and real data demonstrated the superiority of the proposed algorithms.

In this letter, we propose a robust dynamic point clustering method for detecting moving objects in stereo image sequences, which is essential for collision detection in driver assistance system. If multiple objects with similar motions are located in close proximity, dynamic points from different moving objects may be clustered together when using the position and velocity as clustering criteria. To solve this problem, we apply a geometric constraint between dynamic points using line segments. Based on this constraint, we propose a variable K-nearest neighbor clustering method and three cost functions that are defined between line segments and points. The proposed method is verified experimentally in terms of its accuracy, and comparisons are also made with conventional methods that only utilize the positions and velocities of dynamic points.

In this paper, we propose a novel multi-object tracking method to track unknown number of objects with a single camera system. We design the tracking method via probability hypothesis density (PHD) filtering which considers multiple object states and their observations as random finite sets (RFSs). The PHD filter is capable of rejecting clutters, handling object appearances and disappearances, and estimating the trajectories of multiple objects in a unified framework. Although the PHD filter is robust to cluttered environment, it is vulnerable to missed detections. For this reason, we include local observations in an RFS of observation model. Local observations are locally generated near the individual tracks by using on-line trained local detector. The main purpose of the local observation is to handle the missed detections and to provide identity (label information) to each object in filtering procedure. The experimental results show that the proposed method robustly tracks multiple objects under practical situations.