We propose a new orientation-aware scene descriptor ReLoc-Aligner for re-localization of a 3D point cloud. Re-localization within a 3D point cloud map is crucial for conducting Simultaneous Localization and Mapping (SLAM). Existing re-localization or place recognition methods of 3D LiDAR sensor data aim to estimate the current position of the sensor robustly to orientation changes. However, they do not determine the current orientation of the sensor within a 3D point cloud map, which limits their applications to re-localization or loop closing in SLAM. On the other hand, existing methods capable of orientation estimation tend to be slower than them. Our scene descriptor has a property of orientation awareness that enables us to extract the orientation difference between two scans directly from the descriptor. This is useful for the registration of point clouds from a good initial estimate, which leads to better re-localization of a scan. We propose a training method for the new descriptor. In addition, we develop fast querying and re-localization methods using the descriptors. Intensive experiments demonstrate that the proposed method is superior to the existing state-of-the-art methods in both place recognition and orientation estimation.

The COVID-19 pandemic has caused a global public health crisis, leading to increased costs for operating essential quarantine facilities and risks of infection for medical staff. To address these issues, we have developed a non-contact delivery robot, UTD-pro, that can deliver food and supplies to patients in an isolated residential treatment center. In case of emergency situations such as communication failure with the control system or malfunction of specific modules, the noncontact delivery robot should be able to perform commands robustly or return home. Otherwise, as before, medical staffs may have to enter the high-risk environment again. In this paper, we propose a flexible and robust remote control system and a robot task management system that enables the robot to execute commands correctly. Our robot task manager allows for autonomous task execution with a single command from the remote control system. Each robot task operates independently, allowing users to change task plans flexibly during robot operation. The sensor data communication and task execution communication processes are also independent, preventing any issue in one process from affecting the other. By applying our system to the robot, we conducted a long-term delivery experiment for a total of 7.723km for 79 days, and achieved 125 successes out of 149 trials. This experiment proves that our system can lead to stable delivery processes and contribute to the high reliability of the control system and the robot task manager.

For a service robot to assist humans, it should interact with objects of varying sizes and shapes existing in an indoor environment. 3D object detection must be preceded to achieve this goal since it provides the robot with the ability to perceive visual information. Most of the existing methods are anchor-based and predict the bounding box close to the ground truth among multiple candidates. However, it is complex to compute Intersection over Union (IoU) and Non-Maximum Suppression (NMS) per each anchor box. Therefore, we propose keypoint-based monocular 3D object detection, where each object's center location is only needed for reproducing predicted 3D bounding box without extra computation of the anchor boxes. Our 3D object detection also works well even if images are rotated corresponding to the robot's head movement. To properly train our network, the object center is based on a projected 3D location instead of 2D to take advantage of the exact center position of the object. Furthermore, we apply data augmentation using a perspective transformation. The method facilitates adding a small perturbation to the camera rotation angle randomly. We use the SUN RGB-D dataset, which has images taken indoor scenes with camera rotations for training and test set. Our approach particularly shows that the errors of object center location based on a single image reduce 15.4% and 24.2%, respectively, compared to the method without data augmentation.

We present a method to manipulate virtual objects which are constrained complexly and reconfigurable as if they would exist in a real world by using multiple hands simultaneously. A complexly constrained, and reconfigurable object, such as a Rubik's cube, is hard to describe its physical motion constraints, mainly because they are determined by the grasping situation and dynamically changeable. Rather than describing the physical motion constraints in a general form, we more focus on the multiple hand interaction of a complex object. A complex object is divided into multiple subparts which are grasped by each hand, and the constraints between the subparts are optimized for inducing natural and continuous movement. For this, we propose a dynamically adjustable data structure for representing object parts grasped by multiple hands, and an optimization-based pose estimation of the constrained subparts along with their grasped hands. The experiments show that human subjects can manipulate a complexly constrained object such as a Rubik's cube without any difficulty as if it exists in the real-world.

We propose a direct object manipulation system that a user can manipulate objects constrained to each other by his or her hands. Because there is no haptic feedback to a user, manipulation of constrained objects is much harder than that of a simple rigid body. In addition, many physics simulator usually suffers from instability in handling the constrained objects in a virtual system. We solve the problem by adding another micro simulator which enforce the constraint between the objects, and guarantees that the rendered image of an object and its physics counterpart always satisfy the given constraint. The experiment shows that the proposed method makes a user manipulate constrained objects with no difficulties in the way that he or she manipulates such objects in the real world.

We aim to automatically identify all individual leaves in a monocotyledon image. Leaf identification is one of key technologies to acquire plant phenotypes such as a leaf length, a leaf count, and a growth rate. However, it is challenging to identify individual leaves from a monocotyledonous plant image due to their complicated occlusion and similar colors. We adopt a graph-theoretical approach to overcome the leaf occlusion and the color similarity between leaves, and apply a combinatorial optimization technique to simultaneously identify all individual leaves based on local and global characteristics of plants. We propose a plant-wide global optimization to identify all true leaves in a plant, which is formulated as a minimum path cover problem that we call the phytomorphological graph reconstruction. A phytomorphological graph of a plant image is a graph reflecting the plant's structure and is constructed through a plant region extraction and a skeletonization. All candidate leaf paths in the graph are extracted and their leaf path likelihoods are computed from the graph, where the leaf path likelihood means a measure of similarity with true leaves. The candidate leaf paths and their path likelihoods are used in the graph reconstruction process in order to find an optimal subset of the leaf paths which are similar to true leaves and can restore the phytomorphological graph as completely as possible. Experiments show that our proposed system effectively identifies individual leaves from a single 2-dimensional image of rice plants, which enables us to take an accurate and efficient high-throughput measurement of phenotypes for plants.

We propose a method to induce a realistic two-hand manipulation of virtual objects by representing their reactions to multiple contacts of hands and fingers in a unified way. One of the key problems in achieving physics-based virtual interaction is that there is no accurate haptic feedback to the user's hands and fingers, which must introduce a mismatch between the real hands and their virtual counterparts. This problem becomes more serious when multiple hands or fingers are involved to a single virtual object, for example, a hand-over situation from one hand to the other. The key contribution of this paper is to represent and describe the physical status of a virtual object in the local object space and to change the status in the object point of view in order to avoid its drastic changes by the physical-virtual mismatch. The experiments compare the performance of the proposed method with objects in different shapes when using 2D and 3D visualization devices.

This paper describes a new methodology for estimating the relative pose between two 2-D lidars. Scanned points of 2-D lidars do not have enough feature information for correspondence matching. For this reason, additional image sensors or artificial landmarks at known locations have been used to find the relative pose. We propose a novel method of estimating the relative pose between 2-D lidars without any additional sensors or artificial landmarks. By scanning two orthogonal planes, we utilize the coplanarity of the scan points on each plane and the orthogonality of the plane normals. Even if we capture planes which are not exactly orthogonal, the method provides good results using nonlinear optimization. Experiments with both synthetic and real data show the validity of the proposed method. We also derive two degenerate cases: one related to plane poses, and the other caused by the relative pose. To the best of our knowledge, this study provides the first solution for the problem.

We propose an intuitive user interaction system with virtual objects by directly using user's hands through a physics simulation. This system aims to provide a more direct way to manipulate and interact with virtual objects and to induce more realistic reactions of the virtual objects in the interaction than the existing methods. The key contribution of this paper is to model the deformations of the user's hands in two ways: global deformation by the hand poses and the local deformation by contacts with objects. By the two deformation models, objects can be manipulated with better stability and robustness without any predefinition of interaction methods depending on the object shapes, by solely using a physics simulation. The experiment shows that the proposed method enables a user to intuitively manipulate virtual object accurately.

We present a method to register a camera-type sensor and a flat panel display to each other to build a 3D physical user interface. In the system, the sensor can not observe the display directly, which makes the registration non-trivial. We develop an auxiliary device containing a camera which can see the display and a plane pattern which can be seen by the sensor to make a transformation loop between sensors and the display. By using the transformation loop, the relative pose between the sensor and the display can be linearly estimated without any user interaction. Experiments show that the proposed method is accurate enough to build a 3D physical user interface using a commercial display and a low-cost sensor.

This paper describes a new methodology for estimating a relative pose of two 2D laser sensors. Two dimensional laser scan points do not have enough feature information for motion tracking. For this reason, additional image sensors or artificial landmarks have been used to find a relative pose. We propose the method to estimate a relative pose of 2D laser sensors without any additional sensor or artificial landmark. By scanning two orthogonal planes, we utilize only the coplanarity of the scan points on each plane and the orthogonality of the plane normals. Experiments with both synthetic and real data show the validity of the proposed method. To the best of our knowledge this works provides the first solution for the problem.

Object representation is useful for many computer vision tasks, such as object detection, recognition, and tracking. Computer vision tasks must handle situations where unknown objects appear and must detect and track some object which is not in the trained database. In such cases, the system must learn or, otherwise derive, descriptions of new objects. In this paper, we investigate creating a representation of previously unknown objects that newly appear in the scene. The representation creates a viewpoint-invariant and scale-normalized model approximately describing an unknown object with multimodal sensors. Those properties of the representation facilitate 3D tracking of the object using 2D-to-2D image matching. The representation has both benefits of an implicit model (referred to as a view-based model) and an explicit model (referred to as a shape-based model). Experimental results demonstrate the viability of the proposed representation and outperform the existing approaches for 3D-pose estimation.

This paper presents a convenient self-calibration method for an inertial measurement unit (IMU) using matrix factorization. Using limited information about applied loads (accelerations or angular rates) available from natural references, the proposed method can linearly solve all the parameters of an IMU in any configuration of its inertial components. Our factorization-based calibration method exploits the bilinear form of an IMU measurement, which is the product of intrinsic calibration parameters and exerted loads. For a redundant IMU, we prove that partial knowledge of the loads, such as magnitude, can produce a linear solution space for a proper decomposition of the measurement. Theoretical analysis on this linear space reveals that a 1-D null space should be considered when load magnitudes are all equal (e.g., gravity loads). Degenerate load distributions are also geometrically identified to avoid singular measurement collection. Since a triad IMU has a lower number of sensor components than a 4-D parameter space, we propose an iterative factorization in which only initial bias is required. A wide convergence region of the bias can provide an automatic setting of the initial bias as the mean of the measurements. Performance of the proposed method is evaluated with respect to various noise levels and constraint types. Self-calibration capability is demonstrated using natural references, which are gravity for accelerometers and image stream from an attached camera for gyroscopes. Calibration results are globally optimal and identical to those of nonlinear optimization.

We address the problem of 6D pose estimation of a textureless and shiny object from single-view 2D images, for a bin-picking task. For a textureless object like a mechanical part, conventional visual feature matching usually fails due to the absence of rich texture features. Hierarchical template matching assumes that few templates can cover all object appearances. However, the appearance of a shiny object largely depends on its pose and illumination. Furthermore, in a bin-picking task, we must cope with partial occlusions, shadows, and inter-reflections.

By the laws of quantum physics, pixel intensity does not have a true value, but should be a random variable. Contrary to the conventional assumptions, the distribution of intensity may not be an additive Gaussian. We propose to directly model the intensity difference and show its validity by an experimental comparison to the conventional additive model. As a model of the intensity difference, we present a Skellam distribution derived from the Poisson photon noise model. This modeling induces a linear relationship between intensity and Skellam parameters, while conventional variance computation methods do not yield any significant relationship between these parameters under natural illumination. The intensity-Skellam line is invariant to scene, illumination, and even most of camera parameters. We also propose practical methods to obtain the line using a color pattern and an arbitrary image under natural illumination. Because the Skellam parameters that can be obtained from this linearity determine a noise distribution for each intensity value, we can statistically determine whether any intensity difference is caused by an underlying signal difference or by noise. We demonstrate the effectiveness of this new noise model by applying it to practical applications of background subtraction and edge detection.

In this paper, we present an accurate and robust 6D SLAM method that uses multiple 2D sensors, i.e. perspective cameras and planar laser scanners. We have investigated strengths and weaknesses of those two sensors for 6D SLAM by conducting specifically designed experiments, and found that the sensors can complement each other. In order to take full advantages of each approach, we fuse correspondences of those two sensors, rather than individually estimated motions. Correspondences obtained by the two sensors have different characteristics, but can be expressed in a common 2D-3D relation form. We use the correspondences in a single structure from-motion framework. In the initial motion estimation step, we propose a RANSAC-based method to generate and test multiple motion hypotheses by using multiple pools of correspondences, aiming to avoid potential bias of each sensor data. In the later motion refinement step, we introduce a variant of bundle adjustment to consider different types of constraints from the two sensors. The performance of the proposed method is demonstrated both quantitatively by experiments on closed loop sequences and qualitatively by large-scale experiments with DGPS trajectory. The proposed method successfully closes a loop of 320 meters in twenty thousand frames by incremental process only.

We propose a novel inertial-aided KLT feature tracking method robust to camera ego-motions. The conventional KLT uses images only and its working condition is inherently limited to small appearance change between images. When big optical flows are induced by a camera-ego motion, an inertial sensor attached to the camera can provide a good prediction to preserve the tracking performance. We use a low-grade MEMS-based gyroscope to refine an initial condition of the nonlinear optimization in the KLT. It increases the possibility for warping parameters to be in the convergence region of the KLT. For longer tracking with less drift, we use the affine photometric model and it can effectively deal with camera rolling and outdoor illumination change. Extra computational cost caused by this higher-order motion model is alleviated by restraining the Hessian update and GPU acceleration. Experimental results are provided for both indoor and outdoor scenes and GPU implementation issues are discussed.

An imaging sensor made of multiple light-weight non-overlapping cameras is an effective sensor for a small unmanned aerial vehicle that has strong payload limitation. This paper presents a method for motion estimation by assuming that such a multi-camera system is a spherical imaging system (that is, the cameras share a single optical center). We derive analytically and empirically a condition for a multi-camera system to be modeled as a spherical camera. Interestingly, not only does the spherical assumption simplify the algorithms and calibration procedure, but also motion estimation based on that assumption becomes more accurate.

In this paper, we introduce the Skellam distribution as a sensor noise model for CCD or CMOS cameras. This is derived from the Poisson distribution of photons that determine the sensor response. We show that the Skellam distribution can be used to measure the intensity difference of pixels in the spatial domain, as well as in the temporal domain. In addition, we show that Skellam parameters are linearly related to the intensity of the pixels. This property means that the brighter pixels tolerate greater variation of intensity than the darker pixels. This enables us to decide automatically whether two pixels have different colors. We apply this modeling to detect the edges in color images. The resulting algorithm requires only a confidence interval for a hypothesis test, because it uses the distribution of image noise directly. More importantly, we demonstrate that without conventional Gaussian smoothing the noise model-based approach can automatically extract the fine details of image structures, such as edges and corners, independent of camera setting.

Plane-based calibration is now a very popular procedure because of its flexibility. One key step consists in detecting a set of coplanar features, from which the Euclidean structure of the corresponding 3D plane has to be computed. We suggest to use confocal conics as calibration targets, as they offer undeniable advantages over other ones (e.g., points or lines) in terms of detection and estimation, especially in the presence of partial occlusion. We introduce important projective and Euclidean properties of the linear family of conics (i.e., the confocal conic range), spanned by two confocal conics. In particular, we rely on the fact that the circular point-envelope - a rank-2 conic that encodes the 2D Euclidean structure - is a degenerate member of any confocal conic range. This allows us to give closed-form solutions in three cases: one conic with known foci, two confocal conics with known product of ratios of semi axes, and two unknown confocal conics. The performances of the proposed algorithms (consisting of a few lines of Matlab-like code) show up high accuracies for both intrinsic and extrinsic camera parameters. In addition to experiments with synthetic data, a video sequence is processed, showing off the interest of using confocal conics as calibration targets, for augmented reality purposes.

Most change detection methods are based on gray-level images. A gray-level image is regarded as a 1-D projection of three channels of color images. Therefore, more precise change detection results are expected by utilizing color information. We previously developed a change detection scheme using color images. In this paper, we determine which color space should be selected for accurate change detection based on our previous detection scheme. Our method can be applied to various color spaces, including gray-level images. Then we can measure the expected number of error pixels in order to select an appropriate color space which gives the best result among various color spaces. The experiments show that selecting a color space based on measurements results in the fewest error pixels.

We investigate the projective properties of the feature consisting of two concentric circles. We demonstrate there exist geometric and algebraic constraints on its projection. We show how these constraints greatly simplify the recoveries of the affine and Euclidean structures of a 3D plane. As an application, we assess the performances of two camera calibration algorithms.

We present a novel change detection method using a statistical model of the image noise. Most change detection methods are based on gray-level images. However, color images can provide much richer scene information. One major problem to use the color images in change detection is how to combine three components in color space as a detection cue. We use the Euclidean color distance of three channels to measure the difference between two consecutive images. Specifically, we present a new noise model for each color channel. Through this modeling we can estimate the distribution of the Euclidean color distance for unchanged regions. We can find the optimal threshold to detect changes using this estimated distribution. Although we use the optimal threshold, inevitably there may be false classifications. To reject these erroneous cases, we adopt the graph cuts method that efficiently minimizes the global energy, which takes into account the effect of neighboring pixels.

In this paper, we present a camera calibration method using two views of three pairs of concentric circles with known sizes. A new invariant property for circle is introduced to determine the position of the center of the projected circle. Given two image ellipses and their corresponding centers, a cross-ratio based method estimates the center of projected circles using the new invariant property. The accurate center of projected concentric circles provides correct correspondences between ellipses in the image plane and circles in 3D plane. We also demonstrate that two views of three pairs of coplanar concentric circles are enough to determine the intrinsic camera parameters, such as the focal length, the aspect ratio, and the principal point. We validate the performance of the method using both synthetic and real images. Our method shows a comparable performance with respect to similar calibration methods using planes. The use of concentric circles, however, provides correct correspondences between 3D target points and their image points, and greatly simplifies the calibration problem.