As today's smartphone integrates various imaging sensors and Inertial Measurement Units (IMU) and becomes computationally powerful, there is a growing interest in developing smartphone-based visual-inertial (VI) SLAM methods for robotics and computer vision applications. In this paper, we introduce a new SLAM method, called Visual-LiDAR-Inertial Odometry (VLIO), based on an iPhone 12 Pro. VLIO formulates device pose estimation as an optimization problem that minimizes a cost function based on the residuals of the inertial, visual, and depth measurements. We present the first work that 1) characterizes the iPhone's LiDAR in depth measurement and identifies the models for the measurement error and standard deviation, and 2) characterizes pose change estimation with LiDAR data. The measurement models are then used to compute the depth-related and visual-feature-related residuals for the cost function. Also, VLIO tracks varying camera intrinsic parameters (CIP) in real-time and uses them in computing these residuals. Both approaches result in more accurate residual terms and thus more accurate pose estimation. The CIP tracking method eliminates the need of a sophisticated model-fitting process that includes camera calibration and paring of the CIPs and IMU measurements with various phone orientations. Experimental results validate the efficacy of VLIO.

This paper presents a hand-worn assistive device to assist a visually impaired person with object manipulation. The device uses a Google Pixel 3 as the computational platform, a Structure Core (SC) sensor for perception, a speech interface, and a haptic interface for human-device interaction. W-ROMA is intended to assist a visually impaired person to locate a target object (nearby or afar) and guide the user to move towards and eventually take a hold of the object. To achieve this objective, three functions, including object detection, wayfinding, and motion guidance, are developed. Object detection locates the target object’s position if it falls within the camera’s field of view. Wayfinding enables the user to approach the object. The haptic/speech interface guides the user to move close to the object and then guides the hand to reach the object. A new visual-inertial odometery (VIO), called RGBD-VIO, is devised to accurately estimate the device’s pose (position and orientation), which is then used to generate the motion command to guide the user and his/her hand to reach the object. Experimental results demonstrate that RGBD-VIO outperforms the state-of-the-art VIO methods in 6-DOF device pose estimation and the device is effective in assistive object manipulation.

In this paper, we propose a new scheme based on the Sampson distance (SD) to describe visual feature residuals for visual-inertial odometry (VIO). Unlike the epipolar-constraint-based SD for visual odometry (VO), the proposed SD is formulated based on the perspective projection constraint. We proved in theory that the proposed SD retains the good properties of those earlier SD criteria in the literature of VO and it represents a visual feature residual more accurately than the prevailing transfer distance (TD) in existing VIO methods. We formulate three distance criteria, including TD, reprojection error (RE), and SD, and compared their performances by simulation. The results show that the SD is much more accurate than the TD and it is a very accurate estimate of the gold standard criteria—RE. Based on the SD, we modified VINS-Mono by replacing its TD-based visual residuals with the SD-based residuals and study the SD's efficacy in pose estimation by experiments with several public datasets. The results reveal that the SD-based VINS-Mono has a substantial improvement over the original VINS-Mono in pose estimation accuracy. This indicates that the SD is a better distance criterion than the TD for representing visual feature residuals. The proposed SD may find its applications to broader areas in computer vision and robotics.

This paper presents a visual positioning system (VPS) for real-time pose estimation of a robotic navigation aid (RNA) for assistive navigation. The core of the VPS is a new method called depth-enhanced visual-inertial odometry (DVIO) that uses an RGB-D camera and an inertial measurement unit (IMU) to estimate the RNA's pose. The DVIO method extracts the geometric feature (the floor plane) from the camera's depth data and integrates its measurement residuals with that of the visual features and the inertial data in a graph optimization framework for pose estimation. A new measure based on the Sampson error is introduced to describe the measurement residuals of the near-range visual features with a known depth and that of the far-range visual features whose depths are unknown. The measure allows for the incorporation of both types of visual features into graph optimization. The use of the geometric feature and the Sampson error improves pose estimation accuracy and precision. The DVIO method is paired with a particle filter localization (PFL) method to locate the RNA in a 2D floor plan and the information is used to guide a visually impaired person. The PFL reduces the RNA's position and heading error by aligning the camera's depth data with the floor plan map. Together, the DVIO and the PFL allow for accurate pose estimation for wayfinding and 3D mapping for obstacle avoidance. Experimental results demonstrate the usefulness of the RNA in assistive navigation in indoor spaces.

This paper presents VCU-RVI, a new visual inertial odometry (VIO) benchmark with a set of diverse data sequences in different indoor scenarios. The benchmark was captured using an Structure Core (SC) sensor, consisting of an RGB-D camera and an IMU. It provides aligned color and depth images with 640×480 resolution at 30 Hz. The camera's data is synchronized with the IMU's data at 100 Hz. Thirty-nine data sequences covering a total of ~3.7 kilometers trajectory were recorded in various indoor environments by two experimental setups: hand-holding the SC sensor or installing it on a wheeled robot. For the data sequences from the handheld SC, some were recorded in our laboratory under three challenging conditions: fast sensor motion, radical illumination changing, and dynamic objects, and the rest were collected in various indoor spaces outside the laboratory in the East Engineering Building, including corridors, halls, and stairways, during long-distance navigation scenarios. For the data sequences captured using the wheeled robot, half of them were recorded with sufficient IMU excitation in the beginning of the sequence, to meet the need of testing the VIO methods with the requirement of sufficient motion conditions for initialization. We placed three bumpers on the floor of the lab to create an uneven terrain to make the robot motion 6-DOF. The sequences also include data collected from navigational courses with a long trajectory. For trajectory evaluation, a motion capture system is used to generate accurate pose data (at a rate of 120 Hz), which will be used as the ground truth. We conducted experiments to evaluate the state-of-the-art VIO algorithms using our benchmark. These algorithms together with the evaluation tools and the VCU-RVI dataset are made publicly available.

This paper presents a new visual-inertial odometry, term DUI-VIO, for estimating the motion state of an RGB-D camera. First, a Gaussian mixture model (GMM) to is employed to model the uncertainty of the depth data for each pixel on the camera's color image. Second, the uncertainties are incorporated into the VIO's initialization and optimization processes to make the state estimate more accurate. In order to perform the initialization process, we propose a hybrid-perspective-n-point (PnP) method to compute the pose change between two camera frames and use the result to triangulate the depth for an initial set of visual features whose depth values are unavailable from the camera. Hybrid-PnP first uses a 2D-2D PnP algorithm to compute rotation so that more visual features may be used to obtain a more accurate rotation estimate. It then uses a 3D-2D scheme to compute translation by taking into account the uncertainties of depth data, resulting in a more accurate translation estimate. The more accurate pose change estimated by Hybrid-PnP help to improve the initialization result and thus the VIO performance in state estimation. In addition, Hybrid-PnP make it possible to compute the pose change by using a small number of features with a known depth. This improves the reliability of the initialization process. Finally, DUI-VIO incorporates the uncertainties of the inverse depth measurements into the nonlinear optimization process, leading to a reduced state estimation error. Experimental results validate that the proposed DUI-VIO method outperforms the state-of-the-art VIO methods in terms of accuracy and reliability.

This paper introduces a new robotic navigation aid (RNA) for the visually impaired (VI). Two fundamental functions - wayfinding and human-robot interaction (HRI) - are presented for assisted wayfinding. The problem of wayfinding involves planning a path from the RNA's current location to the destination and following the path to get to the destination. To address the problem, we developed a new visual inertial odometry to estimate the RNA's pose by using the image and depth data from an RGB-D camera and the inertial data of an IMU. The estimated pose is used for path planning. To guide the user to follow the planned path, we designed an HRI interface with two guiding modes - the robocane mode and white-came mode. In the robocane mode, the RNA uses a motorized rolling tip to steer itself into the desired direction of travel (DDT) for the user to follow and track the planned path. In the white-cane mode, the RNA uses its speech interface to indicate the DDT to the user by audio messages. In this mode, the user swings the RNA just like using a conventional white cane. To make mode selection effortless, we developed a human intent detection (HID) method based on the decision tree mode. The method can detect the user intent and automatically select the appropriate mode according to the detected intent. Experimental results demonstrate the efficacies of the VIO, HRI, and HID methods for assisted wayfinding.

This study is designed to validate the feasibility of generating identifiable moving patterns using electro-tactile stimulation. An electro-tactile display is built using an array of 16 contacts to deliver the electrical signal to the fingertip skin. This signal can have varying voltages, frequencies or duty cycles to form the most comfortable sensation. Moving patterns can be generated by individually or collectively switching on or off the contacts on the display. This is done to stimulate a moving pattern. In this case, a moving pattern is comparable to a group of frame-by-frame pictures constructing a movie. Similarly, by toggling the contacts in a specific order, a moving pattern can be achieved. A program on a single-board computer (Raspberry Pi) was used to control and generate 6 different patterns. These patterns are delivered to the display and consequently to the fingertip skin of the participants. A total of 8 subjects participated in this study. They filled a questionnaire to indicate the corresponding movement. The results of these experiments were analyzed and a conclusion regarding the direction of the movement was drawn. It became clear that the direction of the movement had a significant impact on the recognition of the patterns.

This paper compares the performance of three state-of-the-art visual-inertial simultaneous localization and mapping (SLAM) methods in the context of assisted wayfinding of the visually impaired. Specifically, we analyze their strengths and weaknesses for assisted wayfinding of a robotic navigation aid (RNA). Based on the analysis, we select the best visual-inertial SLAM method for the RNA application and extend the method by integrating with it a method capable of detecting loops caused by the RNA's unique motion pattern. By incorporating the loop closures in the graph and optimization process, the extended visual-inertial SLAM method reduces the pose estimation error. The experimental results with our own datasets and the TUM VI benchmark datasets confirm the advantage of the selected method over the other two and validate the efficacy of the extended method.

coordinate. Experimental results demonstrate that the proposed method results in accurate pose estimates for positioning the RNA in indoor environments. Based on the PE method, a wayfinding system is developed for localization of the RNA in a home environment. The system uses the estimated pose and the floorplan to locate the RNA user in the home environment and announces the points of interest and navigational commands to the user through a speech interface. NOTE TO PRACTITIONERS: This work was motivated by the limitations of the existing navigation technology for the visually impaired. Most of the existing methods use a point/line measurement sensor for indoor object detection. Therefore, they lack capability in detecting 3D objects and positioning a blind traveler. Stereovision has been used in recent research. However, it cannot provide reliable depth data for object detection. Also, it tends to produce a lower localization accuracy because its depth measurement error quadratically increases with the true distance. This paper suggests a new approach for navigating a blind traveler. The method uses a single 3D time-of-flight camera for both 6-DOF PE and 3D object detection and thus results in a small-sized but powerful RNA. Due to the camera's constant depth accuracy, the proposed egomotion estimation method results in a smaller error than that of existing methods. A new EKF method is proposed to integrate the egomotion into the RNA's 6-DOF pose in the world coordinate system by tracking both visual and geometric features of the operating environment. The proposed method substantially reduces the pose error of a standard EKF method and thus supports a longer range navigation task. One limitation of the method is that it requires a feature-rich environment to work well.

In this paper, we propose a new 3D object recognition method. The method segments a 3D point set into a number of planar patches and extracts the Inter-Plane Relationships (IPRs) for all patches. Based on the IPRs, the method determines the High Level Feature (HLF) for each patch. A Gaussian-Mixture-Model-based plane classifier is then employed to classify each patch into one belonging to a certain model object. Finally, a recursive plane clustering procedure is performed to cluster the classified planes into the model objects. Experimental results demonstrate that the proposed method has high success rates in object recognition with real-world data. Also, the method can be implemented for real-time operation.

This paper presents a new robotic navigation aid, called Co-Robotic Cane (CRC). The CRC uses a 3D camera for both Pose Estimation (PE) and Object Recognition (OR) in an unknown indoor environment. The 6-DOF PE method determines the CRC's pose change by an egomotion estimation method, called Visual Range Odometry (VRO), and the Iterative Closest Point (ICP) algorithm and reduces the pose integration error by a pose graph optimization algorithm. The PE method does not require any prior knowledge of the environment. The OR method detects indoor structures (stairways, doorways, etc.) and objects (tables, computer monitors, etc.) by the Gaussian Mixture Models. Some of structures/objects (e.g., stairways) may be used as navigational waypoints and the others for obstacle avoidance. The CRC is a co-robot. It may detect human intent and collaborate with its user in performing a navigation task. The proposed CRC is the first in its kind.

This paper presents a new plane extraction (PE) method based on the random sample consensus (RANSAC) approach. The generic RANSAC-based PE algorithm may over-extract a plane, and it may fail in case of a multistep scene where the RANSAC procedure results in multiple inlier patches that form a slant plane straddling the steps. The CC-RANSAC PE algorithm successfully overcomes the latter limitation if the inlier patches are separate. However, it fails if the inlier patches are connected. A typical scenario is a stairway with a stair wall where the RANSAC plane-fitting procedure results in inliers patches in the tread, riser, and stair wall planes. They connect together and form a plane. The proposed method, called normal-coherence CC-RANSAC (NCC-RANSAC), performs a normal coherence check to all data points of the inlier patches and removes the data points whose normal directions are contradictory to that of the fitted plane. This process results in separate inlier patches, each of which is treated as a candidate plane. A recursive plane clustering process is then executed to grow each of the candidate planes until all planes are extracted in their entireties. The RANSAC plane-fitting and the recursive plane clustering processes are repeated until no more planes are found. A probabilistic model is introduced to predict the success probability of the NCC-RANSAC algorithm and validated with real data of a 3-D time-of-flight camera-SwissRanger SR4000. Experimental results demonstrate that the proposed method extracts more accurate planes with less computational time than the existing RANSAC-based methods.

This paper presents a recursive method for extracting planar surfaces from noisy range data. The method first transforms the range image into a so-called Enhanced Range Image (ERI) that encodes the local geometric information (surface normals) and global spatial information (coordinates) of the 3D range data. The ERI is then clustered into a number of homogenous groups called Super-Pixels (SPs). By treating the SPs as the nodes a graph is constructed. A new similarity function is proposed to compute the edge weights between the nodes, base on which the graph is recursively partitioned into two segments by the Normalized Cuts (NC) method until an exit condition is met. In this work, the exit condition is that each of the resulting segments is a plane or contains only one SP. After the partitioning process, neighboring planar segments are merged based on their spatial relationships. The recursive approach eliminates the need for a pre-specified segment number that is necessary in the existing NC based image segmentation methods. The ERI coding enhances object surfaces and edges while the effect of its sensitivity to surface normals is suppressed by the similarity function that takes into account the spatial information in computing the edge weights of the graph. The proposed method can be applied to navigation of mobile robots, symbolic map-building, and range data understanding. In this work the range data are captured from a 3D Time-of-Flight imaging sensor-the Swissranger SR-3000.

This paper presents a new method for extracting planar features from noisy range data. The method encodes the local geometric information (surface normals) and global spatial information (coordinates) of 3D data points into an Enhanced Range Image (ERI) which is then clustered into a number of homogeneous groups, called Super Pixels (SPs). The Normalized Cuts (NC) method is employed to the graph built on the SPs and groups the SPs into planar segments. The ERI coding enhances object surfaces and edges while its sensitivity to surface normals is suppressed by the NC measure that takes into account the spatial information of SPs in computing the edge weights of the graph. A binary matrix is constructed to represent the spatial and similarity relationships among the planar segments. We then employ a search algorithm on this matrix to merge homogenous planar segments. The proposed approach is compared with a representative plane segmentation method in various indoor environments and the results demonstrate the efficacy of the proposed method. In this paper, rang data are captured from a 3D imaging sensor-the Swissranger SR-3000.

This paper presents a new method for extracting object edges from range images obtained by a 3D range imaging sensor the SwissRanger SR-3000. In range image preprocessing stage, the method enhances object edges by using surface normal information; and it employs the Hough Transform to detect straight line features in the Normal-Enhanced Range Image (NERI). Due to the noise in the sensor's range data, a NERI contains corrupted object surfaces that may result in unwanted edges and greatly encumber the extraction of linear features. To alleviate this problem, a Singular Value Decomposition (SVD) filter is developed to smooth object surfaces. The efficacy of the edge extraction method is validated by experiments in various environments.

This paper describes a new approach to extract planar features from 3D range data captured by a range imaging sensor-the SwissRanger SR-3000. The focus of this work is to segment vertical and horizontal planes from range images of indoor environments. The method first enhances a range image by using the surface normal information. It then partitions the Normal Enhanced Range Images (NERI) into a number of segments using the Normalized-Cuts (N-Cuts) algorithm. A least-square plane is fit to each segment and the fitting error is used to determine if the segment is planar or not. From the resulting planar segments, each vertical or horizontal segment is labeled based on the normal of its least-square plane. A pair of vertical or horizontal segments is merged if they are neighbors. Through this region growing process, the vertical and horizontal planes are extracted from the range data. The proposed method has a myriad of applications in navigating mobile robots in indoor environments.

This paper presents a terrain mapping system and a terrain traversability analysis method for mobile robot navigation in urban environments. The terrain mapping system employs a 2-D Laser Rangefinder (LRF). In order to generate a reliable elevation map for navigation, a filtering method based on robot motion constraint and the LRF’s characteristics is used to remove range data. A so-called “Polar Traversability Index” measure is proposed to evaluate terrain traversal property. A PTI dictates the level of difficulty for a robot to move along the corresponding direction and it can be used to guide the robot in urban environments. For instance, it enables the robot to traverse wheelchair ramps and avoid curbs when negotiating sidewalks. The efficacy of the PTI has been verified by simulation and experiments in a complete navigation system.

This paper presents an autonomous terrain navigation system for a mobile robot. The system employs a two-dimensional laser range finder (LRF) for terrain mapping. A so-called "traversability field histogram" (TFH) method is proposed to guide the robot. The TFH method first transforms a local terrain map surrounding the robot's momentary position into a traversability map by extracting the slope and roughness of a terrain patch through least-squares plane fitting. It then computes a so-called "polar traversability index" (PTI) that represents the overall difficulty of traveling along the corresponding direction. The PTIs are represented in a form of histogram. Based on this histogram, the velocity and steering commands of the robot are determined. The concept of a virtual valley and an exit condition are proposed and used to direct the robot such that it can reach the target with a finite-length path. The algorithm is verified by simulation and experimental results.

This paper presents a new obstacle negotiation method for mobile robot navigation on rough terrain. The proposed method first transforms a local terrain map surrounding the robot's momentary position into a grid-type traversability map by extracting the slope and roughness of a terrain patch through least-squares plane-fitting. It then computes so-called "polar obstacle densities" for the cells in the traversability map and transforms the traversability map into a traversability field histogram, from which the velocity and the steering command for the robot are determined. Simulation results show that the algorithm is able to navigate the robot to the target with a finite-length path.

This paper introduces a novel filter for terrain mapping with a two-dimensional laser rangefinder. The filter, called the certainty-assisted spatial (CAS) filter, uses the physical constraints on motion continuity and spatial continuity to identify corrupted pixels and missing data in an elevation map. The filter removes the corrupted pixels, fills in the missing data, and leaves the uncorrupted pixels intact so as to preserve the details of a terrain map. Our extensive indoor and outdoor mapping experiments show the CAS filter's superior performance in erroneous data reduction and map detail preservation over conventional filters.

Fuzzy logic systems are promising for efficient obstacle avoidance. However, it is difficult to maintain the correctness, consistency, and completeness of a fuzzy rule base constructed and tuned by a human expert. A reinforcement learning method is capable of learning the fuzzy rules automatically. However, it incurs a heavy learning phase and may result in an insufficiently learned rule base due to the curse of dimensionality. In this paper, we propose a neural fuzzy system with mixed coarse learning and fine learning phases. In the first phase, a supervised learning method is used to determine the membership functions for input and output variables simultaneously. After sufficient training, fine learning is applied which employs reinforcement learning algorithm to fine-tune the membership functions for output variables. For sufficient learning, a new learning method using a modification of Sutton and Barto's model is proposed to strengthen the exploration. Through this two-step tuning approach, the mobile robot is able to perform collision-free navigation. To deal with the difficulty of acquiring a large amount of training data with high consistency for supervised learning, we develop a virtual environment (VE) simulator, which is able to provide desktop virtual environment (DVE) and immersive virtual environment (IVE) visualization. Through operating a mobile robot in the virtual environment (DVE/IVE) by a skilled human operator, training data are readily obtained and used to train the neural fuzzy system.

This paper presents a characterization study of the Sick LMS 200 laser scanner. A number of parameters, such as operation time, data transfer rate, target surface properties, as well as the incidence angle, which may potentially affect the sensing performance, are investigated. A probabilistic range measurement model is built based on the experimental results. The paper also analyzes the mixed pixels problem of the scanner.

This paper presents a novel Behavior Fusion method for the navigation of Autonomous Mobile Vehicle in unknown environments. The proposed navigator consists of an Obstacle Avoider (OA), a Goal Seeker (GS) and a Navigation Supervisor (NS). The fuzzy actions inferred by the OA and the GS are weighted by the NS using the local and global environmental information and fused through fuzzy set operation to produce a command action, from which the final crisp action is determined by defuzzification. Simulation shows that the navigator is able to perform successful navigation task in various unknown environments, and it has smooth action and exceptionally good robustness to sensor noise.

In this paper, an alternative training approach to the EEM-based training method is presented and a fuzzy reactive navigation architecture is described. The new training method is 270 times faster in learning speed; and is only 4% of the learning cost of the EEM method. It also has very reliable convergence of learning; very high number of learned rules (98.8%); and high adaptability. Using the rule base learned from the new method, the proposed fuzzy reactive navigator fuses the obstacle avoidance behaviour and goal seeking behaviour to determine its control actions, where adaptability is achieved with the aid of an environment evaluator. A comparison of this navigator using the rule bases obtained from the new training method and the EEM method, shows that the new navigator guarantees a solution and its solution is more acceptable.

Reinforcement learning based on a new training method previously reported guarantees convergence and an almost complete set of rules. However, there are two shortcomings remained: 1) the membership functions of the input sensor readings are determined manually and take the same form; and 2) there are still a small number of blank rules needed to be manually inserted. To address these two issues, this paper proposes an adaptive fuzzy approach using a supervised learning method based on backpropagation to determine the parameters for the membership functions for each sensor reading. By having different input fuzzy sets, each sensor reading contributes differently in avoiding obstacles. Our simulations show that the proposed system converges rapidly to a complete set of rules, and if there are no conflicting input-output data pairs in the training sets, the proposed system performs collision-free obstacle avoidance.

We propose a novel approach of fusing the fuzzy control actions of the obstacle avoidance, goal seeking and steering behaviors, in which the steering behavior is derived from motion prediction. As such, the navigator is more capable to steer clear of the zone of high collision probability. Through simulation, it has been confirmed that the navigator having this steering behavior can tackle multiple moving obstacles successfully at much higher speed compared with those without. Furthermore, it does not require any a priori knowledge of the obstacle motion.