We present a new semantic map based robot navigation system in the paper. The system takes human voice input, processes multi-modal data including natural languages and RGB-D images, and generates semantic maps for robot navigation. Making use of recent development in image segmentation tools, we integrate robot mapping and localization with a customized real-time object detection model, so that the semantic and navigation layers are efficiently combined for robot navigation purpose. We demonstrate the performance of our developed algorithms in both simulation and real robot experiments. Compared with existing works, we demonstrate applicability to real robot systems and superior performance in terms of success rate.

In this paper, we propose an autonomous gait analysis system consisting of a mobile robot and custom-engineered instrumented insoles. The robot is equipped with an on-board RGB-D sensor, the insoles feature inertial sensors and force sensitive resistors. This system is motivated by the need for a robot companion to engage older adults in walking exercises. Support vector regression (SVR) models were developed to extract accurate estimates of fundamental kinematic gait parameters (i.e., stride length, velocity, foot clearance, and step length), from data collected with the robot's on-board RGB-D sensor and with the instrumented insoles during straight walking and turning tasks. The accuracy of each model was validated against ground-truth data measured by an optical motion capture system with N=10 subjects. Results suggest that the combined use of wearable and robot's sensors yields more accurate gait estimates than either sub-system used independently. Additionally, SVR models are robust to inter-subject variability and type of walking task (i.e., straight walking vs. turning), thereby making it unnecessary to collect subject-specific or task-specific training data for the models. These findings indicate the potential of the synergistic use of autonomous mobile robots and wearable sensors for accurate out-of-the-lab gait analysis.

As humans and mobile robots increasingly coexist in public spaces, their close proximity demands that robots navigate following navigation strategies similar to those exhibited by humans. This could be achieved by learning directly from human demonstration trajectories in a machine learning framework. In this paper, we present a method to learn human navigation behaviors using an imitation learning approach based on generative adversarial imitation learning (GAIL), which has the ability of directly extracting navigation policy. Specifically, we use a large open human trajectory dataset that was experimentally collected in a crowded public space. We then recreate these human trajectories in a 3D robotic simulator, and generate demonstration data using a LIDAR sensor onboard a robot with the robot following the measured human trajectories. We then propose a GAIL based algorithm, which uses occupancy maps generated using LIDAR data as the input, and outputs the navigation policy for robot navigation. Simulation experiments are conducted, and performance evaluation shows that the learned navigation policy generates trajectories qualitatively and quantitatively similar to human trajectories. Compared with existing works using analytical models (such as social force model) to generate human demonstration trajectories, our method learns directly from intrinsic human trajectories, thus exhibits more human-like navigation behaviors.

Pedestrian regulation can prevent crowd accidents and improve crowd safety in densely populated areas. Recent studies use mobile robots to regulate pedestrian flows for desired collective motion through the effect of passive human-robot interaction (HRI). This paper formulates a robot motion planning problem for the optimization of two merging pedestrian flows moving through a bottleneck exit. To address the challenge of feature representation of complex human motion dynamics under the effect of HRI, we propose using a deep neural network to model the mapping from the image input of pedestrian environments to the output of robot motion decisions. The robot motion planner is trained end-to-end using a deep reinforcement learning algorithm, which avoids hand-crafted feature detection and extraction, thus improving the learning capability for complex dynamic problems. Our proposed approach is validated in simulated experiments, and its performance is evaluated. The results demonstrate that the robot is able to find optimal motion decisions that maximize the pedestrian outflow in different flow conditions, and the pedestrian-accumulated outflow increases significantly compared to cases without robot regulation and with random robot motion.

Due to the fast-is-slower phenomenon in emergency escape, it is desirable to regulate pedestrian flows at the exit or a bottleneck. We propose a new robot-assisted pedestrian regulation and study passive human-robot interaction (HRI). A learning-based motion control approach is presented for a robot to efficiently interact with pedestrians for desirable collective motion. We first formulate the problem into an optimal control framework using the pedestrian dynamics description based on existing social force models with embedded HRI forces. To solve the defined optimal control problem, we propose an adaptive dynamic programming (ADP) approach to provide adjustable motion parameters of the robot to efficiently interact with pedestrians so that the regulated pedestrian flow tracks a desired velocity. The ADP control process only uses observed flow information rather than the models of pedestrians, and the ADP method provides feedback control with online learning and control capability. Simulation results demonstrate that the proposed approach can regulate pedestrian flows to desirable speeds by online learning.

Humans and mobile robots will be increasingly cohabiting in the same environments, which has lead to an increase in studies on human robot interaction (HRI). One important topic in these studies is the development of robot navigation algorithms that are socially compliant to humans navigating in the same space. In this paper, we present a method to learn human navigation behaviors using maximum entropy deep inverse reinforcement learning (MEDIRL). We use a large open dataset of pedestrian trajectories collected in an uncontrolled environment as the expert demonstrations. Human navigation behaviors are captured by a nonlinear reward function through deep neural network (DNN) approximation. The developed MEDIRL algorithm takes feature inputs including social affinity map (SAM) that are extracted from human motion trajectories. We perform simulation experiments using the learned reward function, and the performance is evaluated comparing it with the real measured pedestrian trajectories in the dataset. The evaluation results show that the proposed method has acceptable prediction accuracy compared to other state-of-the-art methods, and it can generate pedestrian trajectories similar to real human trajectories with natural social navigation behaviors such as collision avoidance, leader-follower, and split-and-rejoin.

Understanding human behavior is crucial for planning evacuation strategies when an emergency occurs. The social force model, which is a successful quantitative model, has been widely used in investigating human behavior. In this paper, we propose a gradient descent based parameter optimization method to learn the parameters of the social force model from experimental data. Although the original social force model has achieved great success, it does not consider the fact that the response of humans to that happening in front of them is stronger than that happening behind them. In order to model the directional dependency of the interactive force, we propose a modified social force model. Experimental results demonstrate the effectiveness of the modified model.

We present field experiment results of ocean plume surveys conducted with a robotic platform. In our experiments, Rhodamine dye was used to generate a chemical plume in a coastal environment, and an unmanned surface vessel equipped with fluorometer sensors was deployed to conduct plume surveys. We present plume front and cross sectional survey results, where the robot is driven along visible plume boundaries and plume cross sections, respectively. Datasets collected during the experiments are shown together with statistical parameters that quantify the fine-scale plume structure. Experimental evidence shows that the assumption of the existence of a continuous and fixed-value concentration curve used in existing theoretical work for plume front tracking appears too strong for real ocean plumes. The datasets provide characteristic parameters that can be used for further development and verification of simulation models of ocean pollution plumes.

Understanding human mobility is important for the development of intelligent mobile service robots as it can provide prior knowledge and predictions of human distribution for robot-assisted activities. In this paper, we propose a probabilistic method to model human motion behaviors which is determined by both internal and external factors in an indoor environment. While the internal factors are represented by the individual preferences, aims and interests, the external factors are indicated by the stimulation of the environment. We model the randomness of human macro-level movement, e.g., the probability of visiting a specific place and staying time, under the Bayesian framework, considering the influence of both internal and external variables. We use two case studies in a shopping mall and in a college student dorm building to show the effectiveness of our proposed probabilistic human mobility model. Real surveillance camera data are used to validate the proposed model together with survey data in the case study of student dorm.

Due to the faster-is-slower phenomenon in emergency escape, it is desirable to regulate pedestrian flow at the exit or a bottleneck. Modification of pedestrian facilities was previously studied to increase the efficiency and safety by the transportation community. We propose a robot-assisted pedestrian regulation scheme and study passive human-robot interaction (HRI), where the robot acts as a dynamic obstacle that interacts with pedestrians. Such a robot-assisted solution replaces expensive infrastructure modification with real-time reconfigurability. In the paper, we first formulate a robot-assisted flow optimization problem based on the social force models of pedestrian dynamics with embedded HRI forces. We then present an online learning algorithm based on adaptive dynamic programming (ADP) to generate motion control so that the robot can replan and adapt its motion to real-time pedestrian flows. The ADP control process uses observed flow information only but not the models of pedestrians, and provides feedback control with online learning and control capability. Simulation results demonstrate efficiency of the proposed method.

A large number of injuries or deaths may occur when an emergency happens in a crowded public place. The congestion at exits may slow down the egress rate due to the effect of “faster-is-slower”. This inspires us to study how human behavior dynamically changes over time at an emergency in a complex indoor environment. In this paper, we refer the panic of evacuees to their perception of the threat and propose a panic propagation model to model how crowd panic changes during evacuation at an emergency. Combined with the existing social force model, our panic model interprets the self-driven force and interactive forces with others in human mobility. To improve evacuation efficiency, robots are introduced to guide evacuees to escape. Using dynamic environment information, we design an evacuation exit selection algorithm where the optimal exit is automatically selected by the robot with the minimum escape time. In our experiments, a real shopping mall is examined, and the dynamic behavior of panicked evacuees is simulated with the proposed panic model. The evacuation performance of using emergency evacuation robots is evaluated. The improvement of evacuation efficiency validates the effectiveness of our robot-assisted evacuation system.

Using autonomous mobile robots to dynamically track oil plume propagation in ocean environments is challenging. Based on a model of advection-diffusion equation that describes point-source pollution propagation in marine environments, we have previously proposed a model-based estimator-controller design for autonomous robots to dynamically track plume front propagation. In this paper we study the robustness of the controller in a robot simulator, and evaluate it's performance in a realistic environmental model setup. A probabilistic Lagrangian environmental model is used that can capture both the time-averaged, idealized structure and the instantaneous, realistic structure of a dynamic plume. The controller is implemented on the Field Robotics Lab Vehicle Software, which uses Lightweight Communication and Marshalling library to facilitate process communication and for easy transit to field testing. Simulation results are shown with discussions and lessons learned to guide future field experiments.

We study the distributed spectrum sensing in cognitive radio networks. Existing distributed consensus-based fusion algorithms only ensure equal gain combining of local measurements, whose performance may be incomparable to various centralized soft combining schemes. Motivated by this fact, we consider practical channel conditions and link failures, and develop new weighted soft measurement combining without a centralized fusion center. Following the measurement by its energy detector, each secondary user exchanges its own measurement statistics with its local one-hop neighbors, and chooses the information exchanging rate according to the measurement channel condition, e.g., the signal-to-noise ratio (SNR). We rigorously prove the convergence of the new consensus algorithm, and show all secondary users hold the same global decision statistics from the weighted soft measurement combining throughout the network. We also provide distributed optimal weight design under uncorrelated measurement channels. The convergence rate of the consensus iteration is given under the assumption that each communication link has an independent probability to fail, and the upper bound of the iteration number of the$ \epsilon$-convergence is explicitly given as a function of system parameters. Simulation results show significant improvement of the sensing performance compared to existing consensus-based approaches, and the performance of the distributed weighted design is comparable to the centralized weighted combining scheme.

We study robotic tracking of dynamic plume front modeled by the advection-diffusion equation in this paper. Different from existing work purely relying on gradient measurement, the transport model of pollution source is explicitly considered in tracking control design. We first study the problem using a single robot and solve the problem in an estimation and control framework. We then extend it to the multi-robot case in a nearest-neighbor communication structure, and have the robots take formation along the plume front. The distributed control is scalable to a large number of robots. Simulation results show satisfactory performances of the proposed method.

Human indoor localization was previously implemented using wireless sensor networks at the cost of sensing infrastructure deployment. Motivated by high density of smartphones in public spaces, we propose to use a robot-assisted localization system in which the low-cost Kinect sensor and smartphone-based acoustic relative ranging are used to localize moving human targets in indoor environments. An extended Kalman filter based localization algorithm is developed for real-time dynamic position estimation. We present both simulations and real robot-smartphone experiments demonstrating the performance with a localization accuracy of approximately 0.5m.

We consider the problem of source seeking using a group of mobile robots equipped with sensors for concentration measurement (instead of the gradient). In our formulation, each robot maintains a gradient estimation, moves to the source by tracing the gradient, and all together keep a predefined formation in movement. We present two control algorithms with all-to-all and limited communications, respectively. The estimation error is taken into account to derive robust control algorithms. Comparing to existing methods, the proposed algorithm with limited communications is fully distributed. Both theoretical analysis and numerical simulations are given to validate the effectiveness of our methods.

Given a multi-robot system, in order to preserve its geometric shape in a formation, the minimal persistence control addresses questions: (1) what pairwise communication connections have to be prescribed to minimize communication channels, and (2) which orientations of communication links are to be placed between robots. In this paper, we propose a minimal persistence control problem on multi-robot systems with underlying graphs being directed and dynamically switching. We develop distributed algorithms based on the rank of the rigidity matrix and the pebble game method. The feasibility of the proposed methods is validated by simulations on the robotic simulator Webots and experiments on e-puck robot platform.

In this paper, we study the distributed spectrum sensing in cognitive radio networks. Using weighted average consensus algorithm, we develop a weighted soft measurement combining scheme without the centralized fusion center. After the measurement by the energy detector, each secondary user (SU) exchanges their own measurement statistics with its local neighbors, and chooses the information exchanging rate according to the estimated average signal-to-noise ratio (SNR). We prove the convergence of the consensus iteration, and each SU will hold the global decision statistics from the weighted soft measurement combining throughout the network. The proposed scheme is robust with respect to temporary communication link failures. Simulation results show our method has a better performance than the existing average consensus-based approach.

We study the multi-robot synchronization on a segment. The goal is for each robot to move along a subsegment of equal length in equal time interval with potential impacts. To achieve the synchronization, we propose a decentralized algorithm by designing impact law, which does not depend on the position of the robot, but on the time information. Specifically, ¿the time interval between two consecutive impacts¿ is exchanged when the robots meet. We also show how to apply the synchronization algorithm to a planar patrolling problem. Simulation results show the feasibility and robustness of our algorithm.

We study optimal trajectory generation for nonholonomic mobile robots in the presence of moving obstacles. The trajectory is presented by a parameterized higher-order polynomial and is feasible for car-like robots whose motion is nonholonomic. An optimal performance index is set up so that the parameterized trajectory stays close to the shortest straight-line path. Combining with the collision avoidance criterion, optimal collision-free trajectory can be generated real time as the solution is expressed in its closed-form. We show Matlab simulation results to demonstrate the performance of the trajectories.

We consider the problem of generating global feasible trajectories for nonholonomic mobile robots in the presence of moving obstacles. The global trajectory is composed of regional path segments, which are parametric polynomials incorporating collision avoidance criteria. Collision avoidance with moving obstacles is achieved by changing parameters of the regional trajectories, and the collision avoidance parameters are solutions to a second-order inequality and are obtainable analytically. To have a smooth global trajectory, we also develop a smooth irregular curve method to generate continuous boundary conditions to connecting regional trajectories. Steering control laws are constructed by means of differential flatness. The proposed technique works in dynamic environments where a set of global way-points are available, and the velocities of the obstacles are obtainable real time. Simulation results show the effectiveness of the methods.

We study the problem of generating continuous steering control for robots to completely cover a bounded region over a finite time. First, we pack the area by disks of minimum number, and then a neural network based path planning method is adopted to generate complete coverage paths avoiding collisions with stationary and moving objects. Smooth trajectory is developed using parametric polynomial approximation, and continuous steering control is designed to track the trajectory exactly. The algorithm works for car-like robots which have nonholonomic motion constraints, and the control inputs are represented by analytic functions of trajectory parameters. The design is finally extended to cooperative sweeping of multiple robots. Satisfactory performances are observed in simulations

This paper describes our efforts in building an experimental platform to conduct research on cooperative driving in intelligent transportation systems (ITS). A miniature vehicle is developed based on the COTS-BOTS robot and an overhead vision system is used to localize the vehicles. The single lane tracking control algorithm is described first. To control multiple vehicles to drive through an intersection without collision, a distributed cooperative driving algorithm is developed, which is based on velocity planning using search algorithms in the corresponding coordination diagram. Experimental results verify our proposed approach and we believe this platform can serve as a cost-effective testbed to study vehicle-to-vehicle communication based cooperative driving in intelligent transportation systems

We consider the tracking control of a nonholonomic mobile robot with parameter uncertainty and unknown dynamics. A new robust adaptive controller is proposed with the aid of adaptive backstepping and robust control techniques. The proposed controller guarantees that the tracking error converges to a small ball containing the origin. The ball's radius can be adjusted by control parameters. Uncertainties in both of kinematics and dynamics of mobile robots are considered of the first time in the frame of robust and adaptive control in this paper. Simulation results show effectiveness of the proposed controller.

This paper addresses the problem of determining a collision-free path for a mobile robot moving in a dynamically changing environment. By explicitly considering kinematic, model of the robot, the family of feasible trajectories and their corresponding steering controls are derived in a closed form. Then, a new collision avoidance condition is developed for the dynamically changing environment, it consists of a time criterion and a geometrical criterion, and it has explicit physical meanings in both the transformed space and the original working space. By imposing the avoidance condition, one can determine the corresponding steering angle for collision avoidance in a closed form. Such a path meets all boundary conditions, is continuous, and can be updated in real time once a change in the environment is detected. Simulations show that the proposed method is effective.

We propose a distributed and optimal motion planning algorithm for multiple robots. The computationally expensive problem is decomposed into two modules: path planning and velocity planning. The D* search method is applied in both modules, based on either geometric formulation or schedule formulation. Optimization is achieved at the individual robot level by defining cost functions, and also at the team level by a global measurement function reflecting performance indices of interest as a team. Contrary to our knowledge of previous results on multi-robot motion planning that either obtain optimal solutions through centralized and exhaustive computing, or achieve distributed implementations without considering any optimization issues, our approach combines these two features and explicitly optimizes performance functions through a distributed implementation. It is also one of the few that is capable of handling outdoor rough terrain environments and real time replanning. Simulations are shown on a Mars-like rough terrain using a 3D vehicle planner and control simulator. The algorithm was also implemented and successfully run on a group of Nomad 200 indoor robots.