This paper presents our preliminary study on enabling individuals who are legally blind to safely operate mobile robots and vehicles. To achieve this, we developed a teleoperation with accessibility at its core. The system incorporates features that enhance usability and situational awareness, including assistive control based on artificial potential fields to prevent collisions and ensure smooth navigation. It also provides multimodal feedback through (a) haptic vibrations on the gamepad controller, which convey the proximity of nearby objects detected by the robot’s laser sensor, and (b) color-coded overlays that differentiate paths, obstacles, and people through semantic segmentation performed by a deep neural network on the robot’s camera feed. To evaluate its effectiveness, we partnered with the Austin Lighthouse to conduct experiments in which legally blind participants used the system to successfully guide the robot through a testing area with obstacles.

In this paper, we develop a novel and safe control design approach that takes demonstrations provided by a human teacher to enable a robot to accomplish complex manipulation scenarios in dynamic environments. First, an overall task is divided into multiple simpler subtasks that are more appropriate for learning and control objectives. Then, by collecting human demonstrations, the subtasks that require robot movement are modeled by probabilistic movement primitives (ProMPs). We also study two strategies for modifying the ProMPs to avoid collisions with environmental obstacles. Finally, we introduce a rule-base control technique by utilizing a finite-state machine along with a unique means of control design for ProMPs. For the ProMP controller, we propose control barrier and Lyapunov functions to guide the system along a trajectory within the distribution defined by a ProMP while guaranteeing that the system state never leaves more than a desired distance from the distribution mean. This allows for better performance on nonlinear systems and offers solid stability and known bounds on the system state. A series of simulations and experimental studies demonstrate the efficacy of our approach and show that it can run in real time.Note to Practitioners—This paper is motivated by the need to create a teach-by-demonstration framework that captures the strengths of movement primitives and verifiable, safe control. We provide a framework that learns safe control laws from a probability distribution of robot trajectories through the use of advanced nonlinear control that incorporates safety constraints. Typically, such distributions are stochastic, making it difficult to offer any guarantees on safe operation. Our approach ensures that the distribution of allowed robot trajectories is within an envelope of safety and allows for robust operation of a robot. Furthermore, using our framework various probability distributions can be combined to represent complex scenarios in the environment. It will benefit practitioners by making it substantially easier to test and deploy accurate, efficient, and safe robots in complex real-world scenarios. The approach is currently limited to scenarios involving static obstacles, with dynamic obstacle avoidance an avenue of future effort.

In this article, we present a distributed formation control strategy for agents with a variety of dynamics to achieve a desired planar formation. Our approach is based on the barycentric-coordinate-based (BCB) control, which is fully distributed, does not require interagent communication or a common sense of orientation, and can be implemented using relative position measurements acquired by agents in their local coordinate frames. This removes the need for global positioning or alignment of local coordinate frames, which are required across several existing strategies. We show how the BCB control for agents with the simplest dynamical model, i.e., the single-integrator dynamics, can be extended to agents with higher order dynamics such as quadrotors, and nonholonomic agents such as unicycles and cars. Specifically, our extension preserves the desired convergence and robustness guarantees of the BCB approach and is provably robust to saturations in the input and unmodeled linear actuator dynamics for unicycle and car agents. We further show that under our proposed BCB control design, the agents can move along a rotated and scaled control direction without affecting the convergence to the desired formation. This observation is used to design a fully distributed collision avoidance strategy, which is often not considered in the formation control literature. We demonstrate the proposed approach in simulations and further present a distributed robotic platform to test the strategy experimentally. Our experimental platform consists of off-the-shelf equipment that can be used to test and validate other multiagent algorithms. The code and implementation instructions for this platform are available online.

We present a parallel robot mechanism and the constitutive laws that govern the deformation of its constituent soft actuators. Our ultimate goal is the real-time motion-correction of a patient's head deviation from a target pose where the soft actuators control the position of the patient's cranial region on a treatment machine. We describe the mechanism, derive the stress-strain constitutive laws for the individual actuators and the inverse kinematics that prescribes a given deformation, and then present simulation results that validate our mathematical formulation. Our results demonstrate deformations consistent with our radially symmetric displacement formulation under a finite elastic deformation framework.

We present a distributed control strategy for a team of agents to autonomously achieve a desired 3D formation. Our approach is based on local relative position measurements and can be applied to multirotor aerial vehicles. We assume that agents have a common sense of direction, which is used to align the z-axes of their local coordinate frames. However, this assumption is not crucial, and our approach is provably robust to misalignments in the local coordinate frames or measurement inaccuracies. In particular, agents can move along any direction that projects positively onto the desired direction of motion. This property is exploited to design a fully-distributed collision avoidance strategy. We validate the proposed approach experimentally and show that a team of quadrotors can achieve a desired 3D formation without collisions.

Traditional Simultaneous Localization and Mapping (SLAM) approaches build maps based on points, lines or planes. These maps visually resemble the environment but without any semantic or information about the objects in the environment. Recent advancements in machine learning have made object detection highly accurate and reliable with large set of objects. Object detection can effectively help SLAM to incorporate semantics in the mapping process. One of the main obstacles is data association between detected objects over time. We demonstrate a nonparametric statistical approach to solve the data association between detected objects over consecutive frames. Then we use an unsupervised clustering method to identify the existence of objects in the map. The complete process can be run in parallel with SLAM. The performance of our algorithm is demonstrated on several public datasets, which shows promising results in locating objects in SLAM.

Skin surface wounds due to burns, surgeries and chronic illness affect millions of people worldwide. Tissue engineering has become an increasingly popular treatment, but it is a highly manual process. Increasing the automation in tissue engineering could increase the rate of treatment for patients and improve outcomes. We present an initial investigation into an automated in-situ treatment. In our proposed method, a 3D machine vision system detects a skin wound to be treated and then determines the 3D point set corresponding to the wound. The 3D point set is then passed to path planning algorithm for a robot manipulator to move an ink-jet nozzle over the wound and fill the cavity with quick-curing/gelling fluids such collagen and other biomaterials and cell growth promoters. This paper details initial results and experimental validation of each of the proposed steps.

Multistage decision policies provide useful control strategies in high-dimensional state spaces, particularly in complex control tasks. However, they exhibit weak performance guarantees in the presence of disturbance, model mismatch, or model uncertainties. This brittleness limits their use in high-risk scenarios. We present how to quantify the sensitivity of such policies in order to inform of their robustness capacity. We also propose a minimax iterative dynamic game framework for designing robust policies in the presence of disturbance/uncertainties. We test the quantification hypothesis on a carefully designed deep neural network policy; we then pose a minimax iterative dynamic game (iDG) framework for improving policy robustness in the presence of adversarial disturbances. We evaluate our iDG framework on a mecanum-wheeled robot, whose goal is to find a ocally robust optimal multistage policy that achieve a given goal-reaching task. The algorithm is simple and adaptable for designing meta-learning/deep policies that are robust against disturbances, model mismatch, or model uncertainties, up to a disturbance bound. Videos of the results are on the author's website: https://goo.gl/JhshTB, while the codes for reproducing our experiments are on github: https://goo.gl/3G2VBy. A self-contained environment for reproducing our results is on docker: https://goo.gl/Bo7MBe.

Precise patient positioning is fundamental to successful removal of malignant tumors during treatment of head and neck cancers. Errors in patient positioning have been known to damage critical organs and cause complications. To better address issues of patient positioning and motion, we introduce a 3-DOF neuro-adaptive soft-robot, called Soft-NeuroAdapt to correct deviations along 3 axes. The robot consists of inflatable air bladders that adaptively control head deviations from target while ensuring patient safety and comfort. The adaptive-neuro controller combines a state feedback component, a feedforward regulator, and a neural network that ensures correct adaptation. States are measured by a 3D vision system. We validate Soft-NeuroAdapt on a 3D printed head-and-neck dummy, and demonstrate that the controller provides adaptive actuation that compensates for intrafractional deviations in patient positioning.

The Internet of Things (IoT) is fueled by the growth of sensors, actuators, and services that collect and process raw sensor data. Wearable and environmental sensors will be a major component of the IoT and provide context about people and activities that are occurring. It is imperative that sensors in the IoT are synchronized, which increases the usefulness and value of the sensor data and allows data from multiple sources to be combined and compared. Due to the heterogeneous nature of sensors (e.g., synchronization protocols, communication channels, etc.), synchronization can be difficult. In this article, we present novel techniques for synchronizing data from multi-sensor environments based on the events and interactions measured by the sensors. We present methods to determine which interactions can likely be used for synchronization and methods to improve synchronization by removing erroneous synchronization points. We validate our technique through experiments with wearable and environmental sensors in a laboratory environment. Experiments resulted in median drift error reduction from 66% to 98% for sensors synchronized through physical interactions.

Advanced driver assistance systems are the newest addition to vehicular technology. Such systems use a wide array of sensors to provide a superior driving experience. Vehicle safety and driver alert are important parts of these system. This paper proposes a driver alert system to prevent and mitigate adjacent vehicle collisions by proving warning information of on-road vehicles and possible collisions. A dynamic Bayesian network (DBN) is utilized to fuse multiple sensors to provide driver awareness. It detects oncoming adjacent vehicles and gathers ego vehicle motion characteristics using an on-board camera and inertial measurement unit (IMU). A histogram of oriented gradient feature based classifier is used to detect any adjacent vehicles. Vehicles front-rear end and side faces were considered in training the classifier. Ego vehicles heading, speed and acceleration are captured from the IMU and feed into the DBN. The network parameters were learned from data via expectation maximization(EM) algorithm. The DBN is designed to provide two type of warning to the driver, a cautionary warning and a brake alert for possible collision with other vehicles. Experiments were completed on multiple public databases, demonstrating successful warnings and brake alerts in most situations.

This paper presents a novel, decentralized, control-theoretic approach to address collision avoidance for multi-robot systems. We create a virtual obstacle at the mean position of the robots. A control is be designed such that each robot will avoid the closest obstacle when a collision is possible. The closest obstacle can be the virtual obstacle or the nearest robot. We present two such control laws. The first assumes perfect knowledge of the velocities of all nearby robots and can allow a saturated velocity input for each robot. In practice, the velocities of the other robots are hard to measure or estimate precisely. Therefore, the second control law removes the assumption of known velocities based on a high-gain, robust control scheme. We prove the first control scheme is globally asymptotically stable, and the robust control law is globally uniformly ultimately bounded. To verify the effectiveness of the proposed approach, Monte Carlo simulations and experiments have been conducted.

Inertial measurement units (IMUs) including accelerometers and gyroscopes are becoming very common and can be found in cell phones, fitness trackers, and other wearable devices. With the growth in wearable computing and body sensor networks, IMUs are also becoming more prevalent researchenvironments for estimation and tracking of human motion. Wedemonstrate that the accelerometer angle estimate is inaccurate for typical motions and present a method using a kinematic model to correct the accelerometer angle estimate and improve overall orientation estimates using an extended Kalman filter (EKF). Our method improves upon the raw accelerometer orientation estimation method and a gyroscope based EKF method for sensor orientation during motion and at rest.

In this paper, we propose a novel approach for obstacle avoidance. The configuration space of a nonholonomic mobile robot in the special Euclidean group SE(2). Therefore, we divide the whole configuration space into two sub-regions (i.e., a safe region and an obstacle avoidance region). We have derived two functions to define the robot's behavior in each region, which generate velocity vectors that drive the robot away from the obstacle and always towards the goal. We consider the existence of a switching surface between these two sub-regions, and specifically define this switching surface as a local subset of SE(2), i.e. the switching surface includes orientation, defined by the relative angle between the robot heading and bearing to the obstacle. Two different switching signals are proposed based on the switching surface. Lyapunov stability analysis proves the robot will converge to a goal position. To verify the effectiveness of the proposed approach, several simulations and experiments have been conducted.

This paper presents a multistage, real-time optimization algorithm to perform visual searches. A cost function relates the success of the visual search at any given pose in the workspace. This function is measurable at every camera pose, but it is not necessarily known as a function of changing pose, and its gradient with respect to the workspace or joint space coordinates is not known. In the first stage, a camera is constrained on a fixed sphere centered on the robot and searches for a general target direction by multidirectional direct search. In the second stage, the camera is constrained on a series of spheres centered on refined estimates of the search target's position. A series of extremum seeking control (ESC) algorithms are executed to search the best pose to observe the target. Since there is no existing method to detect when ESC has converged, we propose a novel frequency domain criteria to decide when to terminate the current ESC and launch another on the next candidate sphere. We prove that, under certain conditions, ESC will converge to known neighborhood of the optimal point in finite time. Simulations and experiments are presented to illustrate the effectiveness of the proposed algorithm.

In this paper, we present a novel camera-projector system for assisting robot-human interaction. The system is comprised of a stereo camera pair and a DLP projector. The proposed system provides feedback information indicating the robot's perception of the environment and what action a human user desires. Feedback is delivered by iteratively spotlighting objects in the environment using the projector. When the spotlight lands on an object that the human user wants the robot to retrieve, he or she can confirm the object selection, and the robot will perform a grasping task to retrieve the selected object. In this investigation, the proposed camera-projector performs three tasks: 1) Actively detect visually salient objects in the scene from the two camera views using a visual attention model. 2) Locate the detected objects using a trifocal tensor based object matching method, and project a spot of light on the objects to provide information back to the human user. 3) Reconstruct the 3D position of the target to plan robot motion and conduct vision-based control to move a robot manipulator to grasp the object. Experimental results of human-directed object grasping task are presented to demonstrate the functions of the proposed system.

In this work, we investigate aspects of building design that can be optimized. Architectural features that we explore include pillar placement in simple corridors, doorway placement in buildings, and agent placement for information dispersement in an evacuation. The metrics utilized are tuned to the specific scenarios we study, which include continuous flow pedestrian movement and building evacuation. We use Multidimensional Direct Search (MDS) optimization with an extreme barrier criteria to find optimal placements while enforcing building constraints.

This paper presents a novel navigation and control system for autonomous mobile robots that includes path planning, localization, and control. A unique vision-based pose and velocity estimation scheme utilizing both the continuous and discrete forms of the Euclidean homography matrix is fused with inertial and optical encoder measurements to estimate the pose, orientation, and velocity of the robot and ensure accurate localization and control signals. A depth estimation system is integrated in order to overcome the loss of scale inherent in vision-based estimation. A path following control system is introduced that is capable of guiding the robot along a designated curve. Stability analysis is provided for the control system and experimental results are presented that prove the combined localization and control system performs with high accuracy.

This paper proposes a novel approach to terrain classification by wheeled mobile robots, which utilizes vibration data. In our proposed approach, a mobile robot has the ability to categorize terrain types simply by driving over them. Classification of terrain is based on measurements obtained from an inertial measurement unit strapped directly to the robot's chassis. In contrast to the previous approaches, we use acceleration and angular velocity measurements in all cardinal directions to extract over 800 features. Sequential Forward Floating Feature Selection is used to narrow down this large group of features to a set of 15 to 20 that are the most useful. The reduced set of features is used by a Linear Bayes Normal Classifier to classify terrain. Furthermore, different feature sets are generated for different velocity conditions, and the classifier switches based on the current robot velocity. Experimental results are presented that show the strong performance of the proposed system, including 90% accuracy over 20 continuous minutes of driving across different terrains.

This paper presents a novel approach to increase the amount of visual stimuli in sensor measurements using saliency maps. A saliency map is a combination of normalized feature maps in different channels (i.e. color, intensity) to represent the relative strength of visual stimuli in an image. The total saliency is higher when the camera is looking at a scene with more interesting things in the field of view and vise versa. We employ methods of extremum seeking control to find a camera position that corresponds to local maximum saliency value. We combine the global properties of simplex optimization methods with the local search properties and dynamic response of extremum seeking control to create a novel algorithm that is more likely to find a global maximum than conventional extremum seeking control. Simulations and experiments are presented to show the strength of this approach.

This study introduces a novel visual servo controller that is designed to control the pose of the camera to keep multiple objects in the field of view (FOV) of a mobile camera. In contrast with other visual servo methods, the control objective is not formulated in terms of a goal pose or a goal image. Rather, a set of underdetermined task functions are developed to regulate the mean and variance of a set of image features. Regulating these task functions inhibits feature points from leaving the camera FOV. An additional task function is used to maintain a high level of motion perceptibility, which ensures that desired feature point velocities can be achieved. These task functions are mapped to camera velocity, which serves as the system input. A proof of stability is presented for tracking three or fewer targets. Experiments of tracking eight or more targets have verified the performance of the proposed method.

In this paper, we present a visual servo controller that effects optimal paths for a nonholonomic differential drive robot with field-of-view constraints imposed by the vision system. The control scheme relies on the computation of homographies between current and goal images, but unlike previous homography-based methods, it does not use the homography to compute estimates of pose parameters. Instead, the control laws are directly expressed in terms of individual entries in the homography matrix. In particular, we develop individual control laws for the three path classes that define the language of optimal paths: rotations, straight-line segments, and logarithmic spirals. These control laws, as well as the switching conditions that define how to sequence path segments, are defined in terms of the entries of homography matrices. The selection of the corresponding control law requires the homography decomposition before starting the navigation. We provide a controllability and stability analysis for our system and give experimental results.

Control of wheeled vehicles is a difficult problem due to nonholonomic constraints. This problem is compounded by sensor limitations. A previously developed control scheme for a wheeled robot, which keeps a target in the view of a mounted camera, is one solution to the problem. In this paper, we prove the controllability and stability of the control scheme. We present an implementation of the controller, as well as present the results of simulations and physical experiments.

Visual servoing methods are commonly classified as image-based or position-based, depending on whether image features or the camera position define the signal error in the feedback loop of the control law. Choosing one method over the other gives asymptotic stability of the chosen error but surrenders control over the other. This can lead to system failure if feature points are lost or the robot moves to the end of its reachable space. We present a hybrid switched-system visual servo method that utilizes both image-based and position-based control laws. We prove the stability of a specific, state-based switching scheme and present simulated and experimental results.

Vision-based control of wheeled vehicles is a difficult problem due to nonholonomic constraints on velocities. This is further complicated in the control of vehicles with drift terms and dynamics containing fewer actuators than velocity terms. We explore one such system, the wheeled inverted pendulum, embodied by the Segway. We present two methods of eliminating the effects of nonactuated attitude motions and a novel controller based on partial feedback linearization. This novel controller outperforms a controller based on typical linearization about an equilibrium point

We present an example of the use of multi-attribute utility analysis in the design of a robot system. Multi-attribute utility analysis is a tool used by systems engineers to aid in deciding amongst numerous alternatives. Its strength lies in the fact that very different metrics can be compared, and that it takes unto account human preferences and risk attitudes. As a design tool, multi-attribute utility analysis is performed off line, during the system design phase, to choose among possible designs, components, gains, etc. We offer a demonstration of multi-attribute utility analysis in designing a hybrid switched-system visual servo system. We have previously introduced such a system, and here use multi-attribute utility analysis to select a switching algorithm that best suits the needs of a specific user.

In the recent past, many researchers have developed control algorithms for visual servo applications. In this paper, we introduce a new hybrid switched system approach, in which a high-level decision maker selects between two visual servo controllers. We have evaluated our approach with simulations and experiments using three individual visual servo systems and three candidate switching rules. The proposed method is very promising for visual servo tasks in which there is a significant distance between the initial and goal configuration, or the task is one that can cause an individual visual servo system to fail.

In this paper we examine the selection of feature points for visual servoing methods using multiresolution critical-point filters (CPF). With the increased number of feature points made available to us using CPF, we hope to improve the robustness of the system by allowing the algorithm to automatically detect usable feature points on virtually any object without any a priori knowledge of the object. Furthermore, the algorithm revises these points at each iteration to account for events that may have otherwise caused feature points to be lost and led to the visual servo method ending in failure.

Visual servoing methods are commonly classified as image based or position based, depending on whether image features or the robot pose is used in the feedback loop of the control law. Choosing one method over the other gives stability in the chosen state but surrenders all control over the other, which can lead to system failure if feature points are lost or the robot moves to the end of its reachable space. We present a hybrid switched system visual servo method that utilizes both image based and position based control laws. Through a switching scheme we present, this method provides asymptotic stability in both the image and pose and prevent system failure.

Visual servoing has been a viable method of robot manipulator control for more than a decade. Image-based visual servoing (IBVS), in particular, has seen considerable development in recent years. Recently, a number of researchers have reported tasks for which traditional IBVS methods fail, or experience serious difficulties. In response to these difficulties, several methods have been devised that partition the control scheme, allowing troublesome motions to be handled by methods that do not rely solely on the image Jacobian. To date, there has been little research that explores the relative strengths and weaknesses of these methods. In this paper we present such an evaluation. We have chosen three recent visual servo approaches for evaluation, in addition to the traditional IBVS approach. We posit a set of performance metrics that measure quantitatively the performance of a visual servo controller for a specific task. We then simulate each of the candidate visual servo methods for four canonical tasks, under perfect and nonideal experimental conditions.