Robotic search often involves teleoperating vehicles into unknown environments. In such scenarios, prior knowledge of target location or environmental map may be a viable resource to tap into and control other autonomous robots in the vicinity towards an improved search performance. In this article, we test the hypothesis that despite having the same skill, prior knowledge of target or environment affects teleoperator actions, and such knowledge can therefore be inferred through robot movement. To investigate whether prior knowledge can improve human-robot team performance, we next evaluate an adaptive mutual-information blending strategy that admits a time-dependent weighting for steering autonomous robots. Human-subject experiments show that several features including distance travelled by the teleoperated robot, time spent staying still, speed, and turn rate, all depend on the level of prior knowledge and that absence of prior knowledge increased workload. Building on these results, we identified distance travelled and time spent staying still as movement cues that can be used to robustly infer prior knowledge. Simulations where an autonomous robot accompanied a human teleoperated robot revealed that whereas time to find the target was similar across all information-based search strategies, adaptive strategies that acted on movement cues found the target sooner more often than a single human teleoperator compared to non-adaptive strategies. This gain is diluted with number of robots, likely due to the limited size of the search environment. Results from this work set the stage for developing knowledge-aware control algorithms for autonomous robots in collaborative human-robot teams.

Visual analysis in field science experiments often involves classifying objects on experimental images and videos. In this context, developing a reliable and independently validated estimate of mental workload during object classification can enable cognitively responsive task allocation. The goal of this study is to quantify the cognitive load perceived by humans from electroencephalography (EEG) data during an underwater object classification task that was inspired from citizen science studies. During the task, participants were asked to identify one of three possible invasive fish species in short videos of a virtual underwater environment. The virtual environment was modeled to vary fish behavior and environmental factors that are known to be critical in classification. A contextually-relevant secondary task was designed to provide independent validation of cognitive load measures. Several established measures of cognitive load were compared across different weightings on the scalp positions, and the measure that strongly associated with reaction time and a secondary task accuracy was selected for further analysis. Our results show that cognitive load calculated using the difference in power of alpha frequencies best correlates with reaction time and secondary task accuracy. When fit to the environmental factors, cognitive load calculated using this approach was high when the environment was turbid and the fish moved at high speeds. Results from this study have applications in cognitively-responsive human–computer interaction and in developing shared control strategies in human–robot interaction.

Multirobot systems provide a scalable and robust solution for monitoring tasks. In time-intensive missions such as search and rescue, the inclusion of a human has the potential advantage of incorporating prior knowledge about the target location or dynamics. In this article, we develop a general information-theoretic framework to control multiple autonomous robots in search and rescue missions that include a human teleoperator. Human prior knowledge is modeled to capture the target location and dynamics, and mutual-information-based control is formulated to let autonomous robots weigh between two strategies: independent search or assisting the human by staying in proximity. The control actions optimize a weighted sum of normalized mutual information calculated using particle-filtered estimates of the target and the reference robot. We implement the framework to simulate two widely different scenarios designed after search-and-rescue missions from literature, and incorporate varying levels of accuracy in human prior knowledge. Our results indicate that the mission performance depends on how robots weigh between the two strategies, with the amount of the optimal control effort shared between strategies affected by prior knowledge and the number of robots. Comparison with existing strategies points to the benefits of an information-based control in situations where human prior knowledge is inaccurate. The proposed information-theoretic abstraction of the human–robot interaction can be implemented on a wide variety of scenarios and the results highlight the role of human prior knowledge toward effective robotic assistance in time-intensive missions.

This study explored data analytic approaches to assessing young children's engagement in robot-mediated collaborative interaction. To develop our analytic models, we took a case-study approach and looked closely into four children's behaviors during three conversational sessions. Grounded in engagement theory, three sources of multimodal behavioral data (utterances, kinesics, and vocie) were coded through human annotation and automatic speech recognition and analysis. Then, information-theoretic methods were used to uncover nonlinear dependencies (called mutual information) among the multimodal behaviors of each child. From this, we derived a model to compute a compound variable of engagement. This computation produced engagement trends of each child, the engagement relationship between two children in a pair, and the engagement relationship with the robot over time. The computed trends corresponded well with the data from human observations. This approach has implications for quantifying engagement from rich and natural multimodal behaviors.

Gearbox failures cost thousands of lost production hours in plants that use industrial robots. In this context, an automated monitoring system that can warn the user of an impending failure can save precious resources. This problem has been addressed in many other domains through the use of machine learning approaches. However, standard machine learning algorithms are limited in their ability to detect gearbox failures, mainly due to task variability arises from robot-specific data. To improve detection performance of machine learning approaches, in this paper we propose techniques to curate the data prior to building a classification model. In a systematic hypothesis-driven study exploring the effect of different preprocessing techniques, we evaluate training data augmentation with estimated measurements, data differencing to suppress task dependence, inclusion of local variation, and selection of principal components on data collected from 26 industrial robots from the field. Our results show that preprocessing techniques improve the failure detection performance.

We investigate the interaction between a pedestrian crowd and a social robot-a relevant human-robot interaction scenario observed in train stations and entrances to public places. We use a popular agent-based social force model to simulate a pedestrian crowd as it passes near a stationary robot. The modelling framework permits analysis of difficult-to-test situations such as high density crowds and the effects of contagion whereby more people are likely to interact with the robot if it is already surrounded by an audience. Accordingly, we augment the modelling framework to account for varying crowd densities, human-robot interaction, and social influence, and inform the parameter values from empirical studies in literature. Our results show that while the rate-of-interaction, defined as the number of agents located within an interacting distance per minute, increases with flow density, the average interaction-time is independent of the same. We find that inducing social influence through contagion does not have a significant effect on the rate at which agents engage with the robot in dense crowd scenarios, and has a marginal effect at low densities. The interaction-time was found to depend on the interaction-speed at which agents engage with the robot. These results indicate that at normally witnessed flow densities crowding near the robot is unlikely to take place, and the effect of robot design choices supersede the effects of social force or contagion.

We describe a virtual-reality framework for investigating startle-response behavior in fish. Using real-time three-dimensional tracking, we generate looming stimuli at a specific location on a computer screen, such that the shape and size of the looming stimuli change according to the fish's perspective and location in the tank. We demonstrate the effectiveness of the setup through experiments on Giant danio and compute the success rate in eliciting a response. We also estimate visual startle sensitivity by presenting the stimulus from different directions around the fish head. The aim of this work is to provide the basis for quantifying escape behavior in fish schools.

This paper describes a probabilistic framework to estimate the shape and position of multiple fish in a school. We model the fish shape as an ellipsoid with a curvature coefficient that allows us to incorporate bending. An expression for the extremal contour in terms of state parameters is used to derive a likelihood function for shape. We present a motion model that uses curvature as an input to the turning rate. Tracking is performed using a particle filter with joint probabilistic data association. We evaluate our algorithm using simulated data and further characterize its performance using real data from a laboratory experiment with six giant danios.

This paper describes a model-based probabilistic framework for tracking a fleet of laboratory-scale underwater vehicles using multiple fixed cameras. We model the target motion as a steered particle whose dynamics evolve on the special Euclidean group. We provide a likelihood function that extracts three-dimensional position and pose measurements from monocular images using projective geometry. The tracking algorithm uses particle filtering with selective resampling based on a threshold and nearest neighbor data association for multiple targets.We describe results obtained from two tracking experiments: first with one vehicle and a second experiment with two targets. The tracking algorithm for single target experiment is validated using data denial.