Recognition of sequential human activities, such as “sitting down” and “standing up”, is a common but challenging problem in human-robot interaction, which requires modeling their underlying temporal patterns. Although previous sequence modeling methods, such as Hidden Conditional Random Fields (HCRFs), demonstrated satisfactory recognition accuracy, they do not explicitly model the uncertainty in underlying temporal patterns, which can provide valuable information to characterize sequential activities. To address this problem, we introduce a novel Minimum Uncertainty HCRF (MU, or μHCRF). Different from traditional HCRF-based techniques that only utilize the negative log-likelihood of the categories' conditional probability as the loss function, the proposed μ-HCRF also introduces a regularization term to model the underlying temporal pattern of the latent variables. As another theoretical contribution, we provide a derivation to show that the formulated problem has a closed-form solution, and prove that inference of the proposed μHCRF is tractable. Extensive empirical study is performed to evaluate our approach, using four public benchmark datasets. Experimental results have shown that our μHCRFs outperform previous techniques and achieve state-of-the-art performance on human activity recognition, especially on sequential activities.

In human-robot teaming, interpretation of human actions, recognition of new situations, and appropriate decision making are crucial abilities for cooperative robots (“co-robots”) to interact intelligently with humans. Given an observation, it is important that human activities are interpreted the same way by co-robots as human peers so that robot actions can be appropriate to the activity at hand. A novel interpretability indicator is introduced to address this issue. When a robot encounters a new scenario, the pretrained activity recognition model, no matter how accurate in a known situation, may not produce the correct information necessary to act appropriately and safely in new situations. To effectively and safely interact with people, we introduce a new generalizability indicator that allows a co-robot to self-reflect and reason about when an observation falls outside the co-robot's learned model. Based on topic modeling and the two novel indicators, we propose a new Self-reflective Risk-aware Artificial Cognitive (SRAC) model, which allows a robot to make better decisions by incorporating robot action risks and identifying new situations. Experiments both using real-world datasets and on physical robots suggest that our SRAC model significantly outperforms the traditional methodology and enables better decision making in response to human behaviors.

Currently, robot learning of human activities is mainly studied in two largely disconnected domains: high level semantics understanding in human activity recognition, and low level motion trajectory reproduction in robot imitation learning. The critical problem of human activity unified learning (HAUL) was not well studied in previous work. One important challenge is the lack of a representation that can be learned from both levels. We introduce a novel approach to address this HAUL problem at the representation level, by simultaneously learning action labels and motion trajectories from publicly available 3D human skeletal datasets, thus avoiding additional human labor for data collection. Our approach builds a subject and body position independent shared skeleton, and extracts features of skeletal activities based on this model. Then the extracted features are encoded by the parameter set of Gaussian Mixture Models to construct the unified representations. The proposed compact representation can be directly applied to identify activity labels when combined with Support Vector Machines, and can be also employed to generate trajectories of the learned activities when combined with Gaussian Mixture Regression on a robot. Finally, an inverse kinematic mapping is developed to transfer human skeletal trajectories to joint angle sequences in the robot's embodiment. Empirical studies using simulation and real humanoid robots demonstrate that our approach achieves promising performance on robot unified learning of human action labels and motion trajectories, effectively addressing the HAUL problem.

Human activity recognition has a variety of important real-world applications, such as video analysis, surveillance, and human-robot interaction. As a promising video representation method, local spatio-temporal (LST) features have received increasing attention from computer vision, machine learning, and robotics communities. However, approaches based on traditional LST features only use color information, which face several challenges, such as illumination changes and dynamic backgrounds. The recent availability of commercial color-depth cameras makes it much cheaper, faster, and easier to acquire depth information, which provides a potential to implement more discriminative and robust LST features. In this paper, we introduce the new 4-D color-depth (CoDe4D) LST feature that incorporates both intensity and depth information acquired from RGB-D cameras. Our feature detector constructs a saliency map through applying independent filters in xyzt dimension to represent texture, shape and pose variations, and selects its local maxima as interest points. Our multichannel orientation histogram descriptor applies a 4-D support region, which is adaptive to linear perspective view changes, on each interest point. Then, image gradients of color-depth patches within the support region are computed and quantized using a spherical coordinate-based method to form a final feature vector. We build a complete activity recognition system by combining our features with bag-of-features representations and support vector machines. To evaluate the performance of our CoDe4D LST features and the complete system, we conduct experiments using four benchmark color-depth human activity data sets, including UTK Action3-D, Berkeley MHAD, ACT42, and MSR daily activity 3-D data sets. Experimental results demonstrate the promising representative power of our CoDe4D features, which obtain the state-of-the-art performance on activity recognition from RGB-D visual data.

Activity prediction is an essential task in practical human-centered robotics applications, such as security, assisted living, etc., which targets at inferring ongoing human activities based on incomplete observations. To address this challenging problem, we introduce a novel bio-inspired predictive orientation decomposition (BIPOD) approach to construct representations of people from 3D skeleton trajectories. Our approach is inspired by biological research in human anatomy. In order to capture spatio-temporal information of human motions, we spatially decompose 3D human skeleton trajectories and project them onto three anatomical planes (i.e., coronal, transverse and sagittal planes); then, we describe short-term time information of joint motions and encode high-order temporal dependencies. By estimating future skeleton trajectories that are not currently observed, we endow our BIPOD representation with the critical predictive capability. Empirical studies validate that our BIPOD approach obtains promising performance, in terms of accuracy and efficiency, using a physical TurtleBot2 robotic platform to recognize ongoing human activities. Experiments on benchmark datasets further demonstrate that our new BIPOD representation significantly outperforms previous approaches for real-time activity classification and prediction from 3D human skeleton trajectories.

Activity recognition of multi-individuals (ARMI) within a group, which is essential to practical human-centered robotics applications such as childhood education, is a particularly challenging and previously not well studied problem. We present a novel adaptive human-centered (AdHuC) representation based on local spatio-temporal features (LST) to address ARMI in a sequence of 3D point clouds. Our human-centered detector constructs affiliation regions to associate LST features with humans by mining depth data and using a cascade of rejectors to localize humans in 3D space. Then, features are detected within each affiliation region, which avoids extracting irrelevant features from dynamic background clutter and addresses moving cameras on mobile robots. Our feature descriptor is able to adapt its support region to linear perspective view variations and encode multi-channel information (i.e., color and depth) to construct the final representation. Empirical studies validate that the AdHuC representation obtains promising performance on ARMI using a Meka humanoid robot to play multi-people Simon Says games. Experiments on benchmark datasets further demonstrate that our adaptive human-centered representation outperforms previous approaches for activity recognition from color-depth data.

Adaptation is an essential capability for intelligent robots to work in new environments. In the learning framework of Programming by Demonstration (PbD) and Reinforcement Learning (RL), a robot usually learns skills from a latent feature space obtained by dimension reduction techniques. Because the latent space is optimized for a specific environment during the training phase, it typically contains fewer variations. Accordingly, searching for a solution within the latent space can be less effective for robot adaptation to new environments with unseen changes. In this paper, we propose a novel Feature Space Decomposition (FSD) approach to effectively address the robot adaptation problem, which is directly applicable to the learning framework based on PbD and RL. Our FSD method decomposes the high-dimensional original features extracted from the demonstration data into principal and non-principal feature space. Then, the non-principal features are used to form a new low-dimensional search space for autonomous robot adaptation based on RL, which is initialized using a generalized trajectory represented by a Gaussian Mixture Model that is learned from the principal features. The scalability of our FSD approach guarantees that optimal solutions can be found in the new non-principal space, if they exist in the original feature space. Experimental results on real robots validate that our FSD approach enables the robots to effectively adapt to new environments, and is usually able to find optimal solutions more quickly than traditional approaches when significant environment changes occur.

Instruction of students with intellectual disability (ID) presents both unique challenges and a compelling opportunity for socially embedded robots to empower an important group in our population. We propose the creation of an autonomous, intelligent robot instructor (IRI) to teach socially valid life skills to students with ID. We present the construction of a complete IRI system for this purpose. Experimental results show the IRI is capable of teaching a non-trivial life skill to students with ID, and participants feel interaction with the IRI is beneficial.

Understanding human activities is an essential capability for intelligent robots to help people in a variety of applications. Humans perform activities in a continuous fashion, and transitions between temporally adjacent activities are gradual. Our Fuzzy Segmentation and Recognition (FuzzySR) algorithm explicitly reasons about gradual transitions between continuous human activities. Our objective is to simultaneously segment a given video into a sequence of events and recognize the activity contained in each event. The algorithm uniformly segments the video into a sequence of nonoverlapping blocks, each lasting a short period of time. Then, a multivariable time series is formed by concatenating block-level human activity summaries that are computed using topic models over local spatiotemporal features extracted from each block. Through encoding an event as a fuzzy set with fuzzy boundaries to represent gradual transitions, our approach is capable of segmenting the continuous visual data into a sequence of fuzzy events. By incorporating all block summaries contained in an event, our algorithm determines the activity label for each event. To evaluate performance, we conduct experiments using six datasets. Our algorithm shows promising continuous activity segmentation results on these datasets and obtains the event-level activity recognition precision of 42.6%, 60.4%, 65.2%, and 78.9% on the Hollywood-2, CAD-60, ACT $4^2$, and UTK-CAP datasets, respectively.

We present a novel feature description algorithm to describe 3D local spatio-temporal features for human action recognition. Our descriptor avoids the singularity and limited discrimination power issues of traditional 3D descriptors by quantizing and describing visual features in the simplex topological vector space. Specifically, given a feature's support region containing a set of 3D visual cues, we decompose the cues' orientation into three angles, transform the decomposed angles into the simplex space, and describe them in such a space. Then, quadrant decomposition is performed to improve discrimination, and a final feature vector is composed from the resulting histograms. We develop intuitive visualization tools for analyzing feature characteristics in the simplex topological vector space. Experimental results demonstrate that our novel simplex-based orientation decomposition (SOD) descriptor substantially outperforms traditional 3D descriptors for the KTH, UCF Sport, and Hollywood-2 benchmark action datasets. In addition, the results show that our SOD descriptor is a superior individual descriptor for action recognition.

Although many approaches have been developed to form robot coalitions that can achieve a multirobot task, no general methods exist to execute these coalitions, especially when the coordination among the robots is tightly coupled. In this paper, we propose a coordination mechanism as the first step to address coalition execution; it provides a flexible method to reason about synergies with overlapping coalitions (thus enabling multi-tasking robots in multi-robot tasks), which not only improves efficiency, but also reduces resource requirements in task execution. This means that our approach enables tasks that cannot be easily handled before, especially when critical resources are rare but commonly required. Our approach is based on the concept of sensor constraint, which is introduced by the tight coupling (e.g., information sharing) between the robots. We show that our algorithm is sound and complete in finding a coordination solution given a few assumptions, and discuss a distributed implementation. Simulation results are provided to demonstrate the capabilities of this new approach.

Most previous research has focused on classifying single human activities contained in segmented videos. However, in real-world scenarios, human activities are inherently continuous and gradual transitions always exist between temporally adjacent activities. In this paper, we propose a Fuzzy Segmentation and Recognition (FuzzySR) algorithm to explicitly model this gradual transition. Our goal is to simultaneously segment a given video into events and recognize the activity contained in each event. Specifically, our algorithm uniformly partitions the video into a sequence of non-overlapping blocks, each of which lasts a short period of time. Then, a multi-variable time series is creatively formed through concatenating the block-level human activity summaries that are computed using topic models over each block's local spatio-temporal features. By representing an event as a fuzzy set that has fuzzy boundaries to model gradual transitions, our algorithm is able to segment the video into a sequence of fuzzy events. By incorporating all block summaries contained in an event, the proposed algorithm determines the most appropriate activity category for each event. We evaluate our algorithm's performance using two real-world benchmark datasets that are widely used in the machine vision community. We also demonstrate our algorithm's effectiveness in important robotics applications, such as intelligent service robotics. For all used datasets, our algorithm achieves promising continuous human activity segmentation and recognition results.

SUMMARY Building energy modeling (BEM) is an approach to model the energy usage in buildings for design and retrofit purposes. EnergyPlus is the flagship Department of Energy software that performs BEM for different types of buildings. The input to EnergyPlus can often extend in the order of a few thousand parameters that have to be calibrated manually by an expert for realistic energy modeling. This makes it challenging and expensive thereby making BEM unfeasible for smaller projects. In this paper, we describe the ‘Autotune’ research that employs machine learning algorithms to generate agents for the different kinds of standard reference buildings in the US building stock. The parametric space and the variety of building locations and types make this a challenging computational problem necessitating the use of supercomputers. Millions of EnergyPlus simulations are run on supercomputers that are subsequently used to train machine learning algorithms to generate agents. These agents, once created, can then run in a fraction of the time thereby allowing cost‐effective calibration of building models. Published 2014. This article is a US Government work and is in the public domain in the USA.

Kernel methods have difficulties scaling to large modern data sets. The scalability issues are based on computational and memory requirements for working with a large matrix. These requirements have been addressed over the years by using low-rank kernel approximations or by improving the solvers' scalability. However, Least Squares Support Vector Machines (LS-SVM), a popular SVM variant, and Kernel Ridge Regression still have several scalability issues. In particular, the O(n^3) computational complexity for solving a single model, and the overall computational complexity associated with tuning hyper parameters are still major problems. We address these problems by introducing an O(nlog n) approximate l-fold cross-validation method that uses a multi-level circulant matrix to approximate the kernel. In addition, we prove our algorithm's computational complexity and present empirical runtimes on data sets with approximately one million data points. We also validate our approximate method's effectiveness at selecting hyper parameters on real world and standard benchmark data sets. Lastly, we provide experimental results on using a multi level circulant kernel approximation to solve LS-SVM problems with hyper parameters selected using our method.

Zebra fish larvae have become a popular model organism to investigate genetic and environmental factors affecting behavior. However, difficulties exist in the analysis of complex behaviors from a large array of larvae. In this paper, we present the new application of machine learning techniques in bioinformatics to automatically detect and investigate the locomotor activities of zebra fish larvae. To achieve this, twelve features were defined and seven unsupervised learning methods were implemented. Next, seven performance measures were applied to evaluate and compare these methods. In order to empirically evaluate the machine learning algorithms, a large dataset was collected that contained 6847 valid instances. Using this dataset, the characteristics of the features were analyzed and the most appropriate unsupervised learning algorithm, i.e., Unweighted Pair Group Method with Arithmetic mean (UPGMA), for locomotor activity analysis was identified. In addition, UPGMA's ability to reveal underlying patterns of zebra fish locomotor activities was demonstrated. In general, this study shows that machine learning techniques have the potential to construct effective, high-throughput systems to automate the process of identifying zebra fish behaviors influenced by genetic manipulation, pharmaceuticals, and environmental toxins.

The scheduling problem has been studied extensively in the literature. Many algorithms have been developed to operate with different types of processors and tasks. In the robotics domain, when considering each robot as a processor, some of these algorithms can be directly adapted. However, most of the existing algorithms can only handle single-robot tasks, or multi-robot tasks that can be divided into single-robot tasks. As the task requirements may only be partially known, and the available (heterogeneous) robots can dynamically change, robots may be required to cooperate tightly to share different capabilities (i.e., sensors and motors). In such cases, considering scheduling for individual robots is no longer sufficient, since the robots need to work at the coalition level. Although there exist a few algorithms that also support these more complex cases, they do not represent efficient solutions that can be adapted by various multi-robot systems in a convenient manner. In this paper, we propose heuristics to address the multi-robot task scheduling problem at the coalition level, which hides the details of robot specifications, thus allowing these heuristics to be incorporated straightforwardly. These heuristics are easy to implement and efficient enough to run in real time. We provide formal analyses and simulation results to demonstrate and compare their performances.

The ability to perceive humans is an essential requirement for safe and efficient human-robot interaction. In real-world applications, the need for a robot to interact in real time with multiple humans in a dynamic, 3-D environment presents a significant challenge. The recent availability of commercial color-depth cameras allow for the creation of a system that makes use of the depth dimension, thus enabling a robot to observe its environment and perceive in the 3-D space. Here we present a system for 3-D multiple human perception in real time from a moving robot equipped with a color-depth camera and a consumer-grade computer. Our approach reduces computation time to achieve real-time performance through a unique combination of new ideas and established techniques. We remove the ground and ceiling planes from the 3-D point cloud input to separate candidate point clusters. We introduce the novel information concept, depth of interest, which we use to identify candidates for detection, and that avoids the computationally expensive scanning-window methods of other approaches. We utilize a cascade of detectors to distinguish humans from objects, in which we make intelligent reuse of intermediary features in successive detectors to improve computation. Because of the high computational cost of some methods, we represent our candidate tracking algorithm with a decision directed acyclic graph, which allows us to use the most computationally intense techniques only where necessary. We detail the successful implementation of our novel approach on a mobile robot and examine its performance in scenarios with real-world challenges, including occlusion, robot motion, nonupright humans, humans leaving and reentering the field of view (i.e., the reidentification challenge), human-object and human-human interaction. We conclude with the observation that the incorporation of the depth information, together with the use of modern techniques in new ways, we are able to create an accurate system for real-time 3-D perception of humans by a mobile robot.

While most previous research on forming coalitions mainly concentrates on loosely coupled multirobot tasks, a more challenging problem is to address tightly coupled multirobot tasks that involve close robot coordinations, which often require capability sharing. General methods for autonomous capability sharing have been shown to greatly improve the flexibility of distributed systems. However, in addition to the interaction constraints between the robots and the environment as required by the tasks, these methods may introduce additional interaction constraints between the robots based on how the capabilities are shared. The satisfiability of these constraints in the current situation determines the feasibility of the potential coalitions. To achieve system autonomy, the ability to identify the potential coalitions that are feasible for task execution is critical. In this paper, we demonstrate a general approach that incorporates this capability based on the ASyMTRe architecture. The extended architecture, which is called IQ-ASyMTRe, is able to find coalitions in which these required constraints are satisfied. When used to form coalitions, IQ-ASyMTRe sets up only feasible coalitions, thus enabling tasks to be executed autonomously. We formally present the new architecture and prove that it is sound and complete, given certain assumptions. Simulations and experimental results are provided for different applications in which the robots are able to flexibly form coalitions that are ready to execute.

Probabilistic latent semantic analysis (PLSA) has been widely used in the machine learning community. However, the original PLSAs are not capable of modeling real-valued observations and usually have severe problems with over fitting. To address both issues, we propose a novel, regularized Gaussian PLSA (RG-PLSA) model that combines Gaussian PLSAs and hierarchical Gaussian mixture models (HGMM). We evaluate our model on supervised human action recognition tasks, using two publicly available datasets. Average classification accuracies of 97.69% and 93.72% are achieved on the Weizmann and KTH Action Datasets, respectively, which demonstrate that the RG-PLSA model outperforms Gaussian PLSAs and HGMMs, and is comparable to the state of the art.

In our prior work, we proposed the IQ-ASyMTRe architecture with a measure of information quality to reason about forming coalitions in multirobot tasks. The formed coalitions are guaranteed to be executable, given the current configurations of the robots and environment. A cost and a quality measure are associated with each coalition to further determine its utility for the task. In this paper, we show that IQ-ASyMTRe-like architectures can be utilized to significantly reduce the overall complexity of task allocation by considering only executable coalitions. For implementation, we apply a layering technique such that most existing methods for task allocation can be easily incorporated. Furthermore, we introduce a general process to address situations in which no executable coalitions are available for certain tasks, and integrate it with IQ-ASyMTRe to achieve more autonomy. Such an approach is able to autonomously decompose unsatisfied preconditions of the required task behaviors into satisfiable components, in order to generate partial order plans for them accordingly. We show how this process can be implemented using a market-based approach. Simulation results are provided to demonstrate these techniques.

In our prior work, we proposed the IQ-ASyMTRe architecture as a general method to combine coalition formation and execution for tightly-coupled multirobot tasks. IQ ASyMTRe extends the ASyMTRe architecture by introducing several new mechanisms to provide more flexibility for coalition formation as well as to facilitate coalition execution. On the other hand, these mechanisms also change the process of reasoning about solutions and further increase the complexity of the solution space. In this paper, we provide improvements for utilizing the IQ-ASyMTRe architecture based on reasoning about the solution space. We introduce a method in which the exponential growth of the number of potential solutions to be searched can be avoided; instead, the search space is only of linear size for certain tasks. Unnecessary potential solutions are removed to further increase online efficiency. Moreover, the relationships between the created solution space and the complete solution space are studied, and are utilized to provide more coverage of the complete solution space for arbitrary tasks. Although these improvements are discussed with respect to IQ-ASyMTRe, they are also applicable to architectures that approach the generality that IQ-ASyMTRe achieves. Robot simulation and experimental results are provided to demonstrate that the generation and searching of the solution space can be done online (which was impractical previously even for tasks with relatively modest complexities) for certain tasks, and to illustrate how our approach impacts the solution space.

Recognizing human activities from common color image sequences faces many challenges, such as complex backgrounds, camera motion, and illumination changes. In this paper, we propose a new 4-dimensional (4D) local spatio-temporal feature that combines both intensity and depth information. The feature detector applies separate filters along the 3D spatial dimensions and the 1D temporal dimension to detect a feature point. The feature descriptor then computes and concatenates the intensity and depth gradients within a 4D hyper cuboid, which is centered at the detected feature point, as a feature. For recognizing human activities, Latent Dirichlet Allocation with Gibbs sampling is used as the classifier. Experiments are performed on a newly created database that contains six human activities, each with 33 samples with complex variations. Experimental results demonstrate the promising performance of the proposed features for the task of human activity recognition.

Many architectures have been proposed to solve tightly-coupled multirobot tasks (MT) through coalitions of heterogeneous robots. However, several issues remain unaddressed. As coalitions are formed, sensor constraints among robots are also established. For example, in a leader-follower task, follower robots must keep leader robots within their sights, while in a box-pushing task, a supervisor robot needs to track the moving direction of the box and monitor the pushing path to the goal for obstacle avoidance. The question of how to keep these constraints satisfied during the entire execution, from initial configurations to completeness of the task, remains an open issue. In addition, environmental factors, both static and dynamic, can influence the maintenance of the constraints. Moreover, problems arise when the constraints are unsatisfiable given the current circumstances. For example, the sight of the leader might be blocked or there might be obstacles blocking the view of the box. This paper proposes a general method to address these issues for various applications with sensors having certain characteristics. Our approach combines the use of sensor models, environment sampling, measures of information quality, a motion model with sampling, and a constraint model. We believe that this approach offers the first generic formulation of robotic sensor constraints that can be applied to a wide variety of applications. To illustrate this method, we apply the approach to solve robot tracking and navigation tasks both in simulation and with physical robots. Experimental results illustrate the flexibility and robustness of the approach.

In this paper we introduce a system under development to enable humans and robots to collaborate as peers on tasks in a shared physical environment, using only implicit coordination. Our system uses Conditional Random Fields to determine the human's intended goal. We show the effects of using different features to improve accuracy and the time to the correct classification. We compare the performance of the Conditional Random Fields classifiers by testing the classification accuracy with both the full observation sequence, as well as accuracy when the observations are classified as the observations occur. We show that Conditional Random Fields work well for classifying the goal of a human in a box pushing domain where the human can select one of three tasks. We discuss how this research fits into a larger system we are developing for peer-to-peer human robot teams for shared workspace interactions.

Our research focuses on anomaly detection problems in unknown environments using Wireless Sensor Networks (WSN). We are interested in detecting two types of abnormal events: sensory level anomalies (e.g., noise in an office without lights on) and time-related anomalies (e.g., freezing temperature in a mid-summer day).We present a novel, distributed, machine learning based anomaly detector that is able to detect time-related changes. It consists of three components. First, a Fuzzy Adaptive Resonance Theory (ART) neural network classifier is used to label multi-dimensional sensor data into discrete classes and detect sensory level anomalies. Over time, the labeled classes form a sequence of classes. Next, a symbol compressor is used to extract the semantic meaning of the temporal sequence. Finally, a Variable Memory Markov (VMM) model in the form of a Probabilistic Suffix Tree (PST) is used to model and detect time-related anomalies in the environment. To our knowledge, this is the first work that analyzes/models the temporal sensor data using a symbol compressor and PST in a WSN. Our proposed detection algorithm is distributed, “light-weight”, and practical for resource constrained sensor nodes. We verify the proposed approach using a volcano monitoring dataset. Our results show that this approach yields the same performance as the traditional Markov models with much less cost.

This paper presents the IQ-ASyMTRe architecture, which is aimed to address both coalition formation and execution for tightly-coupled multirobot tasks in a single framework. Many task allocation algorithms have been previously proposed without explicitly enabling the sharing of robot capabilities. Inspired by information invariant theory, ASyMTRe was introduced which enables the sharing of sensory and computational capabilities by allowing information to flow among different robots via communication. However, ASyMTRe does not provide a solution for how a coalition should satisfy sensor constraints introduced by the sharing of capabilities while executing the assigned task. Furthermore, conversions among different information types are hardcoded, which limits the flexibility of ASyMTRe. Moreover, relationships between entities (e.g., robots) and information types are not explicitly captured, which may produce infeasible solutions from the start, as the defined information type may not correspond well to the current environment settings. The new architecture introduces a complete definition of information type to guarantee the feasibility of solutions; it also explicitly models information conversions. Inspired by our previous work, IQ-ASyMTRe uses measures of information quality to guide robot coalitions to satisfy sensor constraints (introduced by capability sharing) while executing tasks, thus providing a complete and general solution. We demonstrate the capability of the approach both in simulation and on physical robots to form and execute coalitions that share sensory information to achieve tightly-coupled tasks.

This paper presents a distributed version of our previous work, called SAFDetection, which is a sensor analysis-based fault detection approach that is used to monitor tightly-coupled multi-robot team tasks.While the centralized version of SAFDetection was shown to be successful, a shortcoming of the approach is that it does not scale well to large team sizes. The distributed SAFDetection approach addresses this problem by adapting and distributing the approach across team members. Distributed SAFDetection has the same theoretic foundation as centralized SAFDetection, which maps selected robot sensor data to a robot state by using a clustering algorithm, and builds state transition diagrams to describe the normal behavior of the robot system. However, rather than processing multiple robots' sensor data centralized on a server, distributed SAFDetection performs feature selection and clustering on individual robots to build the normal behavior model of an individual robot and the entire robot team. Fault detection is also accomplished in a distributed manner. We have implemented this distributed approach on a physical robot team and in simulation. This paper presents the results of these experiments, showing that distributed SAFDetection is an efficient approach to detect both local and interactive faults in tightly-coupled multi-robot team tasks. Compared to the centralized version, this approach provides more scalability and reliability.

This paper proposes a multiple robot control algorithm to approach the problem of patrolling an open or closed line. The algorithm is fully decentralized, i.e., no communication occurs between robots or with a central station. Robots behave according only to their sensing and computing capabilities to ensure high scalability and robustness towards robots' fault. The patrolling algorithm is designed in the framework of behavioral control and it is based on the concept of Action: an higher level of abstraction with respect to the behaviors. Each Action is obtained by combining more elementary behaviors in the Null-Space-Behavioral framework. A Finite-State-Automata is designed as supervisor in charge of selecting the appropriate action. The approach has been validated in simulation as well as experimentally with a patrol of 3 Pioneer robots available at the Distributed Intelligence Laboratory of the University of Tennessee.

This paper addresses the challenge of forming appropriate heterogeneous robot teams to solve tightly-coupled, potentially multi-robot tasks, in which the robot capabilities may vary over the environment in which the task is being performed. Rather than making use of a permanent tightly-coupled robot team for performing the task, our approach aims to recognize when tight coupling is needed, and then only form tight cooperative teams at those times. This results in important cost savings, since coordination is only used when the independent operation of the team members would put mission success at risk. Our approach is to define a new semantic information type, called environmentally dependent information, which allows us to capture certain environmentally-dependent perceptual constraints on vehicle capabilities. We define locations at which the robot team must transition between tight and weak cooperation as critical junctures. Note that these critical juncture points are a function of the robot team capabilities and the environmental characteristics, and are not due to a change in the task itself. We calculate critical juncture points by making use of our prior ASyMTRe approach, which can automatically configure heterogeneous robot team solutions to enable sharing of sensory capabilities across robots. We demonstrate these concepts in experiments involving a human-controlled blimp and an autonomous ground robot in a target localization task.

In this paper, we lay the groundwork for extending our previously developed ASyMTRe architecture to enable constructivist learning for multi-robot team tasks. The ASyMTRe architecture automatically configures schemas within, and across, robots to form the highest utility solution that achieves a given multi-robot team task. We believe that the schema- based approach used in ASyMTRe is a useful abstraction not only for forming heterogeneous coalitions, but also for enabling constructivist learning, in which chunks of schemas that solve intermediate subproblems are learned and then made available for future task solutions. However, the existing ASyMTRe search algorithm for finding configurations of schemas that completely solve given tasks (Centralized ASyMTRe - CA) is not well-suited for identifying useful chunks of schemas that could solve intermediate subtasks that may be useful in the future. Thus, in this current work, we explore an Evolutionary Learning (EL) technique for the offline learning of schema chunks that could be saved and used later in an online search (using the regular CA algorithm) for coalition configurations. However, we do not want to sacrifice solution quality in making use of the evolutionary search technique. Thus, we compare the solutions discovered by the EL algorithm with those that are found using CA, as well as with a third algorithm that randomizes the CA algorithm, called RA. Four different applications in simulation are used to evaluate the EL, CA, and RA techniques. Our results show that the EL approach indeed finds solutions of comparable quality to the CA technique, while also providing the added benefit of learning highly fit partial solutions, or schema chunks, that may be beneficial for future tasks via constructivist learning. We conclude by arguing that the combination of the online CA search for solving current multi-robot tasks can be combined with the offline EL approach that can identify intermediate solutions (or schema chunks) that may be useful for future team tasks. This combination should lead to an overall efficiency improvement for identifying coalition formations, as well as for continual learning.

We have developed a novel method to estimate missing observations in wireless sensor networks. We use a hierarchical unsupervised fuzzy ART neural network to represent the data cluster prototypes. We then estimate missing inputs by using a new spatial-temporal imputation technique. We have evaluated this approach through experiments on both real sensor data and artificially generated data. Our experimental results show that our proposed approach performs better than nine other estimation algorithms including moving average and Expectation-Maximization (EM) imputation.

In this paper, we present an anomaly detection system that is able to detect time-related anomalies by using a wireless sensor network and a mobile robot. The sensor network uses an unsupervised fuzzy adaptive resonance theory (ART) neural network to learn and detect intruders in a previously unknown environment. Upon the detection of an intruder, a mobile robot travels to the position where the intruder is detected to investigate by using its camera. The wireless sensor network uses a hierarchical communication/learning structure, where the mobile robot is the root node of the tree. Our fuzzy ART network is based on Kulakov and Davcevpsilas implementation (Kulakov and Davcev, 2005). However, we enhance their work by extending the fuzzy ART neural network with a Markov model to learn a time series and detect time-related anomalies. Finally, a mobile robot is employed to verify whether the detected anomalies were caused by intruders. The proposed architecture is tested on physical hardware. Our results show that our enhanced detection system with mobile robot verification has a higher accuracy and lower false alarm rate than the original fuzzy ART system.

This paper presents a sensor analysis based fault detection approach (which we call SAFDetection) that is used to monitor tightly-coupled multi-robot team tasks. Our approach aims at detecting both physical and logic faults of a robot system with little prior knowledge on the system. We do not need the motion model or a priori knowledge of the possible fault types of the monitored system. Our approach treats the monitored robot system as a black box, with only sensor data available. Thus, we believe the approach is general, and can be used in a wide variety of robot systems performing many different kinds of tasks. Our approach combines data clustering techniques with the generation of a probabilistic state diagram to model the normal operation of the multi-robot system. We have implemented this approach on a physical robot team. This paper presents the results of these experiments, which show that sensor data analyzed from a training phase of normal operation can be used to generate a model of normal robot team operation. This model can then be used to detect many types of abnormal behavior of the system, based purely on monitoring the sensor data of the system.

This paper presents an approach that enables heterogeneous robots to automatically form groups as needed to generate both strongly-cooperative and weakly-cooperative multi-robot task solutions in the same application. The fundamental contribution of this work is the layering of our low-level coalition formation algorithm for generating strongly-cooperative task solutions, with high-level, traditional task allocation methods for weakly-cooperative task solutions. At the low level, coalitions that generate strongly-cooperative multi-robot task solutions are formed using our ASyMTRe-D approach that maps environmental sensors and perceptual and motor schemas to the required flow of information in the robot team, automatically reconfiguring the connections of schemas within and across robots to form efficient solutions. At the high level, a traditional task allocation approach is used to enable individual robots and/or coalitions to compete for weakly-cooperative task assignments through task allocation. We introduce the site clearing task to motivate the work, and then formalize the problem. We then present the approach of layering ASyMTRe-D with task allocation. We validate the approach on a team of robots with the site clearing task. We believe the resulting approach is a flexible system that can handle a broad range of realistic multi-robot applications beyond what is possible using other existing approaches.

Any system that has the capability to diagnose and recover from faults is considered to be a fault-tolerant system. Additionally, the quality of the incorporated fault-tolerance has a direct impact on the overall performance of the system. Hence, being able to measure the extent and usefulness of fault- tolerance exhibited by the system would provide the designer with a useful analysis tool for better understanding the system as a whole. Unfortunately, it is difficult to quantify system fault-tolerance on its own for intelligent systems. A more useful metric for evaluation is the "effectiveness" measure of fault- tolerance. The influence of fault-tolerance towards improving overall performance determines the overall effectiveness or quality of the system. In this paper, we outline application- independent metrics to measure fault-tolerance within the context of system performance. In addition, we also outline potential methods to better interpret the obtained measures towards understanding the capabilities of the implemented system. Furthermore, a main focus of our approach is to capture the effect of intelligence, reasoning, or learning on the effective fault-tolerance of the system, rather than relying purely on traditional redundancy based measures. We show the utility of the designed metrics by applying them to different fault-tolerance architectures implemented for multiple complex heterogeneous multi-robot team applications and comparing system performance. Finally, we contrast the developed metrics with the only other existing method (HWB method) for evaluating (that we are aware of) effective fault-tolerance for multi-robot teams and rate them in terms of their capability to best interpret the workings of the implemented systems. To the best of our knowledge, this is the first metric that attempts to evaluate the quality of learning towards understanding system level fault-tolerance.

This paper presents an adaptive causal model method (adaptive CMM) for fault diagnosis and recovery in complex multi-robot teams. We claim that a causal model approach is effective for anticipating and recovering from many types of robot team errors, presenting extensive experimental results to support this claim. To our knowledge, these results show the first, full implementation of a CMM on a large multi-robot team. However, because of the significant number of possible failure modes in a complex multi-robot application, and the difficulty in anticipating all possible failures in advance, our empirical results show that one cannot guarantee the generation of a complete a priori causal model that identifies and specifies all faults that may occur in the system. Instead, an adaptive method is needed to enable the robot team to use its experience to update and extend its causal model to enable the team, over time, to better recover from faults when they occur. We present our case-based learning approach, called LeaF (for learning-based fault diagnosis), that enables robot team members to adapt their causal models, thereby improving their ability to diagnose and recover from these faults over time

This paper presents a reasoning system that enables a group of heterogeneous robots to form coalitions to accomplish a multirobot task using tightly coupled sensor sharing. Our approach, which we call ASyMTRe, maps environmental sensors and perceptual and motor control schemas to the required flow of information through the multirobot system, automatically reconfiguring the connections of schemas within and across robots to synthesize valid and efficient multirobot behaviors for accomplishing a multirobot task. We present the centralized anytime ASyMTRe configuration algorithm, proving that the algorithm is correct, and formally addressing issues of completeness and optimality. We then present a distributed version of ASyMTRe, called ASyMTRe-D, which uses communication to enable distributed coalition formation. We validate the centralized approach by applying the ASyMTRe methodology to two application scenarios: multirobot transportation and multirobot box pushing. We then validate the ASyMTRe-D implementation in the multirobot transportation task, illustrating its fault-tolerance capabilities. The advantages of this new approach are that it: 1) enables robots to synthesize new task solutions using fundamentally different combinations of sensors and effectors for different coalition compositions and 2) provides a general mechanism for sharing sensory information across networked robots

This paper describes a methodology for automat ically synthesizing task solutions for heterogeneous multi-robot teams. In contrast to prior approaches that require a manual pre definition of how the robot team will accomplish its task (while perhaps automating who performs which task), our approach automates both the how and the who to generate task solution approaches that were not explicitly defined by the designer a priori. The advantages of this new approach are that it: (1) enables the robot team to synthesize new task solutions that use fundamentally different combinations of robot behaviors for different team compositions, and (2) provides a general mechanism for sharing sensory information across networked robots, so that more capable robots can assist less capable robots in accomplishing their objectives. Our approach, which we call ASyMTRe (Automated Synthesis of Multi-robot Task solutions through software Reconfiguration, pronounced “Asymmetry”), is based on mapping environmental, perceptual, and motor control schemas to the required flow of information through the multi-robot system, automatically reconfiguring the connections of schemas within and across robots to synthesize valid and efficient multi-robot behaviors for accomplishing the team objectives. We validate this approach by presenting the results of applying our methodology to two different teaming scenarios: altruistic cooperation involving multi-robot transportation, and coalescent cooperation involving multi-robot box pushing.

This paper presents a distributed reasoning system, called ASyMTRe-D, which enables a team of robots to form coalitions to accomplish a multi-robot task through tightly-coupled sensor sharing. The theoretical foundation of the negotiation protocol is ASyMTRe, an approach we developed previously to synthesize task solutions according to the task requirements and the team composition. The goal of the ASyMTRe approach is to increase the task solution capabilities of heterogeneous multi-robot teams by changing the fundamental abstraction from the typical "task" abstraction to a "schema" abstraction and automatically reconfigure the schemas to address the task at hand. The decision-making in this prior work was fully centralized; the current paper presents a distributed version of this approach based on the contract net protocol, which can achieve higher levels of robustness than the centralized version. The purpose here is not to improve the original protocol, but to apply it to our problem so that the autonomous task solution capabilities of robots can be achieved in a distributed manner. Simulation results are provided to validate the protocol with performance analysis. Finally, we compare ASyMTRe-D with the centralized ASyMTRe. Our future objective is to enable the human designer to specify the desired balance between solution quality and robustness, enabling the reasoning approach to invoke the appropriate level of information-sharing among robots to reach the specified solution characteristics.

This wok describes novel algorithms for planning the deployment of a large number of navigationally-challenged mobile sensor nodes in known indoor environments. Due to cost and power constraints, our mobile sensor nodes have the ability to move and communicate, but they cannot detect or avoid obstacles or localize to the environment. Additionally, they have only minimal capabilities for detecting other robot team members, through the use of a crude camera. Our deployment process, therefore, uses an assistive navigation technique that enables a more capable leader robot, equipped with a laser rangefinder and a camera, to guide several mobile sensor nodes to their deployment positions. To ensure the successful deployment of the mobile sensor nodes, we have developed an autonomous planning process that plans the positions of the sensor nodes based upon a number of constraints, including maintaining line of sight maximizing visibility coverage, avoiding placement in doorways, minimizing obstruction of corridors, and so forth. Additionally, because of the navigational constraints of simple robots following a leader to these deployment positions, our algorithm also derives two leader waypoints for each sensor position, which on strain the motion of the leader path to the deployment position. These leader

This paper presents the design and results of autonomous behaviors for tightly-coupled cooperation in heterogeneous robot teams, specifically for the task of navigation assistance. These cooperative behaviors enable capable, sensor rich ("leader") robots to assist in the navigation of sensor-limited ("simple") robots that have no onboard capabilities for obstacle avoidance or localization, and only minimal capabilities for kin recognition. The simple robots must be dispersed throughout a known, indoor environment to serve as a sensor network. However, because of their navigation limitations, they are unable to autonomously disperse themselves or move to planned sensor deployment positions independently. To address this challenge, we present cooperative behaviors for heterogeneous robots that enable the successful deployment of sensor-limited robots by assistance from more capable leader robots. These heterogeneous cooperative behaviors are quite complex, and involve the combination of several behavior components, including vision-based marker detection, autonomous teleoperation, color marker following in robot chains, laser-based localization, map-based path planning, and ad hoc mobile networking. We present the results of the implementation and extensive testing of these behaviors for deployment in a rigorous test environment. To our knowledge, this is the most complex heterogeneous robot team cooperative task ever attempted on physical robots. We consider it a significant success to have achieved such a high degree of system effectiveness, given the complexity of the overall heterogeneous system.

This paper presents work extending previous research in sensor fault tolerance, classification, and recovery from a single robot to a heterogeneous team of distributed robots. This approach allows teams of robots to share knowledge about the working environment, sensor and task state, to diagnose failures and also communicate to redistribute tasks in the event that a robot becomes inoperable. Our work presents several novel extensions to prior art: distributed fault handling and task management in a dynamic, distributed Java framework. This research was implemented and demonstrated on robots in a lab environment performing a simplified search operation.

This paper presents an approach that enables a mobile "interceptor" robot to intercept targets in an indoor environment using information from a distributed acoustic sensor network. The approach assumes the indoor environment has been previously mapped and that the sensor nodes know their position in the map. The targets are localized in the sensor network based upon local maxima of the acoustic volume. The current target localization information is reported to an interceptor robot, which utilizes a dual wavefront path planner to move from its current location to a location that is within visibility range of a target. Results of the complete implementation of this approach using 70 sensor net robots in the player/stage multi-robot simulator are reported, as well as implementation results to date on a team of physical robots. To our knowledge, this is the first implementation of a multi-robot system that combines the use of an acoustic sensor net for target detection with an interceptor robot that can efficiently reach the moving position of the detected target in indoor environments.

This paper presents an approach for deploying a team of mobile sensor nodes to form a sensor network in indoor environments. The challenge in this work is that the mobile sensor nodes have no ability for localization or obstacle avoidance. Thus, our approach entails the use of more capable "helper" robots that "herd" the mobile sensor nodes into their deployment positions. To extensively explore the issues of heterogeneity in multi-robot teams, we employ the use of two types of helper robots-one that acts as a leader and a second that: 1) acts as a follower and 2) autonomously teleoperates the mobile sensor nodes. Due to limited sensing capabilities, neither of these helper robots can herd the mobile sensor nodes alone; instead, our approach enables the team as a whole to successfully accomplish the sensor deployment task. Our approach involves the use of line-of-sight formation keeping, which enables the follower robot to use visual markers to move the group along the path executed by the leader robot. We present results of the implementation of this approach in simulation, as well as results to date in the implementation on physical robot systems. To our knowledge, this is the first implementation of robot herding using such highly heterogeneous robots, in which no single type of robot could accomplish the sensor network deployment task, even if multiple copies of that robot type were available.

The cooperative leader following task for multi-robot teams is introduced and discussed. We describe the design and implementation of a distributed technique to coordinate team level and robot level behaviors for this task, as well as a multi-threaded framework for the implementation of a multi-robot system with heterogeneous sensing capabilities. This approach enables robots to remain in formation as they deal with other obstacles that may appear within the formation. We describe how single robot behaviors are realized and scheduled. We show some of the results of the team implementations. The proposed approach has been run and validated on a team of robots performing both in indoor and outdoor environments.

We describe a scheme for building terrain maps of realistic outdoor environments by having multiple robotic agents navigate in them. The terrain map combines vision-based depth information of environmental features with an elevation gradient created by fusing vertical displacements obtained from inclinometer pitch angles with DGPS altitude data. Experimental results are presented to illustrate the practical application of this scheme.

We propose a distributed and optimal motion planning algorithm for multiple robots. The computationally expensive problem is decomposed into two modules: path planning and velocity planning. The D* search method is applied in both modules, based on either geometric formulation or schedule formulation. Optimization is achieved at the individual robot level by defining cost functions, and also at the team level by a global measurement function reflecting performance indices of interest as a team. Contrary to our knowledge of previous results on multi-robot motion planning that either obtain optimal solutions through centralized and exhaustive computing, or achieve distributed implementations without considering any optimization issues, our approach combines these two features and explicitly optimizes performance functions through a distributed implementation. It is also one of the few that is capable of handling outdoor rough terrain environments and real time replanning. Simulations are shown on a Mars-like rough terrain using a 3D vehicle planner and control simulator. The algorithm was also implemented and successfully run on a group of Nomad 200 indoor robots.

An extended Kalman filter (EKF)-based algorithm for the localization of a team of robots is described in this paper. The distributed EKF localization scheme is straightforward in that the individual robots maintain a pose estimate using EKFs that are local to every robot. We then show how these results can be extended to perform heterogeneous cooperative localization in the absence or degradation of absolute sensors aboard the team members. The proposed algorithms are implemented using field data obtained from a team of ATRV-mini robots traversing on uneven outdoor terrain.

A current limitation in multi-robot systems research is the lack of metrics that enable the performance comparison of various approaches to cooperative control. Ideally, metrics would be available that provide quantitative evaluations of cooperative robot teams based upon high-level design goals, such as fault tolerance, adaptivity, and so forth. In current research, however, metrics are limited to application-dependent measures. This paper summarizes the metrics that have been used in eight implementations of the ALLIANCE architecture in multi-robot teams. We explore the implications of the chosen metrics and offer suggestions for developing less application-dependent metrics in the future.

Presents our initial results in comparing three algorithms for autonomous robotic mapping using two types of laser scanner data. The algorithms compared are the Markov localization approach of S. Thrun et al. (1998), F. Lu and E. Milios's (1997) iterative dual correspondence algorithm, and C. Touzet's (2000) model-free landmark extraction algorithm. The two types of laser scanner data utilized are the AccuRange laser scanner from Acuity and the SICK laser scanner. We compare these algorithms in terms of the quality of mapped results, computational requirements, and sensitivity to data and odometry errors. While the complete comparison of these algorithms on all these measures is not yet accomplished, our results to date indicate that laser mapping algorithms are not immediately transferrable from one type of laser scanner data to another. Instead, algorithms appear to make implicit assumptions on the quality or content of laser data that play a strong role in the quality of the mapping results.

ALLIANCE is a software architecture that facilitates the fault tolerant cooperative control of teams of heterogeneous mobile robots performing missions composed of loosely coupled subtasks that may have ordering dependencies. ALLIANCE allows teams of robots, each of which possesses a variety of high-level functions that it can perform during a mission, to individually select appropriate actions throughout the mission based on the requirements of the mission, the activities of other robots, the current environmental conditions, and the robot's own internal states. ALLIANCE is a fully distributed, behaviour-based architecture that incorporates the use of mathematically-modeled motivations (such as impatience and acquiescence) within each robot to achieve adaptive action selection. Since cooperative robotic teams usually work in dynamic and unpredictable environments, this software architecture allows the robot team members to respond robustly, reliably, flexibly, and coherently to unexpected environmental changes and modifications in the robot team that may occur due to mechanical failure, the learning of new skills, or the addition or removal of robots from the team by human intervention. The feasibility of this architecture is demonstrated in an implementation on a team of mobile robots performing a laboratory version of hazardous waste cleanup.

An important issue that arises in the automation of many security, surveillance, and reconnaissance tasks is that of monitoring, or observing, the movements of targets navigating in a bounded area of interest. A key research issue in these problems is that of sensor placement-determining where sensors should be located to maintain the targets in view. In complex applications of this type, the use of multiple sensors dynamically moving over time is required. In this paper, we investigate the use of a cooperative team of autonomous sensor-based robots for multi-robot observation of multiple moving targets. We focus primarily on developing the distributed control strategies that allow the robot team to attempt to maximize the collective time during which each object is being observed by at least one robot in the area of interest. Our initial efforts in this problem address the aspects of distributed control in homogeneous robot teams with equivalent sensing and movement capabilities working in an uncluttered, bounded area. This paper first formalizes the problem, discusses related work, and then shows that this problem is NP-hard. We then present a distributed approximate approach to solving this problem that combines low-level multi-robot control with higher-level control. The low-level control is described in terms of force fields emanating from the targets and the robots. The higher level control is presented in the ALLIANCE formalism, which provides mechanisms for fault tolerant cooperative control, and allows robot team members to adjust their low-level actions based upon the actions of their teammates. We then present the results of the implementation of portions of our approach, both in simulation and on physical robots.

An important issue that arises in the automation of many security, surveillance, and reconnaissance tasks is that of monitoring the movements of targets navigating in a bounded area of interest. In this paper, we investigate the use of a cooperative team of autonomous sensor-based robots for the observation of multiple moving targets. We focus primarily on developing the distributed control strategies that allow the robot team to attempt to minimize the total time in which targets escape observation by some robed team member in the area of interest. This paper first formalizes the problem and discusses related work. We then present a distributed approximate approach to solving this problem that combines low-level multi-robot control with higher-level reasoning control based on the ALLIANCE formalism. We analyze the effectiveness of our approach by comparing it to three other feasible algorithms for cooperative control, showing the superiority of our approach for a large class of problems.

A large application domain for multi-robot teams involves task-oriented missions, in which potentially heterogeneous robots must solve several distinct tasks. Previous research addressing this problem in multi-robot systems has largely focused on issues of efficiency, while ignoring the real-world situated robot needs of fault tolerance and adaptivity. This paper addresses this problem by developing an architecture called L-ALLIANCE that incorporates task-oriented action selection mechanisms into a behavior-based system, thus increasing the efficiency of robot team performance while maintaining the desirable characteristics of fault tolerance and adaptivity. We present our investigations of several competing control strategies and derive an approach that works well in a wide variety of multi-robot task-oriented mission scenarios. We provide a formal model of this technique to illustrate how it can be incorporated into any behavior-based system.

Previous research in cooperative robotics has investigated several possible ways of coordinating the actions of cooperative teams-from implicit cooperation through sensory feedback to explicit cooperation using the exchange of communicated messages. These various approaches differ in the extent to which robot team members are aware of, or recognize the actions of their teammates and the extent to which they use this information to effect their own actions. The research described in this paper investigates this issue of robot awareness of team member actions and its effect on cooperative team performance by examining the results of a series of experiments on teams of mobile robots performing a puck moving mission. In these experiments, the author varies the team size (and thus the level of redundancy in team member capabilities) and the level of awareness robots have of their teammates' current actions and evaluate the team's performance using two metrics: time and energy. The results indicate that the impact of action awareness on cooperative team performance is a function not only of team size and the metric of evaluation, but also on the degree to which the effects of actions can be sensed through the world, the relative amount of work that is available per robot, and the cost of replicated actions. Based on these empirical studies, the author discusses the impact of action recognition and robot awareness on cooperative team design.

This research addresses the problem of achieving fault tolerant cooperation within small- to medium-sized teams of heterogeneous mobile robots. We describe a novel behavior-based, fully distributed architecture, called ALLIANCE, that utilizes adaptive action selection to achieve fault tolerant cooperative control in robot missions involving loosely coupled, largely independent tasks. The robots in this architecture possess a variety of high-level functions that they can perform during a mission, and must at all times select an appropriate action based on the requirements of the mission, the activities of other robots, the current environmental conditions, and their own internal states. The software architecture allows the team members to respond robustly and reliably to unexpected environmental changes and modifications in the robot team that may occur due to mechanical failure, the learning of new skills, or the addition or removal of robots from the team by human intervention. After presenting ALLIANCE, we describe in detail our experimental results of an implementation of this architecture on a team of physical mobile robots performing a cooperative box pushing demonstration.>

Recent work toward the development of low-complexity, sensor-based inferencing methods to serve as the initial links of incremental robotic learning systems is described. A multimodal learning approach is proposed in which distributed sensory sources are used to both trigger the observation of and perceive relevant learning instances in a human-robot synergistic framework. Three components of the incremental learning system for the CESARm advanced manipulator testbed are presented that encompass the learning of objects and work area characteristics through the triggering of attention and rote learning, the learning of elemental manipulation tasks by observation of human actions, and the self-assessment of acquired skills and learned knowledge through task performance evaluation. Feasibility experiments with each of these three learning methodologies are presented, and some results are discussed.>