Recognition of human manipulation actions in real-time is essential for safe and effective human-robot interaction and collaboration. The challenge lies in developing a model that is both lightweight enough for real-time execution and capable of generalization. While some existing methods in the literature can run in real-time, they struggle with temporal scalability, i.e., they fail to adapt to long-duration manipulations effectively. To address this, leveraging the generalizable scene graph representations, we propose a new Factorized Graph Sequence Encoder network that not only runs in real-time but also scales effectively in the temporal dimension, thanks to its factorized encoder architecture. Additionally, we introduce Hand Pooling operation, a simple pooling operation for more focused extraction of the graph-level embeddings. Our model outperforms the previous state-of-the-art real-time approach, achieving a 14.3% and 5.6% improvement in F1-macro score on the KIT Bimanual Action (Bimacs) Dataset and Collaborative Action (CoAx) Dataset, respectively. Moreover, we conduct an extensive ablation study to validate our network design choices. Finally, we compare our model with its architecturally similar RGB-based model on the Bimacs dataset and show the limitations of this model in contrast to ours on such an object-centric manipulation dataset. Our code and trained models are available at https://github.com/eneserdo/FGSE.

We need robots more aware of the unintended outcomes of their actions for ensuring safety. This can be achieved by an onboard failure detection system to monitor and detect such cases. Onboard failure detection is challenging with a limited set of onboard sensor setup due to the limitations of sensing capabilities of each sensor. To alleviate these challenges, we propose FINO-Net, a novel multimodal sensor fusion based deep neural network to detect and identify manipulation failures. We also introduce FAILURE, a multimodal dataset, containing 229 real-world manipulation data recorded with a Baxter robot. Our network combines RGB, depth and audio readings to effectively detect failures. Results indicate that fusing RGB with depth and audio modalities significantly improves the performance. FINO-Net achieves %98.60 detection accuracy on our novel dataset. Code and data are publicly available at https://github.com/ardai/fino-net.

Service robots need to handle a variety of everyday manipulation tasks to accomplish household chores such as cooking and cleaning. Successful execution of these tasks is highly dependent on how reliable the robot perceives its environment through noisy sensing. We present a visual world modeling system for service robots to generate and maintain accurate models of their environments for continuous scenarios. This system is designed to provide a generic platform for both humanoid and ground manipulation robots using different types of vision sensors and algorithms. In our particular implementation, visual data processed by different perception algorithms are used for building and continuously updating a world model of the environment. We evaluate our system on a variety of object manipulation scenarios and show that the system produces consistent perception outcomes suitable for different manipulation tasks.

Handling safety is crucial to achieve lifelong autonomy for robots. Unsafe situations might arise during manipulation in unstructured environments due to noises in sensory feedback, improper action parameters, hardware limitations or external factors. In order to assure safety, continuous execution monitoring and failure detection procedures are mandatory. To this end, we present a multimodal failure monitoring and detection system to detect manipulation failures. Rather than relying only on a single sensor modality, we consider integration of different modalities to get better detection performance in different failure cases. In our system, high level proprioceptive, auditory and visual predicates are extracted by processing each modality separately. Then, the extracted predicates are fused. Experiments on a humanoid robot for tabletop manipulation scenarios indicate that the contribution of each modality is different depending on the action in execution and multimodal fusion results in an overall performance increase in detecting failures compared to the performance attained by unimodal processing.

https://doi.org/10.5220/0005711403620369

Robots should avoid potential failure situations to safely execute their actions and to improve their performances. For this purpose, they need to build and use their experience online. We propose online learning-guided planning methods to address this problem. Our method includes an experiential learning process using Inductive Logic Programming (ILP) and a probabilistic planning framework that uses the experience gained by learning for improving task execution performance. We analyze our solution on a case study with an autonomous mobile robot in a multi-object manipulation domain where the objective is maximizing the number of collected objects while avoiding potential failures using experience. Our results indicate that the robot using our adaptive planning strategy ensures safety in task execution and reduces the number of potential failures.

In this paper, we introduce an automated reasoning system for learning object behaviors and interactions through the observation of event sequences. We use an existing system to learn the models of objects and further extend it to model more complex behaviors. Furthermore, we propose a spatio-temporal reasoning based learning method for reasoning about interactions among objects. Experience gained through learning is to be used for achieving goals by these objects. We take The Incredible Machine game (TIM) as the main testbed to analyze our system. Tutorials of the game are used to train the system. We analyze the results of our reasoning system on four different input types: a knowledge base of relations; spatial information; temporal information; and spatio-temporal information from the environment. Our analysis reveals that if a knowledge base about relations is provided, most of the interactions can be learned. We have also demonstrated that our learning method which incorporates both spatial and temporal information gives close results to that of the knowledge-based approach. This is promising as gathering spatio-temporal information does not require prior knowledge about relations. Our second analysis of the spatio-temporal reasoning method in the Electric Box computer game domain verifies the success of our approach.

However, since the real world is partially observable and dynamic, failures may occur during the execution of their actions. These failures should be detected immediately, and the underlying reasons of these failures should be isolated to ensure robustness. In this paper, we propose a probabilistic and temporal model-based failure isolation method that maintains Hierarchical Hidden Markov Models (HHMMs) in order to represent and reason about different failure types. The underlying reason of a failure can be isolated efficiently by multi-hypothesis tracking. 1

When an agent plans a sequence of actions, some unexpected events may occur during the execution of these actions. These unexpected events may prevent the agent to replan and achieve its goal. In this work, our purpose is to recover from plan execution failures by reasoning the causes of these faulties. We combine the TLPlan forward chaining temporal planner with the PROBCOG reasoning tool in order to handle failures. It is also quite important to decide whether the failure we are dealing with is permanent. We propose that inferring some properties of the failure source helps us handle failures and determine the failure types.

We propose a method for learning interactions among objects when intermediate state information is not available. Learning is accomplished by observing a given sequence of actions on different objects. We have selected the Incredible Machine game as a suitable domain for analyzing and learning object interactions. We first present how behaviors are represented by finite state machines using the given input. Then, we analyze the impact of the knowledge about relations on the overall performance. Our analysis includes four different types of input: a knowledge base including part relations; spatial information; temporal information; and spatio-temporal information. We show that if a knowledge base about relations is provided, learning is accomplished to a desired extent. Our analysis also indicates that the spatio-temporal approach is superior to the spatial and the temporal approaches and gives close results to that of the knowledge-based approach.

We consider planning domains with both discrete and continuous changes. Continuous change occurs especially when agents share time-dependent critical resources. In these cases, besides discrete and continuous changes, their interactions should also be taken into consideration. However concurrency of durative actions with interacting continuous effects cannot be exploited by existing temporal planners. To overcome this problem, we propose an action lifting approach and we analyze path sharing problem to illustrate interaction of continuous linear effects in the planning domain.

When the tasks of a mission are interrelated and subject to several resource constraints, more efforts are needed to coordinate robots towards achieving the mission than independent tasks. In this work, we formulate the Coordinated Task Selection Problem (CTSP) to form the basis of an efficient dynamic task selection scheme for allocation of interrelated tasks of a complex mission to the members of a multi-robot team. Since processing times of tasks are not exactly known in advance, the incremental task selection scheme for the eligible tasks prevents redundant efforts as, instead of scheduling all of the tasks, they are allocated to robots as needed. This approach results in globally efficient solutions through mechanisms that form priority based rough schedules and select the most suitable tasks from these schedules. Since our method is targeted at real world task execution, communication requirements are kept limited. Empirical evaluations of the proposed approach are performed on the Webots simulator and the real robots. The results validate that the proposed approach is scalable, efficient and suitable to the real world safe mission achievement.

This paper introduces ArtHistorian, a content-based classification and indexing system that represents the visual content of art paintings by a six-dimensional feature set. The introduced feature set is robust to scale changes and can handle variations in lighting conditions. A nonlinear SVM classifier included in the system learns the characteristics of fundamental art movements and painting styles. A hybrid classifier that combines PCA representation of paintings with the SVM classification is also exploited. It is shown that ArtHistorian is capable of classifying art paintings based on painters as well as art movements with an accuracy of greater than 90% and its false alarm ratio is very small. The developed system enables the user to run content-based queries and to retrieve from painting databases created in XML format.

In this paper, we propose a serving system consisting intelligent agents processing society information in a multi-user domain. The agents use the similarity information on the user preferences for providing services with the best quality. This similarity information is formed by using the clustering techniques. The agents are allowed to interact with others and exchange information based on this information. The agents' beliefs are formed based on the clustering process results and the user feedback for the information gained by the interactions. These beliefs are used to provide a continuous service for the user. The natural division of tasks among the agents reduces the workload for each agent. The cluster structure of the society may be changed according to the satisfaction levels of the users. Some general society parameters of the system are learned by a reinforcement learning method. We designed the agents by using our proposed social interactive agent model. The system model can be used for many of the serving systems by changing the service type.