An object handover between a robot and a human is a coordinated action which is prone to failure for reasons such as miscommunication, incorrect actions and unexpected object properties. Existing works on handover failure detection and prevention focus on preventing failures due to object slip or external disturbances. However, there is a lack of datasets and evaluation methods that consider unpreventable failures caused by the human participant. To address this deficit, we present the multimodal Handover Failure Detection dataset, which consists of failures induced by the human participant, such as ignoring the robot or not releasing the object. We also present two baseline methods for handover failure detection: (i) a video classification method using 3D CNNs and (ii) a temporal action segmentation approach which jointly classifies the human action, robot action and overall outcome of the action. The results show that video is an important modality, but using force-torque data and gripper position help improve failure detection and action segmentation accuracy.

Current approaches in education, while aiming to be universally effective, often struggle to fully adapt to the unique needs and communication styles of individual children; this disparity can limit the children’s engagement and hinder their learning progress. Similarly, parents or guardians, despite their good intentions, may also be unable to provide consistent and personalized support to each child. In this work, we investigate the use of conversational systems for socially assistive robots (SARs) as a potential solution to this problem, as such systems have the potential to allow children to interact and learn at their own pace, in a way that aligns with their communication preferences. To ensure that the robot’s language is suitable for children, we present a system that leverages a combination of natural language processing (NLP) techniques, including dialog management, child-friendly language generation, and context-aware response adaptation; to achieve this, our system combines Rasa for dialog management, GPT-3.5 for language generation, and textstat for language complexity evaluation. We evaluate the suitability of the generated language for a young audience through two user studies with adult participants, one in which the conversational system was embodied in a robot and involved direct interaction between a human and a robot, and another where participants evaluated conversational transcripts from the first study. Our results suggest that the system has the potential to maintain engaging and safe conversations, and adapt its language to individual needs.

Robot behaviour models in socially assistive robotics are typically trained using high-level features, such as a user’s engagement, such that inaccuracies in the feature extraction can have a significant effect on a robot’s subsequent performance. In this paper, we study whether a behaviour model can be meaningfully represented using an end-to-end approach, where multimodal input, concretely visual data and activity information, is directly processed by a neural network. This paper concretely analyses the different building blocks of such a model, such that the aim is to identify a suitable architecture that can meaningfully combine the different modalities for guiding a robot’s behaviour. We conduct the analysis in the context of a sequence learning game, such that we compare different vision-only models that are then combined with an activity processing network into a joint multimodal model. The results of our evaluation on a dedicated dataset from the sequence learning game demonstrate that a multimodal end-to-end behaviour model has potential for assistive robotics — we report an F1 score of around 0.88 across different dataset-based test scenarios — but the real-life transferability strongly depends on whether the data is diverse enough for capturing meaningful variations in real-world scenarios, such as users being at different distances from a robot.

Accurate and robust localization systems are often highly desired in autonomous mobile robots. Existing LiDAR-based localization systems generally use standard particle filters which suffer from the well-known particle degeneracy problem. Furthermore, standard particle filters are ill-suited for handling discrepancies between maps and the actual operating environments. In this work, we present an effective LiDAR-based indoor localization system which addresses these two issues. The particle degeneracy problem is tackled with an efficient implementation of an optimal particle filter. Map discrepancies are then handled with the use of a high-fidelity observation model for accurate particle propagation and a separate low-fidelity observation model for robust weight update. Evaluations were carried out against a standard particle filter baseline on both real-world and simulated data from challenging indoor environments. The proposed system was found to show significantly better performance in-terms of accuracy, robustness to ambiguity, and robustness to map discrepancies. These performance gains were observed even with more than ten times smaller particle set sizes than in the baseline, while the increase in the computation time per particle was only around 20%.

In therapeutic scenarios, robots are sometimes used for imitation activities in which the robot demonstrates a motion and the individual under therapy needs to repeat it. To allow incorporating new types of motions in such activities, the robot should have an ability to learn motions by observing demonstrations from a human, such as a therapist. In this paper, we investigate an approach for acquiring motions from skeleton observations of a human, which are collected by a robot-centric RGB-D camera. The learning process from human body gestures to robot movements is done by mapping the joint angle positions to the robot’s body, such that self-collisions of the end effector are prevented by re-estimating the angles in a safe angular position. We performed both a quantitative and a qualitative evaluation of the method, namely we (i) quantitatively evaluated the motion reproduction error of the procedure by performing a study with QTrobot in which the robot acquired different upper-body dance moves from multiple participants, and (ii) performed a qualitative user study to evaluate the robot’s perceived reproduction. The quantitative evaluation demonstrates the method’s overall feasibility, although the reproduction quality is affected by noise in the skeleton observations, while the qualitative evaluation suggests generally high satisfaction with the robot’s motion, except for motions that are likely to lead to self-collisions and which were reproduced less accurately.

State-of-the-art (SOTA) object detectors are generally evaluated on object detection challenge datasets. However, automotive domain-specific quantitative analysis of detectors is limited and often incomplete. Moreover, evaluation of detectors is mainly focused on average precision (AP) metric, ignoring AP’s limitation such as insensitive to shape of the precision-recall curve. These issues are addressed in this paper by evaluating the most popular SOTA detectors on autonomous driving and autonomous parking datasets. AP weakness is addressed by evaluating detectors using the Localization-Recall-Precision (LRP) metric, which also provides a detailed understanding of a detector. Motivated from LRP, this paper also presents the optimal AP (oAP) metric for the fair comparison of detectors, which was ignored until now. The proposed oAP metric can be easily used with any object detector. In addition, a complete object detection pipeline starting from data collection to deployment on an embedded device is demonstrated in this work. Experimental results are presented graphically to analyze detector from a different perspective.

Different sensors on a robot help in understanding different aspects of the environment they are working in; however, each sensor modality is often processed individually and information from other sensors is not utilized jointly. One of the reasons is different sampling rates and different dimensions of input modalities. In this paper, we use multimodal data fusion techniques such as early, late and intermediate fusion for grasp failure identification using four different 3D convolution-based multimodal neural networks (3D-MNN). Our results on a visual-tactile dataset shows that the performance of the classification task is improved while using multimodal data. In addition, a neural network trained with 30:22 train-test split of multimodal data achieved accuracy comparable to a network trained with 78:22 train-test split of unimodal data1.

MOTIVATION: The majority of biomedical knowledge is stored in structured databases or as unstructured text in scientific publications. This vast amount of information has led to numerous machine learning-based biological applications using either text through natural language processing (NLP) or structured data through knowledge graph embedding models. However, representations based on a single modality are inherently limited. RESULTS: To generate better representations of biological knowledge, we propose STonKGs, a Sophisticated Transformer trained on biomedical text and Knowledge Graphs (KGs). This multimodal Transformer uses combined input sequences of structured information from KGs and unstructured text data from biomedical literature to learn joint representations in a shared embedding space. First, we pre-trained STonKGs on a knowledge base assembled by the Integrated Network and Dynamical Reasoning Assembler consisting of millions of text-triple pairs extracted from biomedical literature by multiple NLP systems. Then, we benchmarked STonKGs against three baseline models trained on either one of the modalities (i.e. text or KG) across eight different classification tasks, each corresponding to a different biological application. Our results demonstrate that STonKGs outperforms both baselines, especially on the more challenging tasks with respect to the number of classes, improving upon the F1-score of the best baseline by up to 0.084 (i.e. from 0.881 to 0.965). Finally, our pre-trained model as well as the model architecture can be adapted to various other transfer learning applications. AVAILABILITY AND IMPLEMENTATION: We make the source code and the Python package of STonKGs available at GitHub (https://github.com/stonkgs/stonkgs) and PyPI (https://pypi.org/project/stonkgs/). The pre-trained STonKGs models and the task-specific classification models are respectively available at https://huggingface.co/stonkgs/stonkgs-150k and https://zenodo.org/communities/stonkgs. SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

When faced with an execution failure, an intelligent robot should be able to identify the likely reasons for the failure and adapt its execution policy accordingly. This paper addresses the question of how to utilise knowledge about the execution process, expressed in terms of learned constraints, in order to direct the diagnosis and experience acquisition process. In particular, we present two methods for creating a synergy between failure diagnosis and execution model learning. We first propose a method for diagnosing execution failures of parameterised action execution models, which searches for action parameters that violate a learned precondition model. We then develop a strategy that uses the results of the diagnosis process for generating synthetic data that are more likely to lead to successful execution, thereby increasing the set of available experiences to learn from. The diagnosis and experience correction methods are evaluated for the problem of handle grasping, such that we experimentally demonstrate the effectiveness of the diagnosis algorithm and show that corrected failed experiences can contribute towards improving the execution success of a robot.

When an autonomous robot learns how to execute actions, it is of interest to know if and when the execution policy can be generalised to variations of the learning scenarios. This can inform the robot about the necessity of additional learning, as using incomplete or unsuitable policies can lead to execution failures. Generalisation is particularly relevant when a robot has to deal with a large variety of objects and in different contexts. In this paper, we propose and analyse a strategy for generalising parameterised execution models of manipulation actions over different objects based on an object ontology. In particular, a robot transfers a known execution model to objects of related classes according to the ontology, but only if there is no other evidence that the model may be unsuitable. This allows using ontological knowledge as prior information that is then refined by the robot’s own experiences. We verify our algorithm for two actions - grasping and stowing everyday objects - such that we show that the robot can deduce cases in which an existing policy can generalise to other objects and when additional execution knowledge has to be acquired.

Execution monitoring is essential for robots to detect and respond to failures. Since it is impossible to enumerate all failures for a given task, we learn from successful executions of the task to detect visual anomalies during runtime. Our method learns to predict the motions that occur during the nominal execution of a task, including camera and robot body motion. A probabilistic U-Net architecture is used to learn to predict optical flow, and the robot’s kinematics and 3D model are used to model camera and body motion. The errors between the observed and predicted motion are used to calculate an anomaly score. We evaluate our method on a dataset of a robot placing a book on a shelf, which includes anomalies such as falling books, camera occlusions, and robot disturbances. We find that modeling camera and body motion, in addition to the learning-based optical flow prediction, results in an improvement of the area under the receiver operating characteristic curve from 0.752 to 0.804, and the area under the precision-recall curve from 0.467 to 0.549.

An essential measure of autonomy in assistive service robots is adaptivity to the various contexts of human-oriented tasks, which are subject to subtle variations in task parameters that determine optimal behaviour. In this work, we propose an apprenticeship learning approach to achieving context-aware action generalization on the task of robot-to-human object hand-over. The procedure combines learning from demonstration and reinforcement learning: a robot first imitates a demonstrator's execution of the task and then learns contextualized variants of the demonstrated action through experience. We use dynamic movement primitives as compact motion representations, and a model-based C-REPS algorithm for learning policies that can specify hand-over position, conditioned on context variables. Policies are learned using simulated task executions, before transferring them to the robot and evaluating emergent behaviours. We additionally conduct a user study involving participants assuming different postures and receiving an object from a robot, which executes hand-overs by either imitating a demonstrated motion, or adapting its motion to hand-over positions suggested by the learned policy. The results confirm the hypothesized improvements in the robot's perceived behaviour when it is context-aware and adaptive, and provide useful insights that can inform future developments.

For robots acting in human-centered environments, the ability to improve based on experience is essential for reliable and adaptive operation; however, particularly in the context of robot failure analysis, experience-based improvement is practically useful only if robots are also able to reason about and explain the decisions they make during execution. In this paper, we describe and analyse a representation of execution-specific knowledge that combines (i) a relational model in the form of qualitative attributes that describe the conditions under which actions can be executed successfully and (ii) a continuous model in the form of a Gaussian process that can be used for generating parameters for action execution, but also for evaluating the expected execution success given a particular action parameterisation. The proposed representation is based on prior, modelled knowledge about actions and is combined with a learning process that is supervised by a teacher. We analyse the benefits of this representation in the context of two actions - grasping handles and pulling an object on a table -such that the experiments demonstrate that the joint relational-continuous model allows a robot to improve its execution based on experience, while reducing the severity of failures experienced during execution.

While executing actions, service robots may experience external faults because of insufficient knowledge about the actions' preconditions. The possibility of encountering such faults can be minimised if symbolic and geometric precondition models are combined into a representation that specifies how and where actions should be executed. This work investigates the problem of learning such action execution models and the manner in which those models can be generalised. In particular, we develop a template-based representation of execution models, which we call δ models, and describe how symbolic template representations and geometric success probability distributions can be combined for generalising the templates beyond the problem instances on which they are created. Our experimental analysis, which is performed with two physical robot platforms, shows that δ models can describe execution-specific knowledge reliably, thus serving as a viable model for avoiding the occurrence of external faults.

Position awareness in unknown and large indoor spaces represents a great advantage for people, everyday pedestrians have to search for specific places, products and services. In this work a positioning solution able to localize the user based on data measured with a mobile device is described and evaluated. The position estimate uses data from smartphone built-in sensors, WiFi (Wireless Fidelity) adapter and map information of the indoor environment (e.g. walls and obstacles). A probability map derived from statistical information of the users tracked location over a period of time in the test scenario is generated and embedded in a map graph, in order to correct and combine the position estimates under a Bayesian representation. PDR (Pedestrian Dead Reckoning), beacon-based Weighted Centroid position estimates, map information obtained from building OpenStreetMap XML representation and probability map users path density are combined using a Particle Filter and implemented in a smartphone application. Based on evaluations, this work verifies that the use of smartphone hardware components, map data and its semantic information represented in the form of a OpenStreetMap structure provide 2.48 meters average error after 1,700 travelled meters and a scalable indoor positioning solution. The Particle Filter algorithm used to combine various sources of information, its radio WiFi-based observation, probability particle weighting process and the mapping approach allowing the inclusion of new indoor environments knowledge show a promising approach for an extensible indoor navigation system.

The design and development of digital electronic systems is mainly performed by use of a hardware description language. To prepare students in electrical engineering for a career in hardware design many universities provide courses on VHDL. The traditional approach in teaching VHDL is mainly by means of textbook examples and simulation provided by software applications. These exercises are perceived as monotonous by the students and do not or only very slightly correspond with actual real-life applications based on FPGAs. Moreover, most real-life applications are too expensive to be equipped in student laboratories. To bridge the gap between a simulation-only environment and affordable real-life applications students should be provided access to remote real-life setups with a 24/7 availability and preferably shared between multiple institutes. The eDiViDe platform (European Digital Virtual Design Lab, http://www.edivide.eu), see Fig. 1, provides students with this unlimited and exciting access to FPGA based setups. Instead of theory-only courses and a quick basic lab, they can work their way through digital design courses testing their skills on real-life setups to trigger their interest. The platform hosts multiple FPGA setups at different European institutes. These setups are accessible through a web-based interface with video feedback. VHDL development is performed offline, given an entity and specific setup information. All further steps of the FPGA toolchain are performed on the platform. A reservation system takes care of the FPGA programming and student interaction with the setups. Similar initiatives provide stable solutions with educational support [1,2,3]. The eDiViDe platform differentiates with a distributed platform across several institutes and with the support for advanced setups. It is the result of a joint effort and easily expandable with additional setups at any location. At this moment following setups are available: greenhouse, stepper motor control, sea noise emulator, state machine workshop, Geffe generator, pong / game of life, traffic light control, MIPS CPU. This set will be extended with more advanced setups that include e.g. a partial reconfiguration workshop for audio/video filters, a side-channel analysis setup and a mars rover playfield. Besides promoting digital design education, the eDiViDe platform creates a channel to make the research activities in the contributing universities more visible. Industry could also benefit from this platform to promote their brand and products to soon to be engineers.

A person has to deal with large and unknown scenarios, for example a client searching for a expositor in a trade fair or a passenger looking for a gate in an airport. Due to the fact that position awareness represents a great advantage for people, a navigation system implemented for a commercial smartphone can help the user to save time and money. In this work a navigation example application able to localize and provide directions to a desired destination in an indoor environment is presented and evaluated. The position of the user is calculated with information from the smartphone builtin sensors, WiFi adapter and floor-plan layout of the indoor environment. A commercial smartphone is used as the platform to implement the example application, due to it's hardware features, computational power and the graphic user interface available for the users. Evaluations verified that room accuracy is achieved for robust localization by using the proposed technologies and algorithms. The used optimal sensor fusion filter for different sources of information and the easy to deploy infrastructure in a new environment show promise for mobile indoor navigation systems.

Swedish wheeled mobile robots have remarkable mobility properties allowing them to rotate and translate at the same time. Being holonomic systems, their kinematics model results in the possibility of designing separate and independent position and heading trajectory tracking control laws. Nevertheless, if these control laws should be implemented in the presence of unaccounted actuator peak velocity limits, the resulting saturated linear and angular velocity commands could interfere with each other thus dramatically affecting the overall expected performance. Based on Lyapunov's direct method, a position and heading trajectory tracking control law for Swedish wheeled robots is developed. It explicitly accounts for actuator velocity saturation by using ideas from a prioritized task based control framework.

This paper applies a new kind of recurrent neu ral networks (RNN) called Echo State Networks (ESN) [4] to the classical problem of motor speed control for a differential drive robot. ESNs can be trained orders of magnitude faster than other RNNs and previous simulation-based investigations showed promising results when ESNs where used as black box models in the domain of system identification or as a low-level plant controllers [11]. This paper validates for the first time the predicted superior simulation results by physical experiments. In order to compare the quality of an ESN controller fairly to the original PID based controller a complete test work flow is established. It consists of an embedded implementation of the ESN motor-controller, a trace facility, a procedure to train a new ESN motor-controller according to these traced data and a user interface to define and control test-drives of the robot. The results prove that the ESN controller shows a slightly better control quality as a PID controller with respect to various important error norms from control theory. For the experiments we used a RoboCup robot and for this special application scenario the ESN controller actually outperforms the PID.

The use of three types of spatio-temporal processing elements is investigated for optical sensory preprocessing in order to solve robot control problems in mobile robotics. The sensory elements are optical analog VLSI silicon retina type devices that do on-chip gradient operations and perform a current mode hysteretic winner-take-all function. Each sensor device extracts a characteristic feature from the optical input: position of highest contrast along a 1-D array, maximum speed along a 1-D array, maximum optical flow on a 2-D array. These are continuously calculated by the respective sensory devices. The sensory devices are applied in a mobile robotics application. They are used for active ball control, ball velocity prediction and active gaze-control for RoboCup Middle-Size League robots.