Dietary assessment is commonly used to monitor the calorie intake of people who wish to regulate their diet. Traditional dietary assessments require participants to actively record their meals for 24 hours, followed by a manual assessment of calorie intake by a health professional. This approach is not effective for use on a daily basis, and thus cannot give a comprehensive analysis of the participant’s diet plan. New dietary assessment techniques are needed to make food monitoring easier and more feasible in a real-world environment. Many research works use image-assisted or image-based techniques to simplify the process and make it more acceptable to the participants. Yet, these works generally focus on analyzing the main course of a meal, and the meal is usually set up in a laboratory setting with a simple background. While finger food plays an important role in diet, as it is easy to lose track of the amount consumed, it is a challenge for dietary assessment. As a result, estimating the total calorie intake of the participant is difficult. In this paper, we present a real-time food monitoring technique that uses computer vision techniques to passively detect when finger food is being grabbed by participants during a live stream, to classify the amount of potato chips grabbed and give immediate feedback to the participants. Two convolutional neural networks (CNN) are used for this purpose: one for detecting the participants’ hands and the other for classifying the chips according to calorie labels. In order to train the CNN to estimate calorie intake, we collected a dataset with more than 26,600 images of volunteers grabbing differing quantities of chips. The macro-average F1 score of our models ranges from 0.75 to 0.88 for a 3-class system, and it ranges from 0.59 to 0.74 for a 4-class system.

This paper proposes a neural network-based user simulator that can provide a multimodal interactive environment for training Reinforcement Learning (RL) agents in collaborative tasks involving multiple modes of communication. The simulator is trained on the existing ELDERLY-AT-HOME corpus and accommodates multiple modalities such as language, pointing gestures, and haptic-ostensive actions. The paper also presents a novel multimodal data augmentation approach, which addresses the challenge of using a limited dataset due to the expensive and time-consuming nature of collecting human demonstrations. Overall, the study highlights the potential for using RL and multimodal user simulators in developing and improving domestic assistive robots.

Human-human collaboration is characterized by a back-and-forth, where an action of one agent elicits the response of the other. This interaction is inherently multimodal and includes both high-level modalities such as language and low-level ones such as force exchanges. In this work, we investigate human collaborative manipulation: we show distinct patterns that can be identified in low-level physical data and that can be interpreted as primitives used by humans to negotiate about various aspects of the motion and to execute the motion. These primitives provide a high-level interpretation of the interaction and can be used to connect low-level behavior to language. We describe the human study used to collect the data, the data analysis process, and discuss how the identified primitives could be used by a robot’s interaction manager to mediate physical Human-Robot Interaction (pHRI).

In a collaborative task and the interaction that accompanies it, the participants often take on distinct roles, and dynamically switch the roles as the task requires. A domestic assistive robot thus needs to have similar capabilities. Using our previously proposed Multimodal Interaction Manager (MIM) framework, this paper investigates how role switching for a robot can be implemented. It identifies a set of primitive subtasks that encode common interaction patterns observed in our data corpus and that can be used to easily construct complex task models. It also describes an implementation on the NAO robot that, together with our original work, demonstrates that the robot can take on different roles. We provide a detailed analysis of the performance of the system and discuss the challenges that arise when switching roles in human-robot interactions.

With rapid advances in social robotics, humanoids and autonomy, robot assistants appear to be within reach. However, robots are still unable to effectively interact with humans in activities of daily living. One of the challenges is in the frequent use of multiple communication modalities when humans engage in collaborative activities. In this paper, we propose a Multimodal Interaction Manager, a framework for an assistive robot that maintains an active multimodal interaction with a human partner while performing physical collaborative tasks. The heart of our framework is a Hierarchical Bipartite Action-Transition Network (HBATN), which allows the robot to infer the state of the task and the dialogue given spoken utterances and observed pointing gestures from a human partner, and to plan its next actions. Finally, we implemented this framework on a robot to provide preliminary evidence that the robot can successfully participate in a task-oriented multimodal interaction.

Object handover is a common physical interaction between humans. It is thus also of significant interest for human-robot interaction. In this paper, we are focused on robot-to-human object handover. The main challenge in this case is how to reduce the failure rate, i.e., to ensure that the object does not fall (object safety), while at the same time allowing the human to easily acquire the object (smoothness). To endow the robot with a failure recovery mechanism, we investigated how humans detect failure during the transfer phase of the handover. We conducted a human study that showed that a human giver primarily relies on vision rather than haptic sensing to detect the fall of the object. Motivated by this study, a robotic handover system is proposed that consists of a motion sensor attached to the robot's gripper, a force sensor at the base of the gripper, and a controller that is capable of regrasping the object if it starts falling. The proposed system is implemented on a Baxter robot and is shown to achieve a smooth and safe handover.

In robot to human object handover, the configuration (position and orientation) in which the object is transferred should be selected so that the handover is safe and comfortable for the human. The trajectory on which the robot moves the object to the point of transfer should be also selected so that the robot intention is clear and the handover feels natural to the human. In this paper, we propose to select the configuration for the transfer and the trajectory to reach this configuration based on what humans do in human-human handovers. We describe a human study designed to investigate the human-human handover and propose an ergonomic model that can predict object transfer position observed in the study. A human-robot experiment is then conducted that shows that the proposed model generates transfer positions that match the preferred height and distance relative to the human.

Humans rely on a multitude of senses to achieve a safe handover. The challenge for a robot-human handover is to both equip the robot with the appropriate sensors as well as devise handover controllers that effectively use them to prevent failures. In this paper we use object acceleration as an indicator of an impending handover failure and propose a handover controller that has re-grasping mechanism to prevent falling of the object in case of an imperfect handover. The work is motivated by our observation that humans primarily rely on vision to prevent handover failure. We also propose a novel acceleration sensing setup that can be integrated into a robot gripper. The handover controller is implemented on Baxter Research robot equipped with the proposed sensor and is shown to be effective in preventing object fall.

Human grasp has been studied extensively and many taxonomies have been developed to classify different grasp types. In this work, we collect a comprehensive list of grasp types and investigate the force distribution patterns among them. We conduct a human study and collect force data in various grasp types. Data collection is performed by utilizing a data glove that has seventeen force sensors. We analyze the measured forces and identify different patterns in force distribution. Employing voting technique over various clustering methods, we propose a robust clustering for the force patterns, namely the grasp taxonomy in force domain. The proposed grasp taxonomy can be exploited in designing robotic prosthetic hands, designing grasp controllers and action recognition in human-robot interaction.