As artificial intelligence techniques become more sophisticated, we anticipate that robots collaborating with humans will develop their own intentions, leading to potential conflicts in interaction. This development calls for advanced conflict resolution strategies in physical human–robot interaction (pHRI), a key focus of our research. We use a machine learning (ML) classifier to detect conflicts during co-manipulation tasks to adapt the robot’s behavior accordingly using an admittance controller. In our approach, we focus on two groups of interactions, namely “harmonious” and “conflicting,” corresponding respectively to the cases of the human and the robot working in harmony to transport an object when they aim for the same target, and human and robot are in conflict when human changes the manipulation plan, e.g. due to a change in the direction of movement or parking location of the object. Co-manipulation scenarios were designed to investigate the efficacy of the proposed ML approach, involving 20 participants. Task performance achieved by the ML approach was compared against three alternative approaches: (a) a rule-based (RB) Approach, where interaction behaviors were rule-derived from statistical distributions of haptic features; (b) an unyielding robot that is proactive during harmonious interactions but does not resolve conflicts otherwise, and (c) a passive robot which always follows the human partner. This mode of cooperation is known as “hand guidance” in pHRI literature and is frequently used in industrial settings for so-called “teaching” a trajectory to a collaborative robot. The results show that the proposed ML approach is superior to the others in task performance. However, a detailed questionnaire administered after the experiments, which contains several metrics, covering a spectrum of dimensions to measure the subjective opinion of the participants, reveals that the most preferred mode of interaction with the robot is surprisingly passive. This preference indicates a strong inclination toward an interaction mode that gives more control to humans and offers less demanding interaction, even if it is not the most efficient in task performance. Hence, there is a clear trade-off between task performance and the preferred mode of interaction of humans with a robot, and a well-balanced approach is necessary for designing effective pHRI systems in the future.

This paper proposes an adaptive admittance controller for improving efficiency and safety in physical human-robot interaction (pHRI) tasks in small-batch manufacturing that involve contact with stiff environments, such as drilling, polishing, cutting, etc. We aim to minimize human effort and task completion time while maximizing precision and stability during the contact of the machine tool attached to the robot’s end-effector with the workpiece. To this end, a two-layered learning-based human intention recognition mechanism is proposed, utilizing only the kinematic and kinetic data from the robot and two force sensors. A "subtask detector" recognizes the human intent by estimating which phase of the task is being performed, e.g., Idle, Tool-Attachment, Driving, and Contact. Simultaneously, a "motion estimator" continuously quantifies intent more precisely during the Driving to predict when Contact will begin. The controller is adapted online according to the subtask while allowing early adaptation before the Contact to maximize precision and safety and prevent potential instabilities. Three sets of pHRI experiments were performed with multiple subjects under various conditions. Spring compression experiments were performed in virtual environments to train the data-driven models and validate the proposed adaptive system, and drilling experiments were performed in the physical world to test the proposed methods’ efficacy in real-life scenarios. Experimental results show subtask classification accuracy of 84% and motion estimation R2score of 0.96. Furthermore, 57% lower human effort was achieved during Driving as well as 53% lower oscillation amplitude at Contact as a result of the proposed system.

This paper proposes a machine learning (ML) approach to detect and resolve motion conflicts that occur between a human and a proactive robot during the execution of a physically collaborative task. We train a random forest classifier to distinguish between harmonious and conflicting human-robot interaction behaviors during object co-manipulation. Kinesthetic information generated through the teamwork is used to describe the interactive quality of collaboration. As such, we demonstrate that features derived from haptic (force/torque) data are sufficient to classify if the human and the robot harmoniously manipulate the object or they face a conflict. A conflict resolution strategy is implemented to get the robotic partner to proactively contribute to the task via online trajectory planning whenever interactive motion patterns are harmonious, and to follow the human lead when a conflict is detected. An admittance controller regulates the physical interaction between the human and the robot during the task. This enables the robot to follow the human passively when there is a conflict. An artificial potential field is used to proactively control the robot motion when partners work in harmony. An experimental study is designed to create scenarios involving harmonious and conflicting interactions during collaborative manipulation of an object, and to create a dataset to train and test the random forest classifier. The results of the study show that ML can successfully detect conflicts and the proposed conflict resolution mechanism reduces human force and effort significantly compared to the case of a passive robot that always follows the human partner and a proactive robot that cannot resolve conflicts.

Drilling a hole on a curved surface with a desired angle is prone to failure when done manually, due to the difficulties in drill alignment and also inherent instabilities of the task, potentially causing injury and fatigue to the workers. On the other hand, it can be impractical to fully automate such a task in real manufacturing environments because the parts arriving at an assembly line can have various complex shapes where drill point locations are not easily accessible, making automated path planning difficult. In this work, an adaptive admittance controller with 6 degrees of freedom is developed and deployed on a KUKA LBR iiwa 7 cobot such that the operator is able to manipulate a drill mounted on the robot with one hand comfortably and open holes on a curved surface with haptic guidance of the cobot and visual guidance provided through an AR interface. Real-time adaptation of the admittance damping provides more transparency when driving the robot in free space while ensuring stability during drilling. After the user brings the drill sufficiently close to the drill target and roughly aligns to the desired drilling angle, the haptic guidance module fine tunes the alignment first and then constrains the user movement to the drilling axis only, after which the operator simply pushes the drill into the workpiece with minimal effort. Two sets of experiments were conducted to investigate the potential benefits of the haptic guidance module quantitatively (Experiment I) and also the practical value of the proposed pHRI system for real manufacturing settings based on the subjective opinion of the participants (Experiment II). The results of Experiment I, conducted with 3 naive participants, show that the haptic guidance improves task completion time by 26% while decreasing human effort by 16% and muscle activation levels by 27% compared to no haptic guidance condition. The results of Experiment II, conducted with 3 experienced industrial workers, show that the proposed system is perceived to be easy to use, safe, and helpful in carrying out the drilling task.

In today's automation driven manufacturing environments, emerging technologies like cobots (collaborative robots) and augmented reality interfaces can help integrating humans into the production workflow to benefit from their adaptability and cognitive skills. In such settings, humans are expected to work with robots side by side and physically interact with them. However, the trade-off between stability and transparency is a core challenge in the presence of physical human robot interaction (pHRI). While stability is of utmost importance for safety, transparency is required for fully exploiting the precision and ability of robots in handling labor intensive tasks. In this work, we propose a new variable admittance controller based on fractional order control to handle this trade-off more effectively. We compared the performance of fractional order variable admittance controller with a classical admittance controller with fixed parameters as a baseline and an integer order variable admittance controller during a realistic drilling task. Our comparisons indicate that the proposed controller led to a more transparent interaction compared to the other controllers without sacrificing the stability. We also demonstrate a use case for an augmented reality (AR) headset which can augment human sensory capabilities for reaching a certain drilling depth otherwise not possible without changing the role of the robot as the decision maker.

With the recent advances in cobot (collaborative robot) technology, we can now work with a robot side by side in manufacturing environments. The collaboration between human and cobot can be enhanced by detecting the intentions of human to make the production more flexible and effective in future factories. In this regard, interpreting human intention and then adjusting the controller of cobot accordingly to assist human is a core challenge in physical human-robot interaction (pHRI). In this study, we propose a classifier based on Artificial Neural Networks (ANN) that predicts intended direction of human movement by utilizing electromyography (EMG) signals acquired from human arm muscles. We employ this classifier in an admittance control architecture to constrain human arm motion to the intended direction and prevent undesired movements along other directions. The proposed classifier and the control architecture have been validated through a path following task by utilizing a KUKA LBR iiwa 7 R800 cobot. The results of our experimental study with 6 participants show that the proposed architecture provides an effective assistance to human during the execution of task and reduces undesired motion errors, while not sacrificing from the task completion time.

Physical human-robot interaction (pHRI) integrates the benefits of human operator and a collaborative robot in tasks involving physical interaction, with the aim of increasing the task performance. However, the design of interaction controllers that achieve safe and transparent operations is challenging, mainly due to the contradicting nature of these objectives. Knowing that attaining perfect transparency is practically unachievable, controllers that allow better compromise between these objectives are desirable. In this article, we propose a multicriteria optimization framework, which jointly optimizes the stability robustness and transparency of a closed-loop pHRI system for a given interaction controller. In particular, we propose a Pareto optimization framework that allows the designer to make informed decisions by thoroughly studying the tradeoff between stability robustness and transparency. The proposed framework involves a search over the discretized controller parameter space to compute the Pareto front curve and a selection of controller parameters that yield maximum attainable transparency and stability robustness by studying this tradeoff curve. The proposed framework not only leads to the design of an optimal controller, but also enables a fair comparison among different interaction controllers. In order to demonstrate the practical use of the proposed approach, integer and fractional order admittance controllers are studied as a case study and compared both analytically and experimentally. The experimental results validate the proposed design framework and show that the achievable transparency under fractional order admittance controller is higher than that of integer order one, when both controllers are designed to ensure the same level of stability robustness.

We present HapTable; a multi-modal interactive tabletop that allows users to interact with digital images and objects through natural touch gestures, and receive visual and haptic feedback accordingly. In our system, hand pose is registered by an infrared camera and hand gestures are classified using a Support Vector Machine (SVM) classifier. To display a rich set of haptic effects for both static and dynamic gestures, we integrated electromechanical and electrostatic actuation techniques effectively on tabletop surface of HapTable, which is a surface capacitive touch screen. We attached four piezo patches to the edges of tabletop to display vibrotactile feedback for static gestures. For this purpose, the vibration response of the tabletop, in the form of frequency response functions (FRFs), was obtained by a laser Doppler vibrometer for 84 grid points on its surface. Using these FRFs, it is possible to display localized vibrotactile feedback on the surface for static gestures. For dynamic gestures, we utilize the electrostatic actuation technique to modulate the frictional forces between finger skin and tabletop surface by applying voltage to its conductive layer. To our knowledge, this hybrid haptic technology is one of a kind and has not been implemented or tested on a tabletop. It opens up new avenues for gesture-based haptic interaction not only on tabletop surfaces but also on touch surfaces used in mobile devices with potential applications in data visualization, user interfaces, games, entertainment, and education. Here, we present two examples of such applications, one for static and one for dynamic gestures, along with detailed user studies. In the first one, user detects the direction of a virtual flow, such as that of wind or water, by putting their hand on the tabletop surface and feeling a vibrotactile stimulus traveling underneath it. In the second example, user rotates a virtual knob on the tabletop surface to select an item from a menu while feeling the knob's detents and resistance to rotation in the form of frictional haptic feedback.

In the near future, humans and robots are expected to perform collaborative tasks involving physical interaction in various different environments such as homes, hospitals, and factories. One important research topic in physical Human-Robot Interaction (pHRI) is to develop tacit and natural haptic communication between the partners. Although there are already several studies in the area of Human-Robot Interaction, the number of studies investigating the physical interaction between the partners and in particular the haptic communication are limited and the interaction in such systems is still artificial when compared to natural human-human collaboration. Although the tasks involving physical interaction such as the table transportation can be planned and executed naturally and intuitively by two humans, there are unfortunately no robots in the market that can collaborate and perform the same tasks with us. In this study, we propose a new controller for the robotic partner that is designed to a) detect the intentions of the human partner through haptic channel using a fuzzy controller b) adjust its contribution to the task via a variable impedance controller and c) resolve the conflicts during the task execution by controlling the internal forces. The results of the simulations performed in Simulink/Matlab show that the proposed controller is superior to the stand-alone standard/variable impedance controllers.

In this paper, we introduce a novel stylus capable of displaying two haptic effects to the user. The first effect is a tactile flow effect up and down along the pen, and the other is a rotation effect about the long axis of the pen. The flow effect is based on the haptic illusion of “apparent tactile motion”, while the rotation effect comes from the reaction torque created by an electric motor placed along the stylus shaft. The stylus is embedded with two vibration actuators at the ends, and a DC motor with a rotating balanced mass in the middle. We show that, it is possible to create flow and rotation effects on the stylus by driving the actuators on the stylus. Furthermore, we show that the timing and the actuation patterns of the vibration actuators and DC motor on the stylus significantly affect the discernibility of the synthesized perceptions; hence these parameters should be selected carefully. Two psychophysical experiments, each performed with 10 subjects, shed light on the discernability of the two haptic effects as a function of various actuation parameters. Our results show that, with carefully selected parameters, the subjects can successfully identify the flow of motion and the direction of rotation with high accuracies.

Use of touch screens in cell phones, touch pads, tablet PS, and information kiosks are getting increasingly widespread. However, the touch screens available in the market today provide the user with limited haptic feedback, resulting in a less satisfying user experience. This workshop aims to bring academicians and scientists from industry together to discuss how haptics can be integrated into touch screens.

We show that vibrotactile feedback displayed through the steering wheel of a car can reduce the perceptual and cognitive load of the driver, leading to less distraction and fewer navigation errors. To demonstrate the concept, two vibration motors are mounted onto the steering wheel of a driving simulator and driving experiments are performed in virtual environments under two different sensory conditions (auditory alone and auditory and vibrotactile feedback together). The results of our experiments with 12 subjects show that, if passenger auditory noise and distraction exist in the environment, the navigation errors (making a wrong turn or taking a wrong exit) are reduced when vibrotactile feedback is displayed to the users in tandem with the GPS-based voice commands.

Haptics has been used as a natural way for humans to communicate with computers in collaborative virtual environments. Human-computer collaboration is typically achieved by sharing control of the task between a human and a computer operator. An important research challenge in the field addresses the need to realize intention recognition and response, which involves a decision making process between the partners. In an earlier study, we implemented a dynamic role exchange mechanism, which realizes decision making by means of trading the parties' control levels on the task. This mechanism proved to show promise of a more intuitive and comfortable communication. Here, we extend our earlier work to further investigate the utility of a role exchange mechanism in dynamic collaboration tasks. An experiment with 30 participants was conducted to compare the utility of a role exchange mechanism with that of a shared control scheme where the human and the computer share control equally at all times. A no guidance condition is considered as a base case to present the benefits of these two guidance schemes more clearly. Our experiment show that the role exchange scheme maximizes the efficiency of the user, which is the ratio of the work done by the user within the task to the energy spent by her. Furthermore, we explored the added benefits of explicitly displaying the control state by embedding visual and vibrotactile sensory cues on top of the role exchange scheme. We observed that such cues decrease performance slightly, probably because they introduce an extra cognitive load, yet they improve the users' sense of collaboration and interaction with the computer. These cues also create a stronger sense of trust for the user towards her partner's control over the task.

New 3D video representations enable new modalities of interaction, such as haptic interaction, with 2D and 3D video for truly immersive media applications. Haptic interaction with video includes haptic structure and haptic motion for new immersive experiences. It is possible to compute haptic structure signals from 3D scene geometry or depth information. This paper introduces the concept of haptic motion, as well as new methods to compute haptic structure and motion signals for 2D video-plus-depth representation. The resulting haptic signals can be rendered using a haptic cursor attached to a 2D or 3D video display.

Investigating virtual environments has become an increasingly interesting research topic for engineers, computer and cognitive scientists, and psychologists. Although there have been several recent studies focused on the development of multimodal virtual environments (VEs) to study human-machine interactions, less attention has been paid to human-human and human-machine interactions in shared virtual environments (SVEs), and to our knowledge, no attention paid at all to what extent the addition of haptic communication between people would contribute to the shared experience. We have developed a multimodal shared virtual environment and performed a set of experiments with human subjects to study the role of haptic feedback in collaborative tasks and whether haptic communication through force feedback can facilitate a sense of being and collaborating with a remote partner. The study concerns a scenario where two participants at remote sites must cooperate to perform a joint task in an SVE. The goals of the study are (1) to assess the impact of force feedback on task performance, (2) to better understand the role of haptic communication in human-human interactions, (3) to study the impact of touch on the subjective sense of collaborating with a human as reported by the participants based on what they could see and feel, and (4) to investigate if gender, personality, or emotional experiences of users can affect haptic communication in SVEs. The outcomes of this research can have a powerful impact on the development of next-generation human-computer interfaces and network protocols that integrate touch and force feedback technology into the internet, development of protocols and techniques for collaborative teleoperation such as hazardous material removal, space station.