As robot technology advances, collaboration between humans and robots will become more prevalent in industrial tasks. When humans run into issues in such scenarios, a likely future involves relying on artificial agents or robots for aid. This study identifies key aspects for the design of future user-assisting agents. We analyze quantitative and qualitative data from a user study examining the impact of on-demand assistance received from a remote human in a human-robot collaboration (HRC) assembly task. We study scenarios in which users require help and we assess their experiences in requesting and receiving assistance. Additionally, we investigate participants’ perceptions of future non-human assisting agents and whether assistance should be on-demand or unsolicited. Through a user study, we analyze the impact that such design decisions (human or artificial assistant, on-demand or unsolicited help) can have on elicited emotional responses, productivity, and preferences of humans engaged in HRC tasks.

The MASTER project introduces an open Extended Reality (XR) platform designed to enhance human-robot collaboration and train workers in robotics within manufacturing settings. It includes modules for creating safe workspaces, intuitive robot programming, and user-friendly human-robot interactions (HRI), including eye-tracking technologies. The development of the platform is supported by two open calls targeting technical SMEs and educational institutes to enhance and test its functionalities. By employing the learning-by-doing methodology and integrating effective teaching principles, the MASTER platform aims to provide a comprehensive learning environment, preparing students and professionals for the complexities of flexible and collaborative manufacturing settings.

In an industrial production context, when a robotic arm is assigned a new task, it must be re-programmed by specialized personnel with the necessary skills and knowledge. In addition, the robot remains offline while being re-programmed, negatively affecting productivity. Mixed reality opens up the opportunity for a production worker with no programming skills to teach a robot by interacting with the hologram of its digital twin, thus not interfering with the physical robot during re-programming. We describe the design and implementation of a mixed-reality interface to instruct trajectories for a robot end-effector by moving the hologram of its digital twin by hand. The interface supports the integrality of this interaction so that the user can configure the end effector location and orientation simultaneously. We also report a user study (n = 14) to characterize this interface in terms of usability, user experience, and estimated temporal efficiency that it can provide when programming robot trajectories.

Abstract It is difficult to estimate the boundaries of the hazard zones generated around autonomous machines and robots when navigating a space shared with them. We investigated the use of multimodal (auditory and/or visual) mixed-reality (MR) displays to warn users about invading such hazards zones and to help them return to safety. Two single-modality auditory and visual displays were designed, which were subjectively comparable as generic hazard displays. An experimental user study was then conducted to compare the designed single-modality displays as well as an audio-visual display that combined both. When the display included auditory information, users returned sooner to safety, although this had a small effect on performance when carrying out an independent navigation task. Additional nuanced possible differences are reported and discussed in relation to display design characteristics, as well as in relation to the limitations of the implementations that are possible with current MR head-mounted display devices.

People who have cognitive disabilities can achieve personal fulfilment and social integration when they access the job market. In the case of working on industrial assembly lines, they can perform at the highest standards when assembly sub-tasks have been adequately adapted. However, the arrival of new production paradigms for the factory of the future (e.g., Industry 4.0) present new challenges and opportunities for workers with cognitive disabilities who will be part of hybrid human-automation assembly cells. We propose that alternative task-sharing approaches with collaborative robots may be more appropriate in the presence of such disabilities, in contrast to practices proposed for non-disabled workers. We describe a representative scenario (assembly of electrical cabinets) around which we are developing our research on this topic.

This paper describes the development and evaluation of Undulating Covers (UnCovers), mobile interfaces that can change their surface texture to transmit information. The Pin Array UnCover incorporates sinusoidal ridges controlled by servomotors, which can change their amplitude and granularity. The Mylar UnCover is a more organic interface that exploits the buckling properties of Mylar, using muscle wires, to change the texture granularity. The prototype development process and evaluations show that very low frequency texture changes, using amplitude or granularity, can be distinguished with high levels of accuracy. Since small changes are perceptible, it is possible to incorporate such interfaces into mobile devices without drastically increasing their size or actuation requirements. Finally, ratings from participants indicate that UnCovers would be appropriate for attention-grabbing, or caring and supportive interpersonal messages.

This paper presents a semantic multimodal interaction approach between humans and industrial robots to enhance the dependability of the collaboration in real industrial settings. Although this generic approach can be applied in different industrial scenarios, this paper explains in detail how it is implemented in a real case to enhance the accuracy of requests interpretation in order to achieve a more efficient, easy to scale and maintain collaboration between humans and robots.

Physical representations of data have existed for thousands of years. Yet it is now that advances in digital fabrication, actuated tangible interfaces, and shape-changing displays are spurring an emerging area of research that we call Data Physicalization. It aims to help people explore, understand, and communicate data using computer-supported physical data representations. We call these representations physicalizations, analogously to visualizations -- their purely visual counterpart. In this article, we go beyond the focused research questions addressed so far by delineating the research area, synthesizing its open challenges and laying out a research agenda.

Device deformation allows new types of gestures to be used in interaction. We identify that the gesture/use-case pairings proposed by interaction designers are often driven by factors relating improved tangibility, spatial directionality and strong metaphorical bonds. With this starting point, we argue that some of the designs may not make use of the full potential of deformation gestures as continuous, bipolar input techniques. In two user studies, we revisited the basics of deformation input by taking a new systematic look at the question of matching gestures with use cases. We observed comparable levels of UX when using bend input in different continuous bipolar interactions, irrespective of the choice of tangibility, directionality and metaphor. We concluded that device bend gestures use their full potential when used to control continuous bipolar parameters, and when quick reactions are needed. From our studies, we also identify relative strengths of absolute and relative mappings, and report a Fitts' law study for device bending input.

Shape-retaining freely-deformable interfaces can take innumerable distinct shapes, and creating specific target configurations can be a challenge. In this paper, we investigate how audio can guide a user in this process, through the use of either musical or metaphoric sounds. In a formative user study, we found that sound encouraged action possibilities and made the affordances of the interface perceivable. We also found that adding audio as a modality along with vision and touch, made a positive contribution to guiding users' interactions with the interface.

We present a study that investigates the potential of combining, within the same interaction cycle, deformation and touch input in a handheld device. Using a flexible, input-only device connected to an external display, we compared a multitouch input technique and two hybrid deformation-plus-touch input techniques (bending and twisting the device, plus either front- or back-touch), in an image-docking task. We compared and analyzed the performance (completion time) and user experience (UX) obtained in each case, using multiple assessment metrics. We found that combining device deformation with front-touch produced the best UX. All the interaction techniques showed the same efficiency in task completion. This was a surprising finding, since multitouch (an integral input technique) was expected to be the most efficient technique in an image docking task (an interaction in an integral perceptual space). We discuss these findings in relation to self-reported qualitative data and observed interaction-procedure metrics. We found that the interaction procedures with the hybrid techniques were more sequential but also more paced. These findings suggest that the benefits of deformation input can still be observed when deformation and touch are combined in an input device.

While gesture taxonomies provide a classification of device-based gestures in terms of communicative intent, little work has addressed the usability differences in manually performing these gestures. In this primarily qualitative study, we investigate how two sets of iconic gestures that vary in familiarity, mimetic and alphabetic, are affected under varying failed recognition error rates (0-20%, 20-40%, 40-60%). Drawing on experiment logs, video observations, subjects' feedback, and a subjective workload assessment questionnaire, results revealed two main findings: a) mimetic gestures tend to evolve into diverse variations (within the activities they mimic) under high error rates, while alphabet gestures tend to become more rigid and structured and b) mimetic gestures were tolerated under recognition error rates of up to 40%, while alphabet gestures incur significant overall workload with up to only 20% error rates. Thus, while alphabet gestures are more robust to recognition errors in keeping their signature, mimetic gestures are more robust to recognition errors from a usability and user experience standpoint, and thus better suited for inclusion into mainstream device-based gesture interaction with mobile phones.

There has been little discussion on how the materials used to create deformable devices, and the subsequent interactions, might influence user performance and preference. In this paper we evaluated how the stiffness and required deformation extent (bending up and down bimanually) of mobile phone-shaped deformable devices influenced how precisely participants were able to move to and maintain target extents of deformation (bend). Given the inherent haptic feedback available from deforming devices (over rigid devices), we also compared performance with, and without, external visual feedback. User perception and preference regarding the different devices were also elicited. Results show that, while device stiffness did not significantly affect task performance, user comfort and preferences were strongly in favour of softer materials (0.45 N·m/rad) and moderate amounts of deformation. Removing external visual feedback led to less precise user input, but inaccuracy remained low enough to suggest non-visual interaction with deformable devices is feasible.

This paper investigates how the perceived physicality of the action of applying force with a finger on a rigid surface (such as on a force-sensing touch screen) can be enhanced using real-time synthesized audio feedback. A selection of rich and evocative audio designs was used. Additionally, audio-tactile cross-modal integration was encouraged, by observing that the main rules of multisensory integration were supported. The study conducted showed that richness of perceived physicality increased considerably, mostly in its auditory expression (what pressing sounded like). In addition, in many instances it was observed that the haptic expression of physicality also increased (what pressing felt like), including some perception of compliance. This last result was particularly interesting as it showed that audio-tactile cross-modal integration might be present.