• Inaccurate AI outputs led to expectation violations. • Expectation violations mediated the effects of AI output accuracy on trust. • Explanations did not moderate the link between accuracy and expectation violations. • For inaccurate AI outputs, explanations led to more trusting behavior. Systems based on Artificial Intelligence (AI) increasingly support decision-making, but their outputs may be inaccurate. Prior research has suggested that explanations might help detect inaccuracies, aiding successful human-AI interaction. This study investigates how the accuracy of system outputs influences users’ trust, trusting behavior, and trustworthiness perceptions, the role of expectation violations in this process, and how explanations for the system outputs influence these effects. In an online study with a 2(explanation vs. no explanation) × 2(accurate vs. inaccurate outputs) between-within design, 218 participants evaluated six job applicants. They received CVs and algorithmic evaluations of applicants’ suitability. For three applicants, outputs were accurate; for the other three, outputs reflected a 40% lower suitability than their true suitability. Half of the participants received explanations. Accurate outputs led to higher trustworthiness, trust, and trusting behavior than inaccurate outputs. Expectation violation fully mediated how accuracy affected trust and trustworthiness, and partially how accuracy influenced trusting behavior. Moreover, there was a significant interaction between explanations and output accuracy concerning trusting behavior: when outputs were accurate, explanations had little effect on trusting behavior; however, when outputs were inaccurate, explanations led to stronger trusting behavior, as participants less strongly deviated from the inaccurate outputs. We conclude that users are able to deviate from inaccurate outputs, and we highlight the importance of expectation violations in this regard. However, our findings also show possible detrimental effects of explanations as they can increase the decisional weight of inaccurate outputs instead of facilitating the detection of inaccuracies.

Trust is a very popular concept in Human–Robot Interaction (HRI) to explain why and how people interact with robots. However, the definition of trust and the methods used to study the concept vary widely, often leading to confusion instead of insight. In this position paper, we discuss possible reasons for the confusion and disagreement by reviewing theory and methods. Our main criticism is that HRI researchers have recently taken an oversimplified approach by not adhering to the process model of trust. Instead, we have primarily measured perceived trustworthiness (often as a proxy for trust) and assumed that it accurately predicts behavior or is satisfactory as an end goal. In addition, many experimental paradigms fail to account for the critical elements of risk and vulnerability that are essential for trust to guide behavior. With this position paper, we aim to shed light on these “trust issues” in HRI and to improve the HRI community’s approach by providing suggestions for enhancing the quality of future research.

In AI decision support, explainability (e.g., disclosing system weaknesses) should help users spot erroneous recommendations, but its efficacy may depend on task difficulty. We ran three experiments in a simulated medical visual detection task. In Experiment 1, we manipulated error difficulty (easy vs. difficult) and explainability (non-XAI vs. XAI). Experiment 2 added a virtually impossible error difficulty. Across both, explainability consistently reduced reliance on incorrect recommendations for difficult errors, showed no benefit for easy errors, and showed a small benefit for impossible errors. Experiment 3 varied error and task difficulty within-subjects and extended these patterns; as task difficulty rose, participants behaved less rationally, exhibiting both under- and overreliance. Notably, these behavioral benefits were generally not accompanied by reduced trust in the AI system. Our findings suggest that disclosing system weaknesses enhances detection of AI errors but is most effective for tasks of moderate difficulty where AI recommendations are still verifiable.

AI-based decision support is increasingly implemented to support operators in dynamic control tasks. While these systems continuously improve, to truly achieve human–system synergy, one must also study humans’ system understanding and behavior. Accordingly, we investigated the impact of explainability instructions regarding a specific system weakness on performance and trust in two experiments (with higher task demands in Experiment 2). Participants performed a dynamic control task with support from either an explainable AI (XAI, information on a system weakness), a non-explainable AI (nonXAI, no information on system weakness), or without support (manual, only in Experiment 2). Results show that participants with XAI support outperformed those in the nonXAI group, particularly in situations where the AI actually erred. Notably, informing users of system weaknesses did not affect trust once they had interacted with the system. In addition, Experiment 2 showed the general benefit of decision support over working manually under higher task demands. These findings suggest that AI support can enhance performance in complex tasks and that providing information on potential system weaknesses aids in managing system errors and resource allocation without compromising trust. • Two experiments examined explainable AI in a dynamic supervisory control task. • One group of subjects was informed about AI weaknesses; i.e. where errors can occur. • Explainability improved performance, especially in cases where the AI erred. • Weakness information did not negatively affect trust in the AI after the interaction. • Explaining AI weaknesses enhanced human–AI collaboration without compromising trust.

In a growing number of application domains, human decision-making is being supported by automated systems. While previous research has focused extensively on the negative consequences of automation support in terms of an overuse of such systems, we argue that this focus has largely overlooked another crucial issue: Humans often deteriorate the performance of automation. Specifically, human–automation dyads commonly perform worse than the system alone because humans, in an attempt to improve decisions, unfortunately interfere with correct system recommendations. This problem will only grow as systems based on artificial intelligence (AI) become more reliable and the gap between human-only and system-only performance continues to widen. We therefore outline the need for research that addresses this persisting and increasingly relevant issue. One approach to counteract this problem is to make systems more transparent and give humans more information on the system. However, while numerous explainability approaches have been brought forward, only very few show convincing effects. To be truly useful, we argue that systems need to be explainable in terms of effective behavioral guidance. Furthermore, beyond just thinking about how to enable humans to better adapt to the system (as is the case with explainability approaches), systems should be more human-centric, taking into account human strengths and weaknesses, and ultimately adapting to humans to enable synergy between humans and AI.

This experiment systematically examines whether, in safety-critical environments such as Air Traffic Control, the negative effects of increasing automation associated with static automation concepts can be mitigated by adaptable automation. Adaptable automation is a form of flexible automation in which the human operator (rather than the system as in adaptive automation) can decide when and to what extent to delegate tasks. A special focus is on its effects on human perception in terms of perceived autonomy and competence, satisfaction, and human role perception. We conducted two online studies using the same dual-task paradigm. Study 1 was conducted with a novice sample via Prolific (N = 93) and study 2 with an expert sample of Air Traffic Controllers (N = 126). Participants were either supported by static information automation, static decision automation, or an adaptable solution that allowed them to switch between the two automation stages. The findings of both studies are similar. Results indicated that, even when humans rarely switched, adaptable automation could increase perceived autonomy, led to high satisfaction, and had positive effects on role perceptions without impairing performance or workload. Furthermore, satisfaction was found to correlate with performance. From a human-centered perspective, flexible concepts seem to be particularly suitable when automation increasingly takes over parts of a job task not only at the stage of information analysis but also at the stage of decision-making.

Anthropomorphism, as a design feature of robots, is widely applied to enhance human-robot interaction in the social domain. Current knowledge of how anthropomorphism influences perception, attitudes, and actual behavior is gained via research with depicted and embodied robots. However, it remains unclear how comparable gained insights of both approaches are. Results of a current meta-analysis suggest that anthropomorphism positively influences subjective and objective measures in case of embodied robots, whereas in case of depicted robots predominantly effects on subjective measures seem to emerge. This follow-up analysis aims to further investigate this difference by using a recoded dataset including data of 41 studies, involving over 3,000 participants. The results illustrate that anthropomorphism investigated via embodied robots indeed facilitates both subjective and objective outcomes. Remarkably, studies concerning effects of anthropomorphism using depicted robots showed positive effects on a subjective level but failed to show any effects on an objective level. In conclusion, the results show that the consequences of anthropomorphism in human-robot interaction depend on how robots are presented to participants. Moreover, they reveal that the transfer of results gained via depicted robots to embodied human-robot interaction might lead to both overestimation on a subjective level and underestimation on an objective level regarding the consequences of anthropomorphism.

The application of anthropomorphic features to robots is generally considered beneficial for human-robot interaction (HRI ). Although previous research has mainly focused on social robots, the phenomenon gains increasing attention in industrial human-Robot interaction as well. In this study, the impact of anthropomorphic design of a collaborative industrial robot on the dynamics of trust and visual attention allocation was examined. Participants interacted with a robot, which was either anthropomorphically or non-anthropomorphically designed. Unexpectedly, attribute-based trust measures revealed no beneficial effect of anthropomorphism but even a negative impact on the perceived reliability of the robot. Trust behavior was not significantly affected by an anthropomorphic robot design during faultless interactions, but showed a relatively steeper decrease after participants experienced a failure of the robot. With regard to attention allocation, the study clearly reveals a distracting effect of anthropomorphic robot design. The results emphasize that anthropomorphism might not be an appropriate feature in industrial HRI as it not only failed to reveal positive effects on trust, but distracted participants from relevant task areas which might be a significant drawback with regard to occupational safety in HRI.