An increasing number of HCI researchers have embraced open science ideas, such as sharing data and analysis code. However, such practices are only meaningful when the shared data and code enable other researchers to reproduce and reuse the reported findings. To investigate the reproducibility of HCI research, we identified all CHI papers that have shared study data and analysis code, and attempted to reproduce the results. We were able to fully reproduce 49% of the papers. We surveyed and interviewed authors, asking them to assess the reproducibility of their own work and to reflect on their motivations and obstacles in doing open science. We discuss what improves and hinders reproducibility and provide recommendations on how to increase reproducibility rates in HCI. While the value of replicability remains contested in HCI, we argue that the more modest goal of reproducibility is desirable.

Effects of embodying virtual avatars are routinely validated experimentally by comparing synchronous and asynchronous movements between virtual and real bodies. This experimental paradigm, however, lacks justification, validation, and standardization. Asynchrony is implemented in numerous ways, such as through delayed, dislocated, or prerecorded movements, and these may impact embodiment and user experience distinctively. An online study ($N = 202$) revealed that variations of asynchrony cause disparate responses to embodiment and user experience, with prerecorded movements distorting embodiment the most. A think-aloud study ($N = 16$) revealed that asynchronous conditions lead to peculiar and oftentimes negative experiences. Furthermore, asynchronous conditions in some cases maintain, rather than break the body ownership illusion, as participants imitate the virtual body. Our results show that asynchrony in experiments on embodiment entails profound validity issues and should therefore be used with caution.

Demand characteristics refer to cues that can inform participants in experiments about the hypothesis and influence their behavior. They lead researchers to erroneously infer non-existing effects, undermining the experimental integrity of empirical studies. Despite a widespread acknowledgment of their confounding influence in experimental psychology, experiments involving humans and computers to a lesser extent consider effects of demand characteristics, as computerized protocols are thought to be immune to some experimenter biases. Furthermore, demand characteristics are considered to mainly effect subjective measures. As a result, demand characteristics often remain uncontrolled in studies involving computers, and in particular for objective measures such as performance. In this paper, we present two experiments that underline the importance of demand characteristics in human–computer interaction experiments. In a text-entry study, we made participants believe they were evaluating a research-based keyboard. This belief led to increased performance and self-reported user experience. In a second study, we conducted a thought experiment on the illusion of body ownership in virtual reality, where the experimental design indicated the study hypothesis. We found hypothesis-compliant responses from participants, even when they did not experience the illusion. We conclude that demand characteristics pose a significant challenge to the interpretation and validity of human–computer experiments, even when they are fully automated. We discuss the implications and offer guidelines to mitigate effects of demand characteristics.

In the 1979 book “The Meaning of Things” Csikszentmihalyi and Rochberg-Halton studied people’s perception of the significance of things in the home. They emphasized how things influence the self, and vice versa. We propose that their method and analytical framework can help to understand the analogous question for smartphones: Why are some apps special to users? Using the framework, we conduct and analyze 60 interviews with people aged 21 to 41; with participants’ consent, we made the anonymized transcripts publicly available. The analysis of the interviews shows that participants find apps special because they are convenient, support personal goals and social communication, help them remember, and serve emotional functions. Participants report that their identity is intertwined with certain apps, even if they are annoying or cause dependency. Importantly, we also find that participants actively regulate their use of apps through their organization and particular use strategies.