Capturing authentic information behaviors, feelings, and attitudes in natural settings is a challenge in interactive information retrieval research (IIR). Quantitative data collection is useful for understanding IIR at scale. Yet common data collection techniques, such as surveys, lack participants’ reasoning behind their choices. Qualitative data collection, such as think aloud and after protocols, support understanding how people behave, think, and feel immediately following an experience, but may be subject to cognitive biases and are challenging to deliver longitudinally. IIR studies have an opportunity to enhance ecological validity by using mixed and multi-method studies. This paper applies learnings from a longitudinal mixed method study that combines the 'in-situ' benefits of collecting real-time data over time with the benefits of retrospectives. This approach has potential to advance SAL research by providing a contextualised approach to longitudinal data collection and can be used to gain deeper insights into subjective experiences.

This study investigated the effect of video abstract (VA) presentation style (slideshow versus animation) on viewer comprehension and user engagement. Video abstracts, short video presentations of journal articles, were selected and used in a randomized between-subjects experiment (N = 290) with Amazon Mechanical Turk (MTurk) participants. The Cognitive Theory of Multimedia Learning (CTML) informed the selection of VAs. Barrett’s Taxonomy of Cognitive and Affective Dimensions of Reading Comprehension (Barrett’s Taxonomy) was used to develop comprehension measures focused on recall and summarization, and user engagement was measured using a questionnaire. The study found that: 1) comprehension outcomes did not vary between slideshow and animation style VAs, 2) animation VAs were perceived to be more engaging than slideshow VAs, and 3) user engagement was weakly negatively correlated with comprehension scores. In other words, animation VAs attracted viewers with their content, but did not lead to increased comprehension. In fact, viewers in the study who reported higher levels of engagement had slightly lower comprehension outcomes.

research-article Share on The Effects of Domain and Search Expertise on Learning Outcomes in Digital Library Use Authors: Heather O'Brien School of Information, The University of British Columbia, Canada School of Information, The University of British Columbia, CanadaView Profile , Amelia Cole School of Information, The University of British Columbia, Canada School of Information, The University of British Columbia, CanadaView Profile , Andrea Kampen School of Information, The University of British Columbia, Canada School of Information, The University of British Columbia, CanadaView Profile , Kathy Brennan Google, USA Google, USAView Profile Authors Info & Claims CHIIR '22: ACM SIGIR Conference on Human Information Interaction and RetrievalMarch 2022 Pages 202–210https://doi.org/10.1145/3498366.3505761Published:14 March 2022Publication History 0citation127DownloadsMetricsTotal Citations0Total Downloads127Last 12 Months127Last 6 weeks4 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

Interest characterizes a cognitive-emotional relationship between people and information and is a key construct in human information interaction. As a motivational variable, interest has been widely studied in psychology and education, but has received less consistent and theory-driven attention in the field of interactive information retrieval (IIR). In this perspective paper we examine the role of interest and review how it has been studied and operationalized in IIR research. We draw upon a survey of 58 research studies that have manipulated, controlled or measured searcher interest in some way. The intent of the paper is to raise the profile of interest as a user-centred variable in IIR and to advocate for more conceptual and methodological consistency in future studies to better evaluate the impact of interest in information search.

The purpose of this paper is to explore the differences in learning outcomes between domain experts and non-experts interacting with a digital library. This paper provides a preliminary analysis of a subset of data from a larger study and compares the learning outcomes of 10 domain experts and 10 non-experts. Participants completed three search tasks designed to elicit different cognitive processes and behaviors; learning outcomes were explored with pre- and post-session written summaries and pre- and post-task questions. We used existing metrics to examine the learning breadth and depth of the written summaries. General searchers wrote longer summaries than domain experts, but there were no significant differences in the learning outcomes between the two groups.

An important area of IR research involves understanding how task characteristics influence search behaviors and outcomes. Task complexity is one characteristic that has received considerable attention. One view of task complexity is through the lens of a priori determinability -- the level of uncertainty about task outcomes and processes experienced by the searcher. In this work, we manipulated the determinability of comparative tasks. Our task manipulation involved modifying the scope of the task by specifying exact items and/or exact (objective or subjective) dimensions to consider as part of the task. This paper reports on a within-subject study (N=144) where we investigated how our task manipulation influenced participants' perceptions, levels of engagement, search effort, and choice of search strategies. Our results suggest a complex relationship between task scope, determinability, and different outcome measures. Our most open-ended tasks were perceived to have low determinability (high uncertainty), but were the least challenging for participants due to satisficing. Furthermore, narrowing the scope of tasks by specifying items had a different effect than by specifying dimensions. Specifying items increased the task determinability (lower uncertainty) and made the task easier, while specifying dimensions did not increase the task determinability and made the task more challenging. A qualitative analysis of participants' queries suggests that searching for dimensions is more challenging than for items. Finally, we observed subtle differences between objective and subjective dimensions. We discuss implications for the design of IIR studies and tools to support users.

Asking questions is a central element of Interactive Information Retrieval (IIR) studies, yet the challenges associated with asking "good" questions is are rarely discussed. This perspectives paper examines the development, evaluation, and application of questionnaires in IIR research. The authors draw upon prior literature and their own research to explore how to best construct and extract meaning from questionnaires. Since building IIR knowledge and theories requires the employment of valid, reliable, dimensional, and sensitive self-report measures, this paper's contribution is an overview of best practices for developing, administering, and analyzing questionaires.

We conducted a study to examine how people interact with and learn from human-interest news stories. Participants engaged with news stories and completed knowledge retention tests to evaluate their ability to remember the content. We analyzed how the stories and the way in which they were presented (audio, video, text) related to knowledge retention. Results demonstrated a complex interaction between topic and knowledge retention, where participants obtained higher scores for the least preferred story. We explored these findings with respect to participants' demographics and user experience.

User engagement (UE) is a quality of user experience characterized by the depth of an actor's cognitive, temporal, and/or emotional investment in an interaction with a digital system. Currently more art than science, UE has gained theoretical and methodological traction over the past decade, yet there is still a need to establish empirical links between UE and desired outcomes (e.g., learning, behavior change), and to understand the myriad user, system, contextual, and so on, factors that predict successful digital engagement. This paper focuses on the relationship between UE and media format as a potential antecedent, and the outcome of learning, operationalized as short‐term knowledge retention. Participants interacted with two human‐interest stories in one of four media formats: video, audio, narrative text, or transcript‐style text; short‐term knowledge retention was measured using post‐task multiple choice and short‐answer questions. It was anticipated that format would have a strong effect on UE, and that more engaged users would recall more information about the stories. However, these hypotheses were not fully supported, and the nature of the relationship between UE and learning was more nuanced than expected. This research has implications for the design of information systems and, more fundamentally, the impetus to make digital environments engaging.

The purpose of this half-day workshop is to bring together academic and industry interactive information retrieval (IIR) researchers with an interest in evaluation methodologies. The workshop articulates contemporary challenges in the investigation of IIR and invites user- and system-oriented researchers to work collaboratively to address these challenges by combining user- and system-centered methodologies in meaningful ways. We anticipate that this workshop will initiate productive knowledge exchange and partnerships that can respond to the increasing user, task, system, and contextual complexity of the IIR field.

User engagement is recognized as an important component of the user experience, but relatively little is known about the effect of engagement on the learning outcomes of such interactions. This experimental user study examines the relationship between user engagement (UE) and comprehension in varied academic reading environments. Forty-one university students interacted with one of two sets of texts presented in 4 conditions in the context of preparing for a class assignment. Employing the User Engagement Scale (UES), we found evidence of a relationship between students' comprehension of the texts and their degree of engagement with them. However, this association was confined to one of the UES subscales and was not consistent across levels of engagement. An examination of additional variables found little evidence that system and content characteristics influenced engagement; however, we noted that all students' reported increased knowledge, but topical interest for non-engaged students declined. Results contribute to existing literature by adding further evidence that the relationship between engagement and comprehension is complex and mediated.

Online form-filling and transactions are extremely common, both for industry and government; and it is important to provide a satisfying user experience during these tasks if customers or citizens are to continue using online channels. However, reliable measures of experience in these cases are limited. Other areas of information interaction, e.g., online search, news, and shopping, are increasingly exploring and attempting to measure the concept of user engagement (UE). In this study, we ask whether UE is an appropriate outcome for the utilitarian activities of online form-filling and transactions.

When it comes to evaluating online information experiences, what metrics matter? We conducted a study in which 30 people browsed and selected content within an online news website. Data collected included psychometric scales (User Engagement, Cognitive Absorption, System Usability Scales), self‐reported interest in news content, and performance metrics (i.e., reading time, browsing time, total time, number of pages visited, and use of recommended links); a subset of the participants had their physiological responses recorded during the interaction (i.e., heart rate, electrodermal activity, electrocmytogram). Findings demonstrated the concurrent validity of the psychometric scales and interest ratings and revealed that increased time on tasks, number of pages visited, and use of recommended links were not necessarily indicative of greater self‐reported engagement, cognitive absorption, or perceived usability. Positive ratings of news content were associated with lower physiological activity. The implications of this research are twofold. First, we propose that user experience is a useful framework for studying online information interactions and will result in a broader conceptualization of information interaction and its evaluation. Second, we advocate a mixed‐methods approach to measurement that employs a suite of metrics capable of capturing the pragmatic (e.g., usability) and hedonic (e.g., fun, engagement) aspects of information interactions. We underscore the importance of using multiple measures in information research, because our results emphasize that performance and physiological data must be interpreted in the context of users' subjective experiences.

Abstract Facilitating engaging user experiences is essential in the design of interactive systems. To accomplish this, it is necessary to understand the composition of this construct and how to evaluate it. Building on previous work that posited a theory of engagement and identified a core set of attributes that operationalized this construct, we constructed and evaluated a multidimensional scale to measure user engagement. In this paper we describe the development of the scale, as well as two large‐scale studies (N=440 and N=802) that were undertaken to assess its reliability and validity in online shopping environments. In the first we used Reliability Analysis and Exploratory Factor Analysis to identify six attributes of engagement: Perceived Usability, Aesthetics, Focused Attention, Felt Involvement, Novelty, and Endurability. In the second we tested the validity of and relationships among those attributes using Structural Equation Modeling. The result of this research is a multidimensional scale that may be used to test the engagement of software applications. In addition, findings indicate that attributes of engagement are highly intertwined, a complex interplay of user‐system interaction variables. Notably, Perceived Usability played a mediating role in the relationship between Endurability and Novelty, Aesthetics, Felt Involvement, and Focused Attention.

Abstract The purpose of this article is to critically deconstruct the term engagement as it applies to peoples' experiences with technology. Through an extensive, critical multidisciplinary literature review and exploratory study of users of Web searching, online shopping, Webcasting, and gaming applications, we conceptually and operationally defined engagement. Building on past research, we conducted semistructured interviews with the users of four applications to explore their perception of being engaged with the technology. Results indicate that engagement is a process comprised of four distinct stages: point of engagement, period of sustained engagement, disengagement, and reengagement. Furthermore, the process is characterized by attributes of engagement that pertain to the user, the system, and user‐system interaction. We also found evidence of the factors that contribute to nonengagement. Emerging from this research is a definition of engagement—a term not defined consistently in past work—as a quality of user experience characterized by attributes of challenge, positive affect, endurability, aesthetic and sensory appeal, attention, feedback, variety/novelty, interactivity, and perceived user control. This exploratory work provides the foundation for future work to test the conceptual model in various application areas, and to develop methods to measure engaging user experiences.