In the era of 'information overload', effective information provision is essential for enabling rapid response and critical decision making. In making sense of diverse information sources, dashboards have become an indispensable tool, providing fast, effective, adaptable, and personalized access to information for professionals and the general public alike. However, these objectives place heavy requirements on dashboards as information systems in usability and effective design. Understanding these issues is challenging given the absence of consistent and comprehensive approaches to dashboard evaluation. In this article we systematically review literature on dashboard implementation in healthcare, where dashboards have been employed widely, and where there is widespread interest for improving the current state of the art, and subsequently analyse approaches taken towards evaluation. We draw upon consolidated dashboard literature and our own observations to introduce a general definition of dashboards which is more relevant to current trends, together with seven evaluation scenarios - task performance, behaviour change, interaction workflow, perceived engagement, potential utility, algorithm performance and system implementation. These scenarios distinguish different evaluation purposes which we illustrate through measurements, example studies, and common challenges in evaluation study design. We provide a breakdown of each evaluation scenario, and highlight some of the more subtle questions. We demonstrate the use of the proposed framework by a design study guided by this framework. We conclude by comparing this framework with existing literature, outlining a number of active discussion points and a set of dashboard evaluation best practices for the academic, clinical and software development communities alike.

Understanding the influence of users’ opinions on their search behavior together with their inherent biases in web search has garnered widespread interest in recent times. This is largely due to the implications of promoting critical thinking, explaining phenomena such as political polarization, or the manifestation of echo chambers. It is important to understand how personal opinions can bias users’ interaction with search results. Moreover, there is a lack of understanding of the impact of user search intents, namely non-purposeful browsing versus searching with a pre-defined goal, on users’ interactions with search results. We take a step towards bridging this knowledge gap through an empirical study in this paper. To do so, we select two controversial topics in abortion and gun control, and invite users to learn about them through ‘Purposeless’ and ‘Purposeful’ web searching. Our findings suggest that users with strong personal opinions exhibit biased interactions with the search results. However, the effect of users’ opinions on their interactions with search results can differ depending on whether users search purposelessly or with a purpose. Our findings advance the current understanding of the effect of users’ opinions in web search sessions, and show that users’ search intents shape their interaction with search results. This work has broad design implications on dealing with bias in interactive information retrieval systems.

Serendipity is highly valued as a process for developing original solutions to problems and for innovation. However, it is difficult to capture and thus difficult to measure, but novelty is a key and critical indicator. In this work, we investigate the relationship between user behavioural actions and perceived novelty in the context of browsing. 180 participants completed an open-ended browsing task, while their behaviour actions were tracked. Each seven-action sequence was analysed with respect to the participant's perception of Novelty. Results showed that 6 of the 7 actions map to a sub-sequence that discriminates between high and low novelty. Notably, switching between exploration and immersion, and checking SERPs about the same request in-depth are indicative of highly perceived novelty. The results show that analysing behavioural action sequences leads to better prediction of novelty, and thus the potential for serendipity, than individual browsing actions.

Evaluating user engagement with search is a critical aspect of understanding how to assess and improve information retrieval systems. While standard techniques for measuring user engagement use questionnaires, these are obtrusive to user interaction, and can only be collected at acceptable intervals. The problem we address is whether there is a less obtrusive and more automatic way to assess how users perceive the search process and outcome. Log files collect behavioural signals (e.g., clicks, queries) from users on a large scale. In this paper, we investigate the potential to predict how users perceive engagement with search by modelling behavioural signals from log files using supervised learning methods. We focus on different engagement dimensions (Perceived Usability, Felt Involvement, Endurability and Novelty) and examine how 37 behavioural features can inform these dimensions. Our results, obtained from 377 in-lab participants undergoing goal-based search tasks, support the connection between perceived engagement and search behaviour. More specifically, we show that time- and query-related features are best suited for predicting user perceived engagement, and suggest that different behavioural features better reflect specific dimensions. We demonstrate the possibility of predicting user-perceived engagement using search behavioural features.

Search is designed primarily for human use, but we still lack efficient user models to evaluate search systems from the user's perspective. The purpose of this research is to identify user behaviour patterns during the search process, and develop a user model to illustrate the relationships between search behaviour and user perception, in order to predict user engagement in a cost-effective manner by using only behaviour data. To achieve this, we will use three datasets collected from previous studies containing both behavioural data from system log files and perception data. Ultimately, this research will yield a cost-efficient predictive model of user engagement that could be used to more effectively evaluate search systems.

Typically, interactive information retrieval (IIR) system evaluations assess search processes and outcomes using a combination of two types of measures: 1. user perception (e.g. users? attitudes of the search experience and outcome); 2. user behaviour (e.g. time and counts of various actions including mouse and keyboard clicks). In general, we assume that they are indicative of the search outcomes (e.g. performance, opinion). However, search is a dynamic process with changing outcomes. Therefore, neither measure solely provides a holistic way of evaluating search. On one hand, user behaviour measures are only descriptive of the outcome, and are not interpretive of the process. That is to say, they lack the rationale behind why those behaviours occurred. Another problem is that some mental activities may not reflect on user behaviour [1]. The challenge with logfiles, which contain behaviour data, is the voluminous number of data points and the need to find a reliable approach to define groups or sets based on behavioural patterns. Not all users are alike and nor do they all take the same approach to search for the same things, as evidenced by the TREC, INEX and CLEF interactive tracks. On the other hand, user perception measures are acquired in such small samples that do not scale to large participant populations, and are rarely measured constantly due to the laborious and time consuming data collection methods (e.g. questionnaire, interview). Moreover, not enough emphasis is put on assessing the reliability of individual perception measures, and the wide usage of likert-type scale limits the interpretation of answers. For a holistic understanding of the search process, we need both perception and behaviour measures. I speculate that user behaviour may predict user perception, and thus we should be able to analyse large-scale files for a greater understanding of the likely human responses.