Abstract Differences in cognitive abilities affect search behaviors, but this has mostly been observed in laboratory experiments. There is limited research on how users search for information in real‐world, naturalistic settings and how real‐world search behaviors relate to cognitive abilities. In this study, we investigated a wide range of behavioral data captured from real‐life search tasks, their association with users' cognitive abilities, and the potential for automatically inferring cognitive abilities from these data. Furthermore, we aimed to determine the data quantity and monitoring duration needed to effectively estimate cognitive abilities from naturalistic behavior. Twenty individuals with βvarying cognitive abilities participated in the experiments in which their everyday search behavior was continuously recorded for 14 days. Their cognitive ability was evaluated through standard tests conducted individually. Data consisted of over 800 h of monitoring, including 2022 queries extracted from 1,442,447 screen frames and associated operating system logs. Using these data, naturalistic search behaviors were associated with cognitive abilities, and predictive models were trained. The results showed that lower selective attention was found to be associated with longer dwelling on selected search results. Faster psychomotor speed and higher fluid intelligence were found to be associated with a greater amount of text read on selected pages. Predictive models exhibited small error rates in predicting cognitive abilities.

Web search ranking models learn from human interactions to improve retrieval performance, but they are presently limited by their use of behavioral factors, such as click-through data or dwell time, that do not account for differences in their users’ cognition. However, it is well understood that users’ behavior varies according to their abilities in processing information, making inferences, and interacting with computing systems. As a result, researchers may miss opportunities to design ranking models that are optimized for their users’ cognitive abilities. To address this, we report an approach for search result re-ranking that incorporates cognitive ability information in the ranking model. We report extensive empirical in-the-wild experiments with data from simulated tasks and real-world tasks of 20 participants to measure, predict, and use these data to train search result re-ranking models. Our results demonstrate that cognitive ability data significantly improve the effectiveness of re-ranking models in simulated-task and real-world conditions, and that cognitive abilities can be predicted from regular user interactions without requiring separate cognitive testing for each user. In particular, the models show improved performance in predicting the position of the documents the users select during search sessions. Our findings show that search engines have significant potential to improve their ranking performance by accounting for users’ cognitive ability.

Our digital life consists of activities that are organized around tasks and exhibit different user states in the digital contexts around these activities. Previous works have shown that digital activity monitoring can be used to predict entities that users will need to perform digital tasks. There have been methods developed to automatically detect the tasks of a user. However, these studies typically support only specific applications and tasks, and relatively little research has been conducted on real-life digital activities. This article introduces user state modeling and prediction with contextual information captured as entities, recorded from real-world digital user behavior, called entity footprinting —a system that records users’ digital activities on their screens and proactively provides useful entities across application boundaries without requiring explicit query formulation. Our methodology is to detect contextual user states using latent representations of entities occurring in digital activities. Using topic models and recurrent neural networks, the model learns the latent representation of concurrent entities and their sequential relationships. We report a field study in which the digital activities of 13 people were recorded continuously for 14 days. The model learned from this data is used to (1) predict contextual user states and (2) predict relevant entities for the detected states. The results show improved user state detection accuracy and entity prediction performance compared to static, heuristic, and basic topic models. Our findings have implications for the design of proactive recommendation systems that can implicitly infer users’ contextual state by monitoring users’ digital activities and proactively recommending the right information at the right time.

Individuals are known to differ in cognitive abilities, affecting their behavior and information processing in digital environments. However, we have a limited understanding of which behaviors are affected, how, and whether some features extracted from digital behavior can predict cognitive abilities. Consequently, researchers may miss opportunities to design and support individuals with personalized experiences and detect those who may benefit from additional interventions. To characterize digital behaviors, we collected 24/7 screen recordings, input behavior, and operating system data from the laptops of 20 adults for two weeks. We use cognitive test results from the same individuals to characterize their cognitive abilities: psychomotor speed, processing speed, selective attention, working memory, and fluid intelligence. Our results from regression analysis, path modeling, and machine learning experiments show that cognitive abilities are associated with differences in digital behavior and that naturalistic behavioral data can predict the cognitive abilities of individuals with small error rates. Our findings suggest naturalistic interaction data as a novel source for modeling cognitive differences.

Information retrieval systems often consider search-session and immediately preceding web-browsing history as the context for predicting users’ present information needs. However, such context is only available when a user’s information needs originate from web context or when users have issued preceding queries in the search session. Here, we study the effect of more extensive context information recorded from users’ everyday digital activities by monitoring all information interacted with and communicated using personal computers. Twenty individuals were recruited for 14 days of 24/7 continuous monitoring of their digital activities, including screen contents, clicks, and operating system logs on Web and non-Web applications. Using this data, a transformer architecture is applied to model the digital activity context and predict representations of personalized information needs. Subsequently, the representations of information needs are used for query prediction, query auto-completion, selected search result prediction, and Web search re-ranking. The predictions of the models are evaluated against the ground truth data obtained from the activity recordings. The results reveal that the models accurately predict representations of information needs improving over the conventional search session and web-browsing contexts. The results indicate that the present practice for utilizing users’ contextual information is limited and can be significantly extended to achieve improved search interaction support and performance.

We study the effect of contextual information obtained from a user’s digital trace on Web search performance. Contextual information is modeled using Dirichlet–Hawkes processes (DHP) and used in augmenting Web search queries. The context is captured by monitoring all naturally occurring user behavior using continuous 24/7 recordings of the screen and associating the context with the queries issued by the users. We report a field study in which 13 participants installed a screen recording and digital activity monitoring system on their laptops for 14 days, resulting in data on all Web search queries and the associated context data. A query augmentation (QAug) model was built to expand the original query with semantically related terms. The effects of context window and source were determined by training context models with temporally varying context windows and varying application sources. The context models were then utilized to re-rank the QAug model. We evaluate the context models by using the Web document rankings of the original query as a control condition compared against various experimental conditions: (1) a search context condition in which the context was sourced from search history; (2) a non-search context condition in which the context was sourced from all interactions excluding search history; (3) a comprehensive context condition in which the context was sourced from both search and non-search histories; and (4) an application-specific condition in which the context was sourced from interaction histories captured on a specific application type. Our results indicated that incorporating more contextual information significantly improved Web search rankings as measured by the positions of the documents on which users clicked in the search result pages. The effects and importance of different context windows and application sources, along with different query types are analyzed, and their impact on Web search performance is discussed.

Everyday digital tasks can highly benefit from systems that recommend the right information to use at the right time. However, existing solutions typically support only specific applications and tasks. In this demo, we showcase EntityBot, a system that captures context across application boundaries and recommends information entities related to the current task. The user’s digital activity is continuously monitored by capturing all content on the computer screen using optical character recognition. This includes all applications and services being used and specific to individuals’ computer usages such as instant messaging, emailing, web browsing, and word processing. A linear model is then applied to detect the user’s task context to retrieve entities such as applications, documents, contact information, and several keywords determining the task. The system has been evaluated with real-world tasks, demonstrating that the recommendation had an impact on the tasks and led to high user satisfaction.

Exploratory search starts with ill-defined goals and involves browsing, learning, and formulating new targets for search. To fluidly support such dynamic search behaviours, we focus on devising interactive visual facets (IVF), visualising information facets to support user comprehension and control of the information space. To do this, we reviewed existing faceted search interfaces and derived two design requirements (DR) that have not been fully addressed to support fluid interactions in exploratory search. We then exemplified the requirements through devising an IVF tool, which coordinates a linear and a categorical facet representing the distribution and summarisation of items, respectively, and providing context for faceted exploration (DR1). To support rapid transitions between search criteria (DR2), the tool introduces a novel design concept of using facets to select items without filtering the item space. Particularly, we propose a filter-swipe technique that enables users to drag a categorical facet value sequentially over linear facet bars to view the items in the intersection of the two facets along with the categorical facet dynamically summarizing the items in the interaction. A user study of 11 participants with realistic email search tasks shows that dynamic suggestions through the timeline navigation can help discover useful suggestions for search; the novel design concept was favoured over using facet values as filters. Based on these practices, we derive IVF design implications for fluid, exploratory searches.

Recommender systems can support everyday digital tasks by retrieving and recommending useful information contextually. This is becoming increasingly relevant in services and operating systems. Previous research often focuses on specific recommendation tasks with data captured from interactions with an individual application. The quality of recommendations is also often evaluated addressing only computational measures of accuracy, without investigating the usefulness of recommendations in realistic tasks. The aim of this work is to synthesize the research in this area through a novel approach by (1) demonstrating comprehensive digital activity monitoring, (2) introducing entity-based computing and interaction, and (3) investigating the previously overlooked usefulness of entity recommendations and their actual impact on user behavior in real tasks. The methodology exploits context from screen frames recorded every 2 seconds to recommend information entities related to the current task. We embodied this methodology in an interactive system and investigated the relevance and influence of the recommended entities in a study with participants resuming their real-world tasks after a 14-day monitoring phase. Results show that the recommendations allowed participants to find more relevant entities than in a control without the system. In addition, the recommended entities were also used in the actual tasks. In the discussion, we reflect on a research agenda for entity recommendation in context, revisiting comprehensive monitoring to include the physical world, considering entities as actionable recommendations, capturing drifting intent and routines, and considering explainability and transparency of recommendations, ethics, and ownership of data.

Web searches often originate from conversations in which people engage before they perform a search. Therefore, conversations can be a valuable source of context with which to support the search process. We investigate whether spoken input from conversations can be used as a context to improve query auto-completion. We model the temporal dynamics of the spoken conversational context preceding queries and use these models to re-rank the query auto-completion suggestions. Data were collected from a controlled experiment and comprised conversations among 12 participant pairs conversing about movies or traveling. Search query logs during the conversations were recorded and temporally associated with the conversations. We compared the effects of spoken conversational input in four conditions: a control condition without contextualization; an experimental condition with the model using search query logs; an experimental condition with the model using spoken conversational input; and an experimental condition with the model using both search query logs and spoken conversational input. We show the advantage of combining the spoken conversational context with the Web-search context for improved retrieval performance. Our results suggest that spoken conversations provide a rich context for supporting information searches beyond current user-modeling approaches.

Understanding users' search behavior has largely relied on the information available from search engine logs, which provide limited information about the contextual factors affecting users' behavior. Consequently, questions such as how users' intentions, task goals, and substances of the users' tasks affect search behavior, as well as what triggers information needs, remain largely unanswered. We report an experiment in which naturalistic information search behavior was captured by analyzing 24/7 continuous recordings of information on participants' computer screens. Written task diaries describing the participants' tasks were collected and used as real‐life task contexts for further categorization. All search tasks were extracted and classified under various task categories according to users' intentions, task goals, and substances of the tasks. We investigated the effect of different task categories on three behavioral factors: search efforts, content‐triggers, and application context. Our results suggest four findings: (i) Search activity is integrally associated with the users' creative processes. The content users have seen prior to searching more often triggers search, and is used as a query, within creative tasks. (ii) Searching within intellectual and creative tasks is more time‐intensive, while search activity occurring as a part of daily routine tasks is associated with more frequent searching within a search task. (iii) Searching is more often induced from utility applications in tasks demanding a degree of intellectual effort. (iv) Users' leisure information‐seeking activity is occurring inherently within social media services or comes from social communication platforms. The implications of our findings for information access and management systems are discussed.

We demonstrate proactive information retrieval via screen surveillance. A user's digital activities are continuously monitored by capturing all content on a user's screen using optical character recognition. This includes all applications and services being exploited and relies on each individual user's computer usage, such as their Web browsing, emails, instant messaging, and word processing. Topic modeling is then applied to detect the user's topical activity context to retrieve information. We demonstrate a system that proactively retrieves information from a user's activity history being observed on the screen when the user is performing unseen activities on a personal computer. We report an evaluation with ten participants that shows high user satisfaction and retrieval effectiveness. Our demonstration and experimental results show that surveillance of a user's screen can be used to build an extremely rich model of a user's digital activities across application boundaries and enable effective proactive information retrieval.