Understanding and modelling user behavior with search results is important to both search engine designers and the design of effectiveness measures. It is well established that users are less likely to view lower ranked search results, and recent research has shown that the type of relevant documents can influence when people stop examining results. However, while existing measures and research consider that relevant documents vary in utility and make use of relevance grades or preference judgments, non-relevant documents are largely all treated the same. In this paper, we show that the nature of non-relevant material affects users’ willingness to further explore a ranked list of search results. We first broaden our notion of non-relevant documents and define a spectrum of possible search engine result pages (SERPs). At one end of the spectrum, the search results were filled with off-topic non-relevant documents, and at the other end, the non-relevant documents were all on-topic, but failed to match the required sub-topic of the search task. We conducted a user study where participants used a mobile search interface to find answers to questions, and collected participants’ behavior while interacting with different SERPs on our spectrum. Our results show that user examination of search results, and time to query abandonment, is influenced by the coherence and type of non-relevant documents included in the SERP. When the SERP is coherent on an egregious topic, users spend the least amount of time before abandoning and are less likely to request to view more results. The time they spend increases as the SERP quality improves, and users are more likely to request to view more results when the SERP contains diversified non-relevant results on multiple subtopics. Our research implies that to improve information retrieval evaluation, we should be assessing the degree of non-relevance in search results as well as the degree of relevance.

When searching the web for answers to health questions, people can make incorrect decisions that have a negative effect on their lives if the search results contain misinformation. To reduce health misinformation in search results, we need to be able to detect documents with correct answers and promote them over documents containing misinformation. Determining the correct answer has been a difficult hurdle to overcome for participants in the TREC Health Misinformation Track. In the 2021 track, automatic runs were not allowed to use the known answer to a topic's health question, and as a result, the top automatic run had a compatibility-difference score of 0.043 while the top manual run, which used the known answer, had a score of 0.259. The compatibility-difference measures the ability of methods to rank correct and credible documents before incorrect and non-credible documents. By using an existing set of health questions and their known answers, we show it is possible to learn which web hosts are trustworthy, from which we can predict the correct answers to the 2021 health questions with an accuracy of 76%. Using our predicted answers, we can promote documents that we predict contain this answer and achieve a compatibility-difference score of 0.129, which is a three-fold increase in performance over the best previous automatic method.

Information retrieval researchers often use eye-tracking to gain insights into searchers' decision making processes. In this paper, we present a visualizing method for summarizing the gaze patterns of multiple searchers on search engine result pages (SERPs). Unlike traditional eye-tracking heat maps, this method includes timing information as part of the visualization, providing additional clarity about searcher fixations as time passes. We demonstrate the visualization technique using eye-tracking data collected as part of a previously published search engine user study and show its value in communicating different patterns of searchers' gaze behavior under different user types and query types. We include a code sample in R to facilitate adoption of the method.

When a search result does not satisfy a user's needs, the user often abandons their query and submits a reformulated query in the hopes of receiving better search results. The action of abandoning search results is termed query abandonment, and previous research has indicated possible reasons for this action, such as dissatisfaction with the result or coming up with a better query. Query abandonment can be seen as a negative or positive signal. As we move closer to understanding when and what causes a user to abandon their query under different qualities of search results, we move forward in the development of an overall understanding of user behaviour with search engines. This can be helpful for developing more accurate evaluation measures and better methods of collecting relevance judgments. In this work project, we plan to study how the quality of both queries and search results can affect the rate and time users abandon their queries. Specifically, we discuss experiments to investigate the rate at which users abandon their query at different levels of search quality, and whether user examination is affected by the type of non-relevant results. User behaviour in our studies will be analyzed with a combination of eye-tracking and questionnaires. The use of eye-tracking will accurately measure user attention to different elements in the search engine results page (SERP) and how far in the search results a user examines before abandoning the query. Questionnaires will provide insights from users' perspectives into the form and quality of submitted queries.

To determine key factors that affect a user's behavior with search results, we conducted a controlled eye-tracking study of users completing search tasks using both desktop and mobile devices. We focus our investigation on users' behavior from their query to the first action they take with the search engine results page (SERP): either a click on a search result or a reformulation of their query. We found that a user deciding to reformulate a query rather than click on a result is best understood as being caused by the user's examination pattern not including a relevant search result. If a user sees a relevant result, they are very likely to click it. Of note, users do not look at all search results and their examination may be influenced by other factors. The key factors we found to explain a user's examination pattern are: the rank of search results, the user type, and the query quality. While existing research has identified rank and user types as important factors affecting examination patterns, to our knowledge, query quality is a new discovery. We found that user queries can be understood as either of weak or strong quality. Weak queries are those that the user may believe are more likely to fail compared to a strong query, and as a result, we find that users modify their examination patterns to view fewer documents when they issue a weak query, i.e. they give up sooner.

A team of six assessors used Dynamic Sampling (Cormack and Grossman 2018) and one hour of assessment effort per topic to form, without pooling, a test collection for the TREC 2018 Common Core Track. Later, official relevance assessments were rendered by NIST for documents selected by depth-10 pooling augmented by move-to-front (MTF) pooling (Cormack et al. 1998), as well as the documents selected by our Dynamic Sampling effort. MAP estimates rendered from dynamically sampled assessments using the xinfAP statistical evaluator are comparable to those rendered from the complete set of official assessments using the standard trec_eval tool. MAP estimates rendered using only documents selected by pooling, on the other hand, differ substantially. The results suggest that the use of Dynamic Sampling without pooling can, for an order of magnitude less assessment effort, yield information-retrieval effectiveness estimates that exhibit lower bias, lower error, and comparable ability to rank system effectiveness.

When search results fail to satisfy users» information needs, users often reformulate their search query in the hopes of receiving better results. In many cases, users immediately requery without clicking on any search results. In this paper, we report on a user study designed to investigate the rate at which users immediately reformulate at different levels of search quality. We had users search for answers to questions as we manipulated the placement of the only relevant document in a ranked list of search results. We show that as the quality of search results decreases, the probability of immediately requerying increases. We find that users can quickly decide to immediately reformulate, and the time to immediately reformulate appears to be independent of the quality of the search results. Finally, we show that there appears to be two types of users. One group has a high probability of immediately reformulating and the other is unlikely to immediately reformulate unless no relevant documents can be found in the search results. While requerying takes time, it is the group of users who are more likely to immediately requery that are able to able find answers to questions the fastest.

High-recall retrieval --- finding all or nearly all relevant documents --- is critical to applications such as electronic discovery, systematic review, and the construction of test collections for information retrieval tasks. The effectiveness of current methods for high-recall information retrieval is limited by their reliance on human input, either to generate queries, or to assess the relevance of documents. Past research has shown that humans can assess the relevance of documents faster and with little loss in accuracy by judging shorter document surrogates, e.g.\ extractive summaries, in place of full documents. To test the hypothesis that short document surrogates can reduce assessment time and effort for high-recall retrieval, we conducted a 50-person, controlled, user study. We designed a high-recall retrieval system using continuous active learning (CAL) that could display either full documents or short document excerpts for relevance assessment. In addition, we tested the value of integrating a search engine with CAL. In the experiment, we asked participants to try to find as many relevant documents as possible within one hour. We observed that our study participants were able to find significantly more relevant documents when they used the system with document excerpts as opposed to full documents. We also found that allowing participants to compose and execute their own search queries did not improve their ability to find relevant documents and, by some measures, impaired performance. These results suggest that for high-recall systems to maximize performance, system designers should think carefully about the amount and nature of user interaction incorporated into the system.

The goal of high-recall information retrieval (HRIR) is to find all or nearly all relevant documents for a search topic. In this paper, we present the design of our system that affords efficient high-recall retrieval. HRIR systems commonly rely on iterative relevance feedback. Our system uses a state-of-the-art implementation of continuous active learning (CAL), and is designed to allow other feedback systems to be attached with little work. Our system allows users to judge documents as fast as possible with no perceptible interface lag. We also support the integration of a search engine for users who would like to interactively search and judge documents. In addition to detailing the design of our system, we report on user feedback collected as part of a 50 participants user study. While we have found that users find the most relevant documents when we restrict user interaction, a majority of participants prefer having flexibility in user interaction. Our work has implications on how to build effective assessment systems and what features of the system are believed to be useful by users.