The integration of artificial intelligence agents into information retrieval systems has prompted two dominant narratives: AI as replacement for human information seeking, and AI as collaborative partner. This perspective paper challenges both framings by proposing a third alternative – that human information seeking behavior and AI agent functionality represent fundamentally incompatible epistemological paradigms. Drawing on information behavior theory and critical AI scholarship, we argue that this incompatibility stems from three dimensions: epistemic orientation (learning vs. pattern reproduction), temporal structure (processual vs. instantaneous), and agentic purpose (uncertainty resolution vs. task execution). Rather than forcing integration through collaboration metaphors or accepting wholesale replacement, we advocate for system designs that acknowledge and preserve these fundamental differences. This reframing has profound implications for the design of information interaction systems, evaluation methodologies, and research directions in human information interaction and retrieval.

As AI agents become more capable of anticipating intent and taking initiative, the ways humans seek, interpret, and act on information are being quietly reshaped. Yet at the heart of every interaction lies a human—curious, uncertain, and contextually situated, whose goals and boundaries cannot be fully captured by data alone. This workshop centers on the human experience of proactivity and personalization in interactive information access, asking how agents can assist without overriding agency, adapt without imposing assumptions, and anticipate without eroding trust. Building on CHIIR’s tradition of bridging information retrieval and human–computer interaction, the workshop will explore when and how proactivity supports human information behavior – enhancing exploration, sense-making, and learning – and when it risks diminishing transparency or control. Through co-design sessions and participatory discussions, we will interrogate concrete design and evaluation dimensions of proactive systems, including timing of initiative, transparency of intent, user control, and their effects on exploration, sense-making, and trust. Ultimately, this workshop seeks to reimagine proactivity not as automation of the search process, but as a collaborative partnership where agents act as companions in the human pursuit of understanding. All resources related to this workshop are available at https://proactive-chiir.github.io/.

With the emergence of search-enabled generative QA systems, users are increasingly turning to tools that browse, aggregate, and reconcile evidence across multiple sources on their behalf. Yet many widely used QA benchmarks remain answerable by retrieving a single relevant passage, making them poorly suited for measuring cross-source sensemaking, such as integrating evidence, tracking causal links, and resolving dependencies across facets of a topic. We present iAgentBench, a dynamic ODQA benchmark that targets these higher-level information needs while keeping questions natural and grounded in realistic information-seeking behavior. iAgentBench draws seed topics from real-world attention signals and uses common user intent patterns to construct user-like questions whose answers require combining evidence from multiple sources, not just extracting a single snippet. Each instance is released with traceable evidence and auditable intermediate artifacts that support contamination checks and enable fine-grained diagnosis of failures in retrieval versus synthesis. Experiments across multiple LLMs show that retrieval improves accuracy, but retrieval alone does not reliably resolve these questions, underscoring the need to evaluate evidence use, not just evidence access.

Conventional recommendation systems (RSs) often optimize for aggregate accuracy, inadvertently underserving users with sparse interaction histories. Although large language models (LLMs) exhibit strong zero- and few-shot ranking capabilities, their use in RS pipelines faces a two-fold challenge: (i) scalability and generalizability : since many evaluations rely on small, randomly sampled user subsets that limit generalization to real-world populations; and (ii) responsible adaptation under resource constraints : since LLMs impose non-trivial cost and latency that preclude indiscriminate use. To address these challenges, we propose a hybrid task allocation framework that proactively allocates ranking tasks between traditional RSs and LLMs to enhance robustness and efficiency. Our strategy works by first identifying weak and inactive users who receive suboptimal ranking performance from RSs. Next, we use an in-context learning approach for such users, wherein each user’s interaction history is contextualized as a distinct ranking task. We evaluate our hybrid framework that integrates eight diverse recommendation algorithms, three popular datasets, and three LLMs (both open- and closed-source), demonstrating that it significantly reduces weak users by approximately 12% while maintaining cost-effectiveness through targeted LLM utilization.

For decades, scholars have emphasized that tasks should be the central focus in Information Retrieval (IR).This point of view holds even more significance with the advent of Generative Artificial Intelligence (GenAI) models, which can, among other capabilities, understand natural language, engage in dialog with users, generate bespoke user interfaces, and power agents to help complete tasks.GenAI presents an unprecedented opportunity to finally realize the potential of tasks in IR, enhance task-focused retrieval and interaction, and create "magical" task completion moments for users.In this paper, we explore the rationale and methodology behind this argument.Traditional IR systems support mostly simple tasks.The emergence of GenAI creates an opportunity for IR systems to help users achieve complex tasks and for the IR community to rekindle its interest and demonstrate leadership in this sizable and significant problem space.We underscore the pivotal role of tasks in IR and introduce new evidence supporting the notion that task-centric approaches, abstracted from specific modalities, represent the future of IR.Building on this foundation, we envision the development, utilization, and evaluation of next-generation IR systems.We propose a promising future where IR agents prioritize users, their tasks, and their situations.However, despite their potential to address task-focused and modality-independent IR, agents alone are insufficient.We propose a robust ecosystem around these agents that transcends traditional queries, questions, prompts, and modalities to address users' fundamental needs, tasks, and goals.

As question answering (QA) systems advance alongside the rapid evolution of foundation models, the need for robust, adaptable, and large-scale evaluation benchmarks becomes increasingly critical. Traditional QA benchmarks are often static and publicly available, making them susceptible to data contamination and memorization by large language models (LLMs). Consequently, static benchmarks may overestimate model generalization and hinder a reliable assessment of real-world performance. In this work, we introduce Dynamic-KGQA, a scalable framework for generating adaptive QA datasets from knowledge graphs (KGs), designed to mitigate memorization risks while maintaining statistical consistency across iterations. Unlike fixed benchmarks, Dynamic-KGQA generates a new dataset variant on every run while preserving the underlying distribution, enabling fair and reproducible evaluations. Furthermore, our framework provides fine-grained control over dataset characteristics, supporting domain-specific and topic-focused QA dataset generation. Additionally, Dynamic-KGQA produces compact, semantically coherent subgraphs that facilitate both training and evaluation of KGQA models, enhancing their ability to leverage structured knowledge effectively. To align with existing evaluation protocols, we also provide static large-scale train/test/validation splits, ensuring comparability with prior methods. By introducing a dynamic, customizable benchmarking paradigm, Dynamic-KGQA enables a more rigorous and adaptable evaluation of QA systems.

In the information retrieval (IR) domain, evaluation plays a crucial role in optimizing search experiences and supporting diverse user intents.In the recent LLM era, research has been conducted to automate document relevance labels.These labels have traditionally been assigned by crowd-sourced workers, a process that is both time consuming and costly.This study focuses on LLM-generated usefulness labels, a crucial evaluation metric that considers the user's search intents and task objectives, an aspect where relevance falls short.Our experiment utilizes task-level, query-level, and document-level features along with user search behavior signals, which are essential in defining the usefulness of a document.Our research finds that (i) pre-trained LLMs can generate moderate usefulness labels by understanding the comprehensive search task session, and (ii) pre-trained LLMs perform better judgment in short search sessions when provided with search session contexts.Furthermore, we investigate whether LLMs can capture the unique divergence between relevance and usefulness, along with conducting an ablation study to identify the most critical metrics for accurate usefulness label generation.In conclusion, this work explores LLM-generated usefulness labels by evaluating critical metrics and optimizing for practicality in real-world settings.

As Large Language Models (LLMs) have rapidly advanced in social reasoning tasks, their applications have expanded to domains such as healthcare and psychology. Given the direct interaction of users with these applications, it is essential to evaluate the performance of LLMs, particularly in human-like social reasoning capabilities. While previous studies have explored human-aligned social reasoning in LLMs, they have not adequately assessed whether the generated reasoning answers stem from the LLMs' memorization of training data or their natural language understanding. In this study, we aim to address this gap by assessing the impact of training data memorization on the human-aligned social reasoning capabilities of LLMs. We introduce IR+CoT (Information Retrieval (IR) + Chain of Thought (CoT)), a framework that leverages retrieved information from input questions to fine-tune prompt templates and employs CoT methods. IR+CoT mitigates the effects of memorization and enhances the LLMs' social reasoning performance. Experiments on three LLMs, using seen (present during the training of the LLMs) and unseen (introduced post-training) questions from Reddit and Lemmy, show that IR+CoT enhances social reasoning and reduces memorization effects. This research's novelty lies in using old and new questions to assess memorization's impact on social reasoning.

Abstract Prior research in human–computer interaction suggests that system‐level clarifications are necessary for understanding user intent and communicating effectively with the user. Such clarifications or explanations could contain the system's abstract knowledge of the search or a functional description of the search process (queries and information sources employed). While these interactions may aid the user and the agent in better understanding each other, very few studies have explored the influence of such clarifications on the users' search experience. This research examines whether and how system‐level clarifications (or explanations) affect the user experience when searching through spoken dialogues. We analyzed user satisfaction and preferences in systems with and without explicit clarifications in a within‐subjects Wizard‐of‐Oz user study. We recruited 25 participants and collected user–system interaction data for 50 search sessions. The user feedback was collected using pre‐ and post‐task surveys and exit interviews. Statistical and qualitative analysis of user responses yielded some interesting findings. While Wilcoxon Signed Rank Test found that using explicit system‐level clarifications had no positive influence on the user's search experience, the overall search experience degraded with system clarifications (Z = −2.066, p = 0.04). The user interview data provided valuable insights into how and when clarifications should be offered to the user.

A recommender system is tasked with effectively analyzing a user’s preferences and interactions to provide personalized recommendations. This calls for extracting and connecting various heterogeneous data while preserving their temporal relations. Graph neural networks (GNNs) have proven to be highly suitable in recommendation systems for connecting different types of user behavioral signals. However, they inherently lack ability to capture temporal aspects of underlying data. This shortcoming prevents them from explicating and utilizing task information, which is shown to be instrumental in many information retrieval applications. To overcome this limitation, we propose a new Task-based Graph Neural Network model (TGNN) focusing on identifying users’ underlying tasks within their temporal multi-behavior, specifically in each session. The model consists of three modules: (1) a sequential meta-path module that captures a temporal sequence of users’ behaviors; (2) a graph neural network layer that models the relationships between different information items and users into task representations; and (3) a recommendation layer that utilizes a collaborative filtering method to generate top-N recommendations based on the model’s comprehension of users’ tasks. The novelty of our approach lies in understanding users’ tasks through their temporal behavior, enabling more accurate personalization. The results of evaluative experiments on three publicly available datasets demonstrate the effectiveness of our task-based recommendation model compared to 10 baselines and indicate a promising research direction for task-oriented recommender systems.

Understanding user intents in information access scenarios can help us provide more relevant and personalized search results and recommendations. However, analyzing user intents is not easy, especially for emerging forms of Web search such as Artificial Intelligence (AI)-driven chat. To understand user intents from retrospective log data, we need a way to label them with meaningful categories that capture their diversity and dynamics. Existing methods rely on manual or Machine-Learned (ML) labeling, which is either expensive or inflexible for large and dynamic datasets. Large Language Models (LLMs) could generate rich and relevant concepts, descriptions, and examples for user intents using log data of user interactions. However, using LLMs to generate a user intent taxonomy and applying it for a given Information Retrieval (IR) application can be problematic for two main reasons: (1) such a taxonomy is not externally validated; and (2) there may be an undesirable feedback loop if an LLM does both these tasks without external validation. To address this, we propose a new methodology with human experts and assessors to verify the quality of the LLM-generated taxonomy. We also present an end-to-end pipeline that uses an LLM with Human-in-the-Loop (HITL) to produce, refine, and apply labels for user intent analysis in log data. We demonstrate its effectiveness by uncovering new insights into user intents from search and chat logs from the Microsoft Bing Web search engine. The novelty in this research stems from the method for generating purpose-driven user intent taxonomies with strong validation. Our approach not only helps remove methodological and practical bottlenecks from intent-focused research, but also provides a new framework for generating, validating, and applying other kinds of taxonomies in a scalable and adaptable way, with reasonable human effort.

As Large Language Models (LLMs) are integrated into various sectors, ensuring their reliability and safety is crucial. This necessitates rigorous probing and auditing to maintain their effectiveness and trustworthiness in practical applications. Subjecting LLMs to varied iterations of a single query can unveil potential inconsistencies in their knowledge base or functional capacity. However, a tool for performing such audits with a easy to execute workflow, and low technical threshold is lacking. In this demo, we introduce "AuditLLM," a novel tool designed to audit the performance of various LLMs in a methodical way. AuditLLM's primary function is to audit a given LLM by deploying multiple probes derived from a single question, thus detecting any inconsistencies in the model's comprehension or performance. A robust, reliable, and consistent LLM is expected to generate semantically similar responses to variably phrased versions of the same question. Building on this premise, AuditLLM generates easily interpretable results that reflect the LLM's consistency based on a single input question provided by the user. A certain level of inconsistency has been shown to be an indicator of potential bias, hallucinations, and other issues. One could then use the output of AuditLLM to further investigate issues with the aforementioned LLM. To facilitate demonstration and practical uses, AuditLLM offers two key modes: (1) Live mode which allows instant auditing of LLMs by analyzing responses to real-time queries; and (2) Batch mode which facilitates comprehensive LLM auditing by processing multiple queries at once for in-depth analysis. This tool is beneficial for both researchers and general users, as it enhances our understanding of LLMs' capabilities in generating responses, using a standardized auditing platform.

Theory of mind (ToM) reasoning involves understanding that others have intentions, emotions, and thoughts, which is crucial for regulating one's reasoning. Although large language models (LLMs) excel in tasks such as summarization, question answering, and translation, they still face challenges with ToM reasoning, especially in open-ended questions. Despite advancements, the extent to which LLMs truly understand ToM reasoning and how closely it aligns with human ToM reasoning remains inadequately explored in open-ended scenarios. Motivated by this gap, we assess the abilities of LLMs to perceive and integrate human intentions and emotions into their ToM reasoning processes within open-ended questions. Our study utilizes posts from Reddit's ChangeMyView platform, which demands nuanced social reasoning to craft persuasive responses. Our analysis, comparing semantic similarity and lexical overlap metrics between responses generated by humans and LLMs, reveals clear disparities in ToM reasoning capabilities in open-ended questions, with even the most advanced models showing notable limitations. To enhance LLM capabilities, we implement a prompt tuning method that incorporates human intentions and emotions, resulting in improvements in ToM reasoning performance. However, despite these improvements, the enhancement still falls short of fully achieving human-like reasoning. This research highlights the deficiencies in LLMs' social reasoning and demonstrates how integrating human intentions and emotions can boost their effectiveness.

Transforming unstructured text into structured and meaningful forms, organized by useful category labels, is a fundamental step in text mining for downstream analysis and application. However, most existing methods for producing label taxonomies and building text-based label classifiers still rely heavily on domain expertise and manual curation, making the process expensive and time-consuming. This is particularly challenging when the label space is under-specified and large-scale data annotations are unavailable. In this paper, we address these challenges with Large Language Models (LLMs), whose prompt-based interface facilitates the induction and use of large-scale pseudo labels. We propose TnT-LLM, a two-phase framework that employs LLMs to automate the process of end-to-end label generation and assignment with minimal human effort for any given use-case. In the first phase, we introduce a zero-shot, multi-stage reasoning approach which enables LLMs to produce and refine a label taxonomy iteratively. In the second phase, LLMs are used as data labelers that yield training samples so that lightweight supervised classifiers can be reliably built, deployed, and served at scale. We apply TnT-LLM to the analysis of user intent and conversational domain for Bing Copilot (formerly Bing Chat), an open-domain chat-based search engine. Extensive experiments using both human and automatic evaluation metrics demonstrate that TnT-LLM generates more accurate and relevant label taxonomies when compared against state-of-the-art baselines, and achieves a favorable balance between accuracy and efficiency for classification at scale.

We observe a recent trend toward applying large language models (LLMs) in search and positioning them as effective information access systems. While the interfaces may look appealing and the apparent breadth of applicability is exciting, we are concerned that the field is rushing ahead with a technology without sufficient study of the uses it is meant to serve, how it would be used, and what its use would mean. We argue that it is important to reassert the central research focus of the field of information retrieval, because information access is not merely an application to be solved by the so-called ‘AI’ techniques du jour. Rather, it is a key human activity, with impacts on both individuals and society. As information scientists, we should be asking what do people and society want and need from information access systems and how do we design and build systems to meet those needs? With that goal, in this conceptual article we investigate fundamental questions concerning information access from user and societal viewpoints. We revisit foundational work related to information behavior, information seeking, information retrieval, information filtering, and information access to resurface what we know about these fundamental questions and what may be missing. We then provide our conceptual framing about how we could fill this gap, focusing on methods as well as experimental and evaluation frameworks. We consider the Web as an information ecosystem and explore the ways in which synthetic media, produced by LLMs and otherwise, endangers that ecosystem. The primary goal of this conceptual article is to shed light on what we still do not know about the potential impacts of LLM-based information access systems, how to advance our understanding of user behaviors, and where the next generations of students, scholars, and developers could fruitfully invest their energies.

Despite the considerable advancements in modern search systems for assisting users in search tasks of varying types, support for complex tasks that call for multi-round interactions remains challenging. Identifying users’ tasks is essential to understanding their evolving information needs and search goals during search sessions to simulate and achieve real-time adaptive search retrievals; thus, it is a crucial research thrust in interactive information retrieval (IIR). While a series of descriptive and formal models have been proposed to characterize complex information search sessions, only a few focus on leveraging dynamic task features in search personalizations to support users in different task stages in an adaptive fashion. This preliminary study presents a heterogeneous graph neural network model for extracting and representing tasks to better understand users’ interactive search processes by connecting tasks with search interactions. Our approach’s novelty lies in our application of task representation learning, which enables systems to extract hidden task information from users’ search behaviors. The results of our evaluative experiments on TREC Session track data highlight the value of our proposed task representation model and illustrate a promising research direction on task-oriented intelligent systems.

The importance of tasks in information retrieval (IR) has been long argued for, addressed in different ways, often ignored, and frequently revisited. For decades, scholars made a case for the role that a user’s task plays in how and why that user engages in search and what a search system should do to assist. But for the most part, the IR community has been too focused on query processing and assuming a search task to be a collection of user queries, often ignoring if or how such an assumption addresses the users accomplishing their tasks. With emerging areas of conversational agents and proactive IR, understanding and addressing users’ tasks has become more important than ever before. In this paper, we provide various perspectives on where the state-of-the-art is with regard to tasks in IR, what are some of the bottlenecks in deriving and using task information, and how do we go forward from here. In addition to covering relevant literature, the paper provides a synthesis of historical and current perspectives on understanding, extracting, and addressing task-focused search. To ground ongoing and future research in this area, we present a new framing device for tasks using a tree-like structure and various moves on that structure that allow different interpretations and applications. Presented as a combination of synthesis of ideas and past works, proposals for future research, and our perspectives on technical, social, and ethical considerations, this paper is meant to help revitalize the interest and future work in task-based IR.

The prominent model of retrieving, evaluating, and using relevant information from databases, collections, and the web is going through a significant transformation. This is largely due to wide-scale availability of various generative AI systems that can take in natural language inputs and generate highly customized natural language text, images, audio, and videos. This transformation in how people seek and access information will have profound impacts on users, developers, and policymakers. It is already changing many sectors including education, health, and commerce. But the hopes and hypes of generative AI are often not clear as we get swept up by either the current capabilities and limitations of this technology in the short term or fear from speculative future in the long term. Instead, I believe we need to approach this area pragmatically and with scientific curiosity, scholarly rigor, and societal responsibility. In this talk, I will highlight some of the opportunities and challenges for information access stemming from recent advancements in generative AI. For instance, there are new possibilities now for addressing accessibility, low-resource domains, and bias in training data using generative AI tools. On the other hand, there are new challenges concerning hallucination, toxicity, and information provenance. It is clear that we want to benefit from what AI systems are capable of, but how do we do that while curbing some of these problems? I will argue that the solution is multifaceted and complex -- some will require technical advancements and others will call for policy changes. We will need to not only build information systems with fairness, transparency, and accountability in mind, but also train a new generation of developers, policymakers, and of course the users. The goal here is to cut through both hype and fear and think pragmatically about the future of information access.

Recommender systems play an essential role in the choices people make in domains such as entertainment, shopping, food, news, employment, and education. The machine learning models underlying these recommender systems are often enormously large and black-box in nature for users, content providers, and system developers alike. It is often crucial for all stakeholders to understand the model's rationale behind making certain predictions and recommendations. This is especially true for the content providers whose livelihoods depend on the recommender system. Drawing motivation from the practitioners' need, in this work, we propose a recourse framework for recommender systems, targeted towards the content providers. Algorithmic recourse in the recommendation setting is a set of actions that, if executed, would modify the recommendations (or ranking) of an item in the desired manner. A recourse suggests actions of the form: ''if a feature changes X to Y, then the ranking of that item for a set of users will change to X.'' Furthermore, we demonstrate that RecRec is highly effective in generating valid, sparse, and actionable recourses through an empirical evaluation of recommender systems trained on three real-world datasets. To the best of our knowledge, this work is the first to conceptualize and empirically test a generalized framework for generating recourses for recommender systems.

ChatGPT and similar large language model (LLM) based conversational agents have brought shock waves to the research world. Although astonished by their human-like performance, we find they share a significant weakness with many other existing conversational agents in that they all take a passive approach in responding to user queries. This limits their capacity to understand the users and the task better and to offer recommendations based on a broader context than a given conversation. Proactiveness is still missing in these agents, including their ability to initiate a conversation, shift topics, or offer recommendations that take into account a more extensive context. To address this limitation, this tutorial reviews methods for equipping conversational agents with proactive interaction abilities.

Conversational agents, or commonly known as dialogue systems, have gained escalating popularity in recent years. Their widespread applications support conversational interactions with users and accomplishing various tasks as personal assistants. However, one key weakness in existing conversational agents is that they only learn to passively answer user queries via training on pre-collected and manually-labeled data. Such passiveness makes the interaction modeling and system-building process relatively easier, but it largely hinders the possibility of being human-like hence lowering the user engagement level. In this tutorial, we introduce and discuss methods to equip conversational agents with the ability to interact with end users in a more proactive way. This three-hour tutorial is divided into three parts and includes two interactive exercises. It reviews and presents recent advancements on the topic, focusing on automatically expanding ontology space, actively driving conversation by asking questions or strategically shifting topics, and retrospectively conducting response quality control.

For users working on a complex search task, it is common to address different goals at various stages of the task through query iterations. While addressing these goals, users go through different task states as well. Understanding these task states latent under users' interactions is crucial in identifying users' changing intents and search behaviors to simulate and achieve real-time adaptive search recommendations and retrievals. However, the availability of sizeable real-world web search logs is scarce due to various ethical and privacy concerns, thus often challenging to develop generalizable task-aware computation models. Furthermore, session logs with task state labels are rarer. For many researchers who lack the resources to directly and at scale collect data from users and conduct a time-consuming data annotation process, this becomes a considerable bottleneck to furthering their research. Synthetic search sessions have the potential to address this gap. This paper shares a parsimonious model to simulate synthetic web search sessions with task state information, which interactive information retrieval (IIR) and search personalization studies could utilize to develop and evaluate task-based search and retrieval systems.

Many disciplines, including the broad Field of Information (iField), have been offering Data Science (DS) programs. There have been significant efforts exploring an individual discipline's identity and unique contributions to the broader DS education landscape. To advance DS education in the iField, the iSchool Data Science Curriculum Committee (iDSCC) was formed and charged with building and recommending a DS education framework for iSchools. This paper reports on the research process and findings of a series of studies to address important questions: What is the iField identity in the multidisciplinary DS education landscape? What is the status of DS education in iField schools? What knowledge and skills should be included in the core curriculum for iField DS education? What are the jobs available for DS graduates from the iField? What are the differences between graduate-level and undergraduate-level DS education? Answers to these questions will not only distinguish an iField approach to DS education but also define critical components of DS curriculum. The results will inform individual DS programs in the iField to develop curriculum to support undergraduate and graduate DS education in their local context.

Search systems, like many other applications of machine learning, have become increasingly complex and opaque. The notions of relevance, usefulness, and trustworthiness with respect to information were already overloaded and often difficult to articulate, study, or implement. Newly surfaced proposals that aim to use large language models to generate relevant information for a user’s needs pose even greater threat to transparency, provenance, and user interactions in a search system. In this perspective paper we revisit the problem of search in the larger context of information seeking and argue that removing or reducing interactions in an effort to retrieve presumably more relevant information can be detrimental to many fundamental aspects of search, including information verification, information literacy, and serendipity. In addition to providing suggestions for counteracting some of the potential problems posed by such models, we present a vision for search systems that are intelligent and effective, while also providing greater transparency and accountability.

Many information access and machine learning systems, including recommender systems, lack transparency and accountability. High-quality recommendation explanations are of great significance to enhance the transparency and interpretability of such systems. However, evaluating the quality of recommendation explanations is still challenging due to the lack of human-annotated data and benchmarks. In this paper, we present a large explanation dataset named RecoExp, which contains thousands of crowdsourced ratings of perceived quality in explaining recommendations. To measure explainability in a comprehensive and interpretable manner, we propose ExpScore, a novel machine learning-based metric that incorporates the definition of explainability from various perspectives (e.g., relevance, readability, subjectivity, and sentiment polarity). Experiments demonstrate that ExpScore not only vastly outperforms existing metrics and but also keeps itself explainable. Both the RecoExp dataset and open-source implementation of ExpScore will be released for the whole community. These resources and our findings can serve as forces of public good for scholars as well as recommender systems users.

Abstract With the emerging needs of creating fairness‐aware solutions for search and recommendation systems, a daunting challenge exists of evaluating such solutions. While many of the traditional information retrieval (IR) metrics can capture the relevance, diversity, and novelty for the utility with respect to users, they are not suitable for inferring whether the presented results are fair from the perspective of responsible information exposure. On the other hand, existing fairness metrics do not account for user utility or do not measure it adequately. To address this problem, we propose a new metric called FAIR. By unifying standard IR metrics and fairness measures into an integrated metric, this metric offers a new perspective for evaluating fairness‐aware ranking results. Based on this metric, we developed an effective ranking algorithm that jointly optimized user utility and fairness. The experimental results showed that our FAIR metric could highlight results with good user utility and fair information exposure. We showed how FAIR related to a set of existing utility and fairness metrics and demonstrated the effectiveness of our FAIR‐based algorithm. We believe our work opens up a new direction of pursuing a metric for evaluating and implementing the FAIR systems.

Abstract In information seeking and retrieval research, researchers often collect data about users' behaviors to predict task characteristics and personalize information for users. The reliability of user behavior may be directly influenced by data collection methods. This article reports on a mixed‐methods study examining the impact of study setting (laboratory setting vs. remote setting) and task authenticity (authentic task vs. simulated task) on users' online browsing and searching behaviors. Thirty‐six undergraduate participants finished one lab session and one remote session in which they completed one authentic and one simulated task. Using log data collected from 144 task sessions, this study demonstrates that the synthetic lab study setting and simulated tasks had significant influences mostly on behaviors related to content pages (e.g., page dwell time, number of pages visited per task). Meanwhile, first‐query behaviors were less affected by study settings or task authenticity than whole‐session behaviors, indicating the reliability of using first‐query behaviors in task prediction. Qualitative interviews reveal why users were influenced. This study addresses methodological limitations in existing research and provides new insights and implications for researchers who collect online user search behavioral data.

People engage in search episodes as they have a task or a problematic situation. Often this task is not clearly expressed by the information seeker, nor directly supported by the search system. People also list their tasks using tools such as to-do applications, and while many of these could be search tasks, there is a lack of that recognition or a possible bridge to a search system. In the work reported here, we aim to create that bridge by analyzing data on both sides. In task management, we examined 1,000 to-do tasks annotated by human assessors for their appropriateness for a search engine and created a simple process to learn that classification. In search, we analyzed millions of queries in a search engine log to understand how often queries represent tasks that people express in to-do lists. Our results show that (1) we can accurately predict which of the to-do tasks are appropriate as search queries; and (2) such tasks do indeed show up in search engines as a substantial segment. Together, these findings outline an opportunity to link explicitly expressed tasks to search queries and vice versa. This has implications for both task completion and query understanding.

Search systems have unprecedented influence on how and what information people access. These gateways to information on the one hand create an easy and universal access to online information, and on the other hand create biases that have shown to cause knowledge disparity and ill-decisions for information seekers. Most of the algorithms for indexing, retrieval, and ranking are heavily driven by the underlying data that itself is biased. In addition, orderings of the search results create position bias and exposure bias due to their considerable focus on relevance and user satisfaction. These and other forms of biases that are implicitly and sometimes explicitly woven in search systems are becoming increasing threats to information seeking and sense-making processes. In this tutorial, we will introduce the issues of biases in data, in algorithms, and overall in search processes and show how we could think about and create systems that are fairer, with increasing diversity and transparency. Specifically, the tutorial will present several fundamental concepts such as relevance, novelty, diversity, bias, and fairness using socio-technical terminologies taken from various communities, and dive deeper into metrics and frameworks that allow us to understand, extract, and materialize them. The tutorial will cover some of the most recent works in this area and show how this interdisciplinary research has opened up new challenges and opportunities for communities such as SIGIR.

Abstract The Information Schools, also referred to as iSchools, have a unique approach to data science with three distinct components: human‐centeredness, socially responsible, and rooted in context. In this position paper, we highlight and expand on these components and show how they are integrated in various research and educational activities related to data science that are being carried out at iSchools. We argue that the iSchool way of doing data science is not only highly relevant to the current times, but also crucial in solving problems of tomorrow. Specifically, we accentuate the issues of developing insights and solutions that are not only data‐driven, but also incorporate human values, including transparency, privacy, ethics, fairness, and equity. This approach to data science has meaningful implications on how we educate the students and train the next generation of scholars and policymakers. Here, we provide some of those design decisions, rooted in evidence‐based research, along with our perspective on how data science is currently situated and how it should be advanced in iSchools.

Complex search tasks that involve uncertain solution space and multi-round search iterations are integral to everyday life and information-intensive workplace practices, affecting how people learn, work, and resolve problematic situations. However, current search systems still face plenty of challenges when applied in supporting users engaging in complex search tasks. To address this issue, we seek to explore the dynamic nature of complex search tasks from process-oriented perspective by identifying and predicting implicit task states. Specifically, based upon the Web search logs and user annotation data (regarding information seeking intentions in local search steps, in-situ search problems, and help needed) collected from 132 search sessions in two controlled lab studies, we developed two task state frameworks based on intention state and problem-help state respectively and examined the connection between task states and search behaviors. We report that (1) complex search tasks of different types can be deconstructed and disambiguated based on the associated nonlinear state transition patterns; and (2) the identified task states that cover multiple subtle factors of user cognition can be predicted from search behavioral signals using supervised learning algorithms. This study reveals the way in which complex search tasks are unfolded and manifested in users' search interactions and paves the way for developing state-aware adaptive search supports and system evaluation frameworks.

Search and recommender systems have unprecedented influence on how and what information people access. These gateways to information on the one hand create an easy and universal access to online information, and on the other hand create biases that have shown to cause knowledge disparity and ill-decisions for information seekers. Most of the algorithms for indexing, retrieval, ranking, and recommendation are heavily driven by the underlying data that itself is biased. In addition, ordering of the search and recommendation results create position bias and exposure bias due to their considerable focus on relevance and user satisfaction. These and other forms of biases that are implicitly and some times explicitly woven in search and recommender systems are becoming increasing threats to information seeking and sense-making processes. In this tutorial, we will introduce the issues of biases in search and recommendation and show how we could think about and create systems that are fairer, with increasing diversity and transparency. Specifically, the tutorial will present several fundamental concepts such as relevance, novelty, diversity, bias, and fairness using socio-technical terminologies taken from various communities, and dive deeper into metrics and frameworks that allow us to understand, extract, and materialize them. The tutorial will cover some of the most recent works in this area and show how this interdisciplinary research has opened up new challenges and opportunities for communities such as RecSys.

Recommender systems offer great opportunity not only for users to discover new content, but also for the providers of that content to find new audience, followers, and fans. Users often come to a recommender system with certain expectations about what it will recommend to them, and a recommender system that is optimized for creating opportunities for content creators may provide recommendations that are very different from what a user is expecting. We hypothesize that some users’ expectations have a much wider range of acceptability than others, and users with more ”receptivity” to subversion of their expectations are likely to accept such divergence in the recommended content. Understanding users’ responses to such recommendations is vital to platforms that need to serve multiple stakeholders. In this work we investigate logged behavioral responses of users of an audio streaming platform to recommendations that deviate from their expectation, or “divergent” recommendations. We present three classes of listener response to divergent recommendations that can be identified in interaction logs with the aim of predicting which users can be targeted for future divergent recommendations. We derive a number of user characteristics based on user’s music consumption which we think are predictive of user’s receptivity, train models to predict receptivity of these users, and run a live A/B test to validate our approach by correlating with engagement.

There has been growing attention on fairness considerations recently, especially in the context of intelligent decision making systems. For example, explainable recommendation systems may suffer from both explanation bias and performance disparity. We show that inactive users may be more susceptible to receiving unsatisfactory recommendations due to their insufficient training data, and that their recommendations may be biased by the training records of active users due to the nature of collaborative filtering, which leads to unfair treatment by the system. In this paper, we analyze different groups of users according to their level of activity, and find that bias exists in recommendation performance between different groups. Empirically, we find that such performance gap is caused by the disparity of data distribution, specifically the knowledge graph path distribution in this work. We propose a fairness constrained approach via heuristic re-ranking to mitigate this unfairness problem in the context of explainable recommendation over knowledge graphs. We experiment on several real-world datasets with state-of-the-art knowledge graph-based explainable recommendation algorithms. The promising results show that our algorithm is not only able to provide high-quality explainable recommendations, but also reduces the recommendation unfairness in several aspects.

While great strides are made in the field of search and recommendation, there are still challenges and opportunities to address information access issues that involve solving tasks and accomplishing goals for a wide variety of users. Specifically, we lack intelligent systems that can detect not only the request an individual is making (what), but also understand and utilize the intention (why) and strategies (how) while providing information. Many scholars in the fields of information retrieval, recommender systems, productivity (especially in task management and time management), and artificial intelligence have recognized the importance of extracting and understanding people's tasks and the intentions behind performing those tasks in order to serve them better. However, we are still struggling to support them in task completion, e.g., in search and assistance, it has been challenging to move beyond single-query or single-turn interactions. The proliferation of intelligent agents has opened up new modalities for interacting with information, but these agents will need to be able to work more intelligently in understanding the context and helping the users at task level. This tutorial will introduce the attendees to the issues of detecting, understanding, and using task and task-related information in an information episode (with or without active searching). Specifically, it will cover several recent theories, models, and methods that show how to represent tasks and use behavioral data to extract task information. It will then show how this knowledge or model could contribute to addressing emerging retrieval and recommendation problems.

Explainable recommendation and search attempt to develop models or methods that not only generate high-quality recommendation or search results, but also interpretability of the models or explanations of the results for users or system designers, which can help to improve the system transparency, persuasiveness, trustworthiness, and effectiveness, etc. This is even more important in personalized search and recommendation scenarios, where users would like to know why a particular product, web page, news report, or friend suggestion exists in his or her own search and recommendation lists. The workshop focuses on the research and application of explainable recommendation, search, and a broader scope of IR tasks. It will gather researchers as well as practitioners in the field for discussions, idea communications, and research promotions. It will also generate insightful debates about the recent regulations regarding AI interpretability, to a broader community including but not limited to IR, machine learning, AI, Data Science, and beyond.

In Interactive Information Retrieval (IIR) experiments the user's gaze motion on web pages is often recorded with eye tracking. The data is used to analyze gaze behavior or to identify Areas of Interest (AOI) the user has looked at. So far, tools for analyzing eye tracking data have certain limitations in supporting the analysis of gaze behavior in IIR experiments. Experiments often consist of a huge number of different visited web pages. In existing analysis tools the data can only be analyzed in videos or images and AOIs for every single web page have to be specified by hand, in a very time consuming process. In this work, we propose the reading protocol software which breaks eye tracking data down to the textual level by considering the HTML structure of the web pages. This has a lot of advantages for the analyst. First and foremost, it can easily be identified on a large scale what has actually been viewed and read on the stimuli pages by the subjects. Second, the web page structure can be used to filter to AOIs. Third, gaze data of multiple users can be presented on the same page, and fourth, fixation times on text can be exported and further processed in other tools. We present the software, its validation, and example use cases with data from three existing IIR experiments.

According to the cognitive viewpoint of information retrieval (IR) research, a search task can be conceptualized as a sequence of information seeking intentions which both motivate and are influenced by search behaviors. While the behavioral effects of task features have been thoroughly discussed in a large body of literature, how different information seeking intentions in query segments serve as bridges between task and Web search behavior still remains unexplored. To develop a more comprehensive, multi-level (i.e., task level, intention level, and behavior level) understanding of Web search, the authors analyzed intention and search behavior data collected from 693 query segments generated by 40 participants in a controlled lab setting, seeking to answer two main research questions: 1) from task to intention : how do different task features affect users' information seeking intentions at different stages of a search session? 2) from intention to behavior : How is a user's search behavior associated with their information seeking intentions in the current and next query segments respectively? The results demonstrate that: 1) Task features significantly affected the frequency of occurrence of most of the information seeking intentions, and these effects gradually faded away as search sessions proceeded; 2) The presences of a variety of intentions in both current and subsequent query segments were connected with and detectable by different subsets of behavioral measures. This study contributes to the understanding of the connections between task, intentions in query segments, and search behavior, and thereby has implications for designing system affordances for supporting different intentions and search activities in various task stages and contexts.

Expectation disconfirmation refers to the situation where a user's perceived performance of a system disconfirms her original expectation. Previous information systems studies have demonstrated that expectation disconfirmation can significantly affect a system user's behavior and experience. Inspired by this finding, we go beyond the traditional approach that focuses on the final post-search perception and study the expectation disconfirmation problem in Web search. Our study investigates task difficulty expectation disconfirmation and demonstrates that: (1) unexpectedly difficult task can significantly decrease a user's perceived level of search success and increase the perceived time pressure; (2) the size and direction of task difficulty expectation disconfirmation are significantly associated with Web search behavior; (3) it is possible to predict the state of expectation disconfirmation (especially the negative, unexpectedly difficult cases) based on search behavioral features. This study demonstrates the value of integrating expectation disconfirmation approach with interactive IR research and thus may encourage future researchers to further explore the effects of other aspects of users' expectations and post-search perceptions.

As understanding of web search behavior grows, researchers rapidly develop new study designs to capture and understand search behavior. Researchers have restricted time in which to design a study, develop a collection tool, collect data, analyze it, and report new insights. In particular, sufficient time and development skills are often required to create a tool that meets the needs of any particular web search behavior study. Coagmento is a tool that is developed for facilitating many of the needs for designing and running a lab study, from executing a session flow to collecting log data. By streamlining the programming of unique parts for a specific study, Coagmento helps researchers tailor various parts of running a user study, lowering the barrier for designing and conducting lab study experiments. One-click interactions with a graphical user interface permit researchers to operate through a web-based administrative service to generate stages, search tasks, and questionnaires for their interactive information retrieval studies. In this demonstration, Coagmento provides a solution to increase efficiency in the production of laboratory experiments for web search behavior.

In this research, we investigate the behavioral patterns exhibited in different search sessions as users attempt to complete search tasks of increasing cognitive complexity. The search tasks, which are exploratory in nature, have been designed using the Taxonomy of Educational Objectives, and are presented to the users hierarchically. We capture naturalistic search behavior of the users in real world (non-lab) setting using a Chrome browser plugin. The research analyzes the web log data of the users to assess if and how the web search behavior of the users changes over different search sessions. We also look at the different demographic factors like age and gender, educational factors like the academic background, read and write proficiency in English, and search skills to determine if these factors influence the web search behavior of the users. Our results indicate that search sessions have significant effects on the web search behavior of the users. Most of the web search behaviors differed significantly across search sessions. Of the secondary factors, gender showed significant effect on the query reformulations (measured using average edit distance between queries) and query length (measured using number of words per query) while year of study affected only the average query length. Search experience had significant effect on all the web search behaviors.

Interactive information retrieval (IIR) researchers often conduct laboratory studies to understand the relationship between people seeking information and information retrieval systems. They develop extensive data collection methods and tools create new understanding about the relationship between observable behaviors, searcher context, and underlying cognition, to better support people's information seeking. Yet aside from the problems of data size, realism, and demographics, laboratory studies are limited in the number and nature of phenomena they can study. Hence, data collected in laboratories contains different searcher populations and collects non-overlapping user and task characteristics. While research analyses and collection methods are isolated, how can we further IIR's mission of broad understanding? We approach this as a structure learning problem on incomplete data, determining the extent to which incomplete data can be used to predict user and task characteristics from interactions. In particular, we examine whether combining heterogeneous data sets is more effective than using a single data set alone in prediction. Our results indicate that adding external data significantly improves predictions of searcher characteristics, task characteristics, and behaviors, even when the data does not contain identical information about searchers.

Explainable recommendation and search attempt to develop models or methods that not only generate high-quality recommendation or search results, but also interpretability of the models or explanations of the results for users or system designers, which can help to improve the system transparency, persuasiveness, trustworthiness, and effectiveness, etc. This is even more important in personalized search and recommendation scenarios, where users would like to know why a particular product, web page, news report, or friend suggestion exists in his or her own search and recommendation lists. The workshop focuses on the research and application of explainable recommendation, search, and a broader scope of IR tasks. It will gather researchers as well as practitioners in the field for discussions, idea communications, and research promotions. It will also generate insightful debates about the recent regulations regarding AI interpretability, to a broader community including but not limited to IR, machine learning, AI, Data Science, and beyond.

Implicit detection of relevance has been approached by many during the last decade. From the use of individual measures to the use of multiple features from different sources (multimodality), studies have shown the feasibility to automatically detect whether a document is relevant. Despite promising results, it is not clear yet to what extent multimodality constitutes an effective approach compared to unimodality. In this article, we hypothesize that it is possible to build unimodal models capable of outperforming multimodal models in the detection of perceived relevance. To test this hypothesis, we conducted three experiments to compare unimodal and multimodal classification models built using a combination of 24 features. Our classification experiments showed that a univariate unimodal model based on the left‐click feature supports our hypothesis. On the other hand, our prediction experiment suggests that multimodality slightly improves early classification compared to the best unimodal models. Based on our results, we argue that the feasibility for practical applications of state‐of‐the‐art multimodal approaches may be strongly constrained by technology, cultural, ethical, and legal aspects, in which case unimodality may offer a better alternative today for supporting relevance detection in interactive information retrieval systems.

In this paper, we investigate the relationship between searching and learning, by conceptualizing information seeking as a learning process, and learning as an outcome of the information seeking process. We present the participants with four search tasks, each of them designed to represent different cognitive levels of learning. Through quantitative analysis of the participants» Web search logs, we examine how individual search behavior is influenced by different task complexity levels as we present the tasks in a hierarchical order. We also explore how the perceived learning outcomes and processes, and the different learning actions, are related to the levels of cognitive complexity. By analyzing the search logs, self-reports, interview data, and the reports, both quantitatively and qualitatively, we infer that searching and learning are not isolated but co-existing processes. Distinct search patterns and learning outcomes were observed in tasks of different cognitive complexities, and overlapping learning actions were observed for the different tasks.

When users are looking for information on the Web, they show different behavior for different task types, e.g., for fact finding vs. information gathering tasks. For example, related work in this area has investigated how this behavior can be measured and applied to distinguish between easy and difficult tasks. In this work, we look at the searcher's behavior in the domain of journalism for four different task types, and additionally, for two different topics in each task type. Search behavior is measured with a number of session variables and correlated to subjective measures such as task difficulty, task success and the usefulness of documents. We acknowledge prior results in this area that task difficulty is correlated to user effort and that easy and difficult tasks are distinguishable by session variables. However, in this work, we emphasize the role of the task topic - in and of itself - over parameters such as the search results and read content pages, dwell times, session variables and subjective measures such as task difficulty or task success. With this knowledge researchers should give more attention to the task topic as an important influence factor for user behavior.

We explore the gap between 1) statistically significant relationships between task and browsing behavior and 2) predicting task type from such behaviors. Previous literature has shown relationships between Web browsing behavior and person»s corresponding search task. We find statistically significant browser features for detecting task - comparing the features to previous literature - and apply this knowledge to task classification of search sessions. Even though significant features improve prediction over baselines, it is not by much. We suggest that a more subtle treatment of such features should go beyond statistical significance. In some cases, considering personal patterns may be required for effective prediction.

In this demo, we present Coagmento, a Web-based, open-source tool for information seeking projects that collects information for individuals and groups and helps facilitate collaborative information seeking. Coagmento has been used in information retrieval and human-computer interaction studies to investigate individual and group information seeking behaviors in a lab or a field setting. In this demo, we discuss what Coagmento is, its past uses in prior studies, and its present state. We also discuss current work in progress. With Coagmento recently passing its 10th anniversary, we discuss our intention to make it a tool that is easy to configure for a human information behavior researcher with little programming skill.

In this demo we present IRIS, an open-source framework that provides a set of simple and modular document operators that can be combined in various ways to create more interesting and advanced functionality otherwise unavailable during most information search sessions. Those functionalities include summarization, ranking, filtering and query. The goal is to support users looking for, collecting, and synthesizing information. The system is also easily extendable, allowing for customized functionality for users during information sessions and researchers studying higher levels of abstraction for information retrieval. The demo shows the front end interactions using a browser plug-in that offers new interactions with documents during search sessions, as well as the back-end components driving the system.

In an information seeking episode, users often look for sources in online and offline environments depending on the task at hand. However, at most times users consider factors such as ease, time taken to complete the task, and the number of sources to be consulted as the essential factors while fulfilling the information seeking task. In our study, we explore the role of different cost variables -- ease, time taken to complete the task, and the number of sources consulted -- as the factors to be explored based on different cognitive task complexity levels, from Bloom»s taxonomy, by conducting a user study. We study the different search behaviors shown by users in online and offline environments based on the different cognitive task complexity levels and the three cost variables. We observed intriguing results that show factors such as ease, time, and the number of sources play a role in source selection while completing the tasks. Our study is a novel proposition in that we explore research in the direction of source selection based on different cognitive task complexity levels. The findings will contribute to shaping how tasks should be designed to use sources in a helpful and convenient manner. Moreover, the results also advance our understanding of the role that different affordances play in online and offline search behavior.

People often have difficulty in expressing their information needs. Many times this results from a lack of clarity about the task at hand, or the way an information or search system works. In addition, people may not know what they do not know. The former is addressed by search systems by providing recommendations, whereas there are no good solutions for the latter problem. Even when a search system makes recommendations, they are limited to suggesting objects such as queries and documents only. They do not consider providing suggestions for strategies, people, or processes. This Perspective Paper addresses it by showing how to investigate the nature of the work a person is doing, predicting the potential problems they may encounter, and providing help to overcome those problems. This help could be an object such as a document or a query, a strategy, or a person. This whole process is referred to as Information Fostering. Beyond crafting a general-purpose recommender system, Information Fostering is the idea of providing proactive suggestions and help to information seekers. This could allow them avoid potential problems and capture promising opportunities from a search process before it is too late. The current paper presents this new perspective by outlining desired characteristics of an Information Fostering system, envisioning application scenarios, and proposing a set of potential methods for moving forward. Beyond these details, the primary purpose of this paper is to offer a new viewpoint that looks at the other side of the information seeking coin, by bringing together ideas from human-computer interaction, information retrieval, recommender systems, and education.

This paper examines how individuals judge the accuracy, adequacy, relevance, and trustworthiness of different types of impersonal and interpersonal information sources and how task type influences their evaluation process. 53 participants from diverse backgrounds recruited via Amazon»s Mechanical Turk performed four simulated information seeking tasks. This study analyzed the data collected from participants» self-reported information seeking experiences in online logbooks and follow-up semi-structured interviews with 23 participants by applying both qualitative and quantitative methods. The findings suggest that task type and information source type affect individuals» information quality judgment, and they perceive websites are more accurate than interpersonal sources, though the latter can be trustworthy. Moreover, their understanding of the type of information also affects their quality judgment. For example, they prefer factual information to opinions in some situations.

One of the emerging and important problems in Interactive Information Retrieval research is predicting search tasks. Given a searcher's behavior during a search session, can the searcher's task be predicted? Which aspects of the task can be predicted, and how quickly and how accurately can the prediction be made? Much past literature has examined relationships between browsing behavior and task type at a statistical level, and recent work is moving towards prediction. While one may think whole session measures are useful for prediction, recent findings on common measures have suggested the contrary. Can less of the session still be useful? We examine the opposite end: the first query. Using multiple data sets for comparison, our results suggest that first query measures can be at least as good as -- and sometimes better than -- whole session measures for certain task type predictions.

Community Question‐Answering (CQA) sites have become popular venues where people can ask questions, seek information, or share knowledge with a user community. Although responses on CQA sites are obviously slower than information retrieved by a search engine, one of the most frustrating aspects of CQAs occurs when an asker's posted question does not receive a reasonable answer or remains unanswered. CQA sites could improve users' experience by identifying potential answerers and routing appropriate questions to them. In this paper, we predict the potential answerers based on question content and user profiles. Our approach builds user profiles based on past activity. When a new question is posted, the proposed method computes scores between the question and all user profiles to find the potential answerers. We conduct extensive experimental evaluations on two popular CQA sites ‐ Yahoo! Answers and Stack Overflow ‐ to show the effectiveness of our algorithm. The results show that our technique is able to predict a small group of 1000 users from which at least one user will answer the question with a probability higher than 50% in both CQA sites. Further analysis indicates that topic interest and activity level can improve the correctness of our approach.

The workshop on the evaluation of collaborative information retrieval and seeking (ECol) is held in conjunction with the ACM SIGIR Conference on Human Information Interaction & Retrieval (CHIIR) in Oslo, Norway. To make the workshop active and the participant pro-active, we released datasets and tools so as to help researchers contributing to the formalization of evaluation frameworks for challenging collaborative tasks. The workshop is split into two parts. First, a presentation session. Then, the afternoon is devoted to group discussion addressing challenges of evaluating and designing models for social and collaborative search.

Community Question Answering (CQA) services have become popular methods to seek and share information. In CQA, users with an information need, or askers, post a question that community members can answer. This question-answering process allows both askers and answerers to learn through the exchange of information. CQA services have also been widely used in the education domain, as some of such services are designed specifically for students' information seeking. However, due to insufficient knowledge, lack of experience, and other reasons, students often struggle in producing quality or even appropriate content. This low quality production causes their content to be flagged or deleted, further discouraging them from participating in the CQA process and instigating a vicious cycle of bad users and bad content. In an effort to break this cycle, the work reported here focuses on identifying users whose postings demonstrate a high deletion rate with a presumption that the bad content is an indication of a struggling student rather than a malicious user. In this work, experiments are conducted on a large student-oriented online CQA community to understand struggling students' behaviors. A framework is proposed to find these users based solely on their activities. Finally, community feedback (i.e. human judgment) such as moderator evaluation or community votes for good content is used to detect these users in the early stages of their respective struggles. To evaluate this framework, we used data from Brainly, a large educational CQA service that is used in two different markets with more than 3.7 million users and 10.7 million answers. The results show that the human judgment feature identifies early-stage struggling users with high accuracy. Identifying these struggling users (students) could help educators to determine suitable ways to help their students instead of presuming them to be bad users and cutting them off from the community.

In an information seeking episode, attributes such as relevance, quality, and the nature of the information sought/obtained are directly related to the nature and the quality of the query or question that represents an information need. It is, therefore, imperative that we identify potential problems with such representation to make the information seeking outcome and the experience more successful. In this paper, we investigate the question quality for the educational community question answering (CQA) site Brainly by examining 2,000 questions, of which 1,000 were answered and 1,000 were unanswered. Two human assessors rated the quality of each question on a scale from 1-5 based on factors such as ambiguity, poor syntax, lack of information, complexity, inappropriateness, and inconsistency. We then identified different textual features that are needed to detect question quality. A logistic regression classifier was built to categorize question features based on the rating scale and textual features present in the question. The results show higher ROC curves for ambiguity, lack of information, inappropriateness, complexity and excessive information; and lower ROC values for poor syntax and inconsistency among the questions. The findings demonstrate that the classifier failed to perform when faced with ill-framed or inconsistent phrases in a question. The work described here presents a method for identifying high and low-quality questions, the knowledge of which could be instrumental in helping reformulate users' questions and present them to a system or a community for more successful processes and outcomes.

Information source selection is essential to individuals' information seeking behaviors. Existing studies have focused on the criteria seekers employ when choosing information sources, such as sources' accessibility and quality, as well as the contextual factors that shape a seeker's selection. However, existing findings are somewhat conflicting and lack in-depth understanding of the reasons behind individuals' choices. The study reported here invited 53 participants from diverse backgrounds to perform simulated information seeking tasks over a two-day period and to report their experiences and findings in an online logbook. Semi-structured interviews with 23 of them were also conducted in order to examine the issues that arose from the logbooks. Our preliminary findings present several factors associated with participants' choices between impersonal and interpersonal sources. While interpersonal sources are deemed to be more suitable in capturing the context of an information query and providing personalized information, impersonal sources are found to be more accessible under a time constraint while depicting no emotion towards sensitive issues.

To address the issue of data sparsity and cold-start in recommender system, social information (e.g., user-user trust links) has been introduced to complement rating data for improving the performances of traditional model-based recommendation techniques such as matrix factorization (MF) and Bayesian personalized ranking (BPR). Although effective, the utilization of the explicit user-user relationships extracted directly from such social information has three main limitations. First, it is difficult to obtain explicit and reliable social links. Only a small portion of users indicate explicitly their trusted friends in recommender systems. Second, the “cold-start” users are “cold” not only on rating but also on socializing. There is no significant amount of explicit social information that can be useful for “cold-start” users. Third, an active user can be socially connected with others who have different taste/preference. Direct usage of explicit social links may mislead recommendation. To address these issues, we propose to extract implicit and reliable social information from user feedbacks and identify top-k semantic friends for each user. We incorporate the top-k semantic friends information into MF and BPR frameworks to solve the problems of ratings prediction and items ranking, respectively. The experimental results on three real-world datasets show that our proposed approaches achieve better results than the state-of-the-art MF with explicit social links (with 3.0% improvement on RMSE), and social BPR (with 9.1% improvement on AUC).

It has been shown that people attempt to accomplish a variety of intentions during the course of an information seeking session, and there is reason to believe that these different information seeking intentions can benefit from system support tailored to each such intention. We address the problem of predicting the presence of such intentions during an information seeking session, through analysis of observable user search behaviors. We present results of a study of 40 participants, each working on two different journalism tasks, which investigated how their search behaviors could indicate their intentions. Using 725 query-segments captured from this study, we demonstrate that information seeking intentions can be predicted with a simple classification model using a linear combination of search behavior features that can be logged with a browser plug-in.

This investigation of new approaches to improving collaboration, user/librarian experiences, and sustainability for virtual reference services (VRS) reports findings from a grant project titled “Cyber Synergy: Seeking Sustainability between Virtual Reference and Social Q&A Sites” (Radford, Connaway, & Shah, –2014). In‐depth telephone interviews with 50 VRS librarians included questions on collaboration, referral practices, and attitudes toward Social Question and Answer (SQA) services using the Critical Incident Technique (Flanagan, ). The Community of Practice (CoP) (Wenger, ; Davies, ) framework was found to be a useful conceptualization for understanding VRS professionals' approaches to their work. Findings indicate that participants usually refer questions from outside of their area of expertise to other librarians, but occasionally refer them to nonlibrarian experts. These referrals are made possible because participants believe that other VRS librarians are qualified and willing collaborators. Barriers to collaboration include not knowing appropriate librarians/experts for referral, inability to verify credentials, and perceived unwillingness to collaborate. Facilitators to collaboration include knowledge of appropriate collaborators who are qualified and willingness to refer. Answers from SQA services were perceived as less objective and authoritative, but participants were open to collaborating with nonlibrarian experts with confirmation of professional expertise or extensive knowledge.

Query recommendation predominantly relies on search logs to use existing queries for recommendation, typically calculating query similarity metrics or transition probabilities from the log. While effective, such recommendations are limited to the queries, words, and phrases in the log. They hence do not recommend potentially useful, entirely novel queries. Recent query recommendation methods have proposed generating queries on a topical or thematic level, though current approaches are limited to generating single words. We propose a hybrid method for constructing multi-word queries in this generative sense. It uses Latent Dirichlet Allocation to generate a topic for exploration and skip-gram modeling to generate queries from the topic. According to additional evaluation metrics we present, our model improves diversity and has some room for improving relevance, yet offers an interesting avenue for query recommendation.

We present a method for extracting the self-reported intentions of users engaged in an information seeking episode. We recruited participants to conduct search sessions and subsequently asked them to self-report their intentions. A total of 27 users participated in a lab study, during which they worked on two search tasks. After each search session, participants indicated their intentions during that session while viewing a video replay. Results indicate that the set of search intentions provided to participants was sufficient to account for intentions in four journalism-related information seeking tasks: a copy editing task, interview preparation task, relationships task, and story pitch task. The results also suggest regular patterns in intentions that can be exploited for identification of task type as well as potential applications to personalization and recommendation during a search episode.

Online Q & A services are information sources where people identify their information need, formulate the need in natural language, and interact with one another to satisfy their needs. Even though in recent years online Q & A has considerably grown in popularity and impacted information‐seeking behaviors, we still lack knowledge about what motivates people to ask a question in online Q & A environments. Yahoo! Answers and WikiAnswers were selected as the test beds in the study, and a sequential mixed method employing an Internet‐based survey, a diary method, and interviews was used to investigate user motivations for asking a question in online Q & A services. Cognitive needs were found as the most significant motivation, driving people to ask a question. Yet, it was found that other motivational factors (e.g., tension free needs) also played an important role in user motivations for asking a question, depending on asker's contexts and situations. Understanding motivations for asking a question could provide a general framework of conceptualizing different contexts and situations of information needs in online Q & A . The findings have several implications not only for developing better question‐answering processes in online Q & A environments, but also for gaining insights into the broader understanding of online information‐seeking behaviors.

Most information retrieval (IR) systems consider relevance, usefulness, and quality of information objects (documents, queries) for evaluation, prediction, and recommendation, often ignoring the underlying search process of information seeking. This may leave out opportunities for making recommendations that analyze the search process and/or recommend alternative search process instead of objects. To overcome this limitation, we investigated whether by analyzing a searcher's current processes we could forecast his likelihood of achieving a certain level of success with respect to search performance in the future. We propose a machine‐learning‐based method to dynamically evaluate and predict search performance several time‐steps ahead at each given time point of the search process during an exploratory search task. Our prediction method uses a collection of features extracted from expression of information need and coverage of information. For testing, we used log data collected from 4 user studies that included 216 users (96 individuals and 60 pairs). Our results show 80–90% accuracy in prediction depending on the number of time‐steps ahead. In effect, the work reported here provides a framework for evaluating search processes during exploratory search tasks and predicting search performance. Importantly, the proposed approach is based on user processes and is independent of any IR system.

In explicit collaborative search, two or more individuals coordinate their efforts toward a shared goal. Every day, Internet users with similar information needs have the potential to collaborate. However, online search is typically performed in solitude. Existing search systems do not promote explicit collaborations, and collaboration opportunities (collabportunities) are missed. In this article, we describe a method to evaluate the feasibility of transforming these collabportunities into recommendations for explicit collaboration. We developed a technique called pseudocollaboration to evaluate the benefits and costs of collabportunities through simulations. We evaluate the performance of our method using three data sets: (a) data from single users' search sessions, (b) data with collaborative search sessions between pairs of searchers, and (c) logs from a large‐scale search engine with search sessions of thousands of searchers. Our results establish when and how collabportunities would significantly help or hinder the search process versus searches conducted individually. The method that we describe has implications for the design and implementation of recommendation systems for explicit collaboration. It also connects system‐mediated and user‐mediated collaborative search, whereby the system evaluates the likely benefits of collaborating for a search task and helps searchers make more informed decisions on initiating and executing such a collaboration.

Personalization of support for information seeking depends crucially on the information retrieval system's knowledge of the task that led the person to engage in information seeking. Users work during information search sessions to satisfy their task goals, and their activity is not random. To what degree are there patterns in the user activity during information search sessions? Do activity patterns reflect the user's situation as the user moves through the search task under the influence of his or her task goal? Do these patterns reflect aspects of different types of information-seeking tasks? Could such activity patterns identify contexts within which information seeking takes place? To investigate these questions, we model sequences of user behaviors in two independent user studies of information search sessions (N = 32 users, 128 sessions, and N = 40 users, 160 sessions). Two representations of user activity patterns are used. One is based on the sequences of page use; the other is based on a cognitive representation of information acquisition derived from eye movement patterns in service of the reading process. One of the user studies considered journalism work tasks; the other concerned background research in genomics using search tasks taken from the TREC Genomics Track. The search tasks differed in basic dimensions of complexity, specificity, and the type of information product (intellectual or factual) needed to achieve the overall task goal. The results show that similar patterns of user activity are observed at both the cognitive and page use levels. The activity patterns at both representation layers are able to distinguish between task types in similar ways and, to some degree, between tasks of different levels of difficulty. We explore relationships between the results and task difficulty and discuss the use of activity patterns to explore events within a search session. User activity patterns can be at least partially observed in server-side search logs. A focus on patterns of user activity sequences may contribute to the development of information systems that better personalize the user's search experience.

Can the activity patterns of page use during information search sessions discriminate between different types of information seeking tasks? We model sequences of interactions with search result and content pages during information search sessions. Two representations are created: the sequences of page use and a cognitive representation of page interactions. The cognitive representation is based on logged eye movement patterns of textual information acquisition via the reading process. Page sequence actions from task sessions (n=109) in a user study are analyzed. The study tasks differed from one another in basic dimensions of complexity, specificity,level, and the type of information product (intellectual or factual). The results show that differences in task types can be measured at both the level of observations of page type sequences and at the level of cognitive activity on the pages. We discuss the implications for personalization of search systems, measurement of task similarity and the development of user-centered information systems that can support the user's current and expected search intentions.

Most of the previous approaches surrounding collaborative information retrieval (CIR) provide either a user-based mediation, in which the system only supports users' collaborative activities, or a system-based mediation, in which the system plays an active part in balancing user roles, re-ranking results, and distributing them to optimize overall retrieval performance. In this paper, we propose to combine both of these approaches by a role mining methodology that learns from users' actions about the retrieval strategy they adapt. This hybrid method aims at showing how users are different and how to use these differences for suggesting roles. The core of the method is expressed as an algorithm that (1) monitors users' actions in a CIR setting; (2) discovers differences among the collaborators along certain dimensions; and (3) suggests appropriate roles to make the most out of individual skills and optimize IR performance. Our approach is empirically evaluated and relies on two different laboratory studies involving 70 pairs of users. Our experiments show promising results that highlight how role mining could optimize the collaboration within a search session. The contributions of this work include a new algorithm for mining user roles in collaborative IR, an evaluation methodology, and a new approach to improve IR performance with the operationalization of user-driven system-mediated collaboration.

The notions that information seeking is not always a solitary activity and that people working in collaboration for information intensive tasks should be studied and supported have become more prevalent in recent years. Several new research questions, methodologies, and systems have emerged around these notions that may prove to be useful beyond the field of collaborative information seeking (CIS), with relevance to the broader area of information seeking and behavior. This article provides an overview of such key research work from a variety of domains, including library and information science, computer‐supported cooperative work, human‐computer interaction, and information retrieval. It starts with explanations of collaboration and how CIS fits in different contexts, emphasizing the interactive, intentional, and mutually beneficial nature of CIS activities. Relations to similar and related fields such as collaborative information retrieval, collaborative information behavior, and collaborative filtering are also clarified. Next, the article presents a synthesis of various frameworks and models that exist in the field today, along with a new synthesis of 12 different dimensions of group activities. A discussion on issues and approaches relating to evaluating various parameters in CIS follows. Finally, a list of known issues and challenges is presented to provide an overview of research opportunities in this field.

Communication and coordination are considered essential components of successful collaborations, and provision of awareness is a highly valuable feature of a collaborative information seeking (CIS) system. In this article, we investigate how providing different kinds of awareness support affects people's coordination behavior in a CIS task, as reflected by the communication that took place between them. We describe a laboratory study with 84 participants in 42 pairs with an experimental CIS system. These participants were brought to the laboratory for two separate sessions and given two exploratory search tasks. They were randomly assigned to one of the three systems, defined by the kind of awareness support provided. We analyzed the messages exchanged between the participants of each team by coding them for their coordination attributes. With this coding, we show how the participants employed different kinds of coordination during the study. Using qualitative and quantitative analyses, we demonstrate that the teams with no awareness, or with only personal awareness support, needed to spend more time and effort doing coordination than those with proper group awareness support. We argue that appropriate and adequate awareness support is essential for a CIS system for reducing coordination costs and keeping the collaborators well coordinated for a productive collaboration. The findings have implications for system designers as well as cognitive scientists and CIS researchers in general.

The assumption of information seekers being independent and IR problem being individual has been challenged often in the recent past, with an argument that the next big leap in search and retrieval will come through incorporating social and collaborative aspects of information seeking. This half-day tutorial will introduce the student to theories, methodologies, and tools that focus on information retrieval/seeking in collaboration. The student will have an opportunity to learn about the social aspect of IR with a focus on collaborative information seeking (CIS) situations, systems, and evaluation techniques. The course is intended for those interested in social and collaborative aspects of IR (from both academia and industry), and requires only a general understanding of IR systems and evaluation.

The course will introduce the student to theories, methodologies, and tools that focus on information retrieval/seeking in collaboration. The student will have an opportunity to learn about the social aspect of IR with a focus on collaborative information seeking (CIS) situations, systems, and evaluation techniques. Traditionally, IR is considered an individual pursuit, and not surprisingly, the majority of tools, techniques, and models developed for addressing information need, retrieval, and usage have focused on single users. The assumption of information seekers being independent and IR problem being individual has been challenged often in the recent past. This course will introduce such works to the students, with an emphasis on understanding models and systems that support collaborative search or browsing. In addition, the course will provide samples of data collected through several experiments to demonstrate various mining and analysis techniques.

Online question‐answering (Q&A) services are becoming increasingly popular among information seekers. We divide them into two categories, social Q&A (SQA) and virtual reference (VR), and examine how experts (librarians) and end users (students) evaluate information within both categories. To accomplish this, we first performed an extensive literature review and compiled a list of the aspects found to contribute to a “good” answer. These aspects were divided among three high‐level concepts: relevance, quality, and satisfaction. We then interviewed both experts and users, asking them first to reflect on their online Q&A experiences and then comment on our list of aspects. These interviews uncovered two main disparities. One disparity was found between users’ expectations with these services and how information was actually delivered among them, and the other disparity between the perceptions of users and experts with regard to the aforementioned three characteristics of relevance, quality, and satisfaction. Using qualitative analyses of both the interviews and relevant literature, we suggest ways to create better hybrid solutions for online Q&A and to bridge the gap between experts’ and users’ understandings of relevance, quality, and satisfaction, as well as the perceived importance of each in contributing to a good answer.

It is typically expected that when people work together, they can often accomplish goals that are difficult or even impossible for individuals. We consider this notion of the group achieving more than the sum of all individuals' achievements to be the synergic effect in collaboration. Similar expectation exists for people working in collaboration for information seeking tasks. We, however, lack a methodology and appropriate evaluation metrics for studying and measuring the synergic effect. In this paper we demonstrate how to evaluate this effect and discuss what it means to various collaborative information seeking (CIS) situations. We present a user study with four different conditions: single user, pair of users at the same computer, pair of users at different computers and co-located, and pair of users remotely located. Each of these individuals or pairs was given the same task of information seeking and usage for the same amount of time. We then combined the outputs of single independent users to form artificial pairs, and compared against the real pairs. Not surprisingly, participants using different computers (co-located or remotely located) were able to cover more information sources than those using a single computer (single user or a pair). But more interestingly, we found that real pairs with their own computers (co-located or remotely located) were able to cover more unique and useful information than that of the artificially created pairs. This indicates that those working in collaboration achieved something greater and better than what could be achieved by adding independent users, thus, demonstrating the synergic effect. Remotely located real teams were also able to formulate a wider range of queries than those pairs that were co-located or artificially created. This shows that the collaborators working remotely were able to achieve synergy while still being able to think and work independently. Through the experiments and measurements presented here, we have also contributed a unique methodology and an evaluation metric for CIS.

Question answering (QA) helps one go beyond traditional keywords-based querying and retrieve information in more precise form than given by a document or a list of documents. Several community-based QA (CQA) services have emerged allowing information seekers pose their information need as questions and receive answers from their fellow users. A question may receive multiple answers from multiple users and the asker or the community can choose the best answer. While the asker can thus indicate if he was satisfied with the information he received, there is no clear way of evaluating the quality of that information. We present a study to evaluate and predict the quality of an answer in a CQA setting. We chose Yahoo! Answers as such CQA service and selected a small set of questions, each with at least five answers. We asked Amazon Mechanical Turk workers to rate the quality of each answer for a given question based on 13 different criteria. Each answer was rated by five different workers. We then matched their assessments with the actual asker’s rating of a given answer. We show that the quality criteria we used faithfully match with asker’s perception of a quality answer. We furthered our investigation by extracting various features from questions, answers, and the users who posted them, and training a number of classifiers to select the best answer using those features. We demonstrate a high predictability of our trained models along with the relative merits of each of the features for such prediction. These models support our argument that in case of CQA, contextual information such as a user’s profile, can be critical in evaluating and predicting content quality.

Abstract Support for explicit collaboration in information‐seeking activities is increasingly recognized as a desideratum for search systems. Several tools have emerged recently that help groups of people with the same information‐seeking goals to work together. Many issues for these collaborative information‐seeking (CIS) environments remain understudied. The authors identified awareness as one of these issues in CIS, and they presented a user study that involved 42 pairs of participants, who worked in collaboration over 2 sessions with 3 instances of the authors' CIS system for exploratory search. They showed that while having awareness of personal actions and history is important for exploratory search tasks spanning multiple sessions, support for group awareness is even more significant for effective collaboration. In addition, they showed that support for such group awareness can be provided without compromising usability or introducing additional load on the users.

Information retrieval research has demonstrated that system performance does not always correlate positively with user performance, and that users often assign positive evaluation scores to search systems even when they are unable to complete tasks successfully. This research investigated the relationship between objective measures of system performance and users' perceptions of that performance. In this study, subjects evaluated the performance of four search systems whose search results were manipulated systematically to produce different orderings and numbers of relevant documents. Three laboratory studies were conducted with a total of eighty-one subjects. The first two studies investigated the effect of the order of five relevant and five nonrelevant documents in a search results list containing ten results on subjects' evaluations. The third study investigated the effect of varying the number of relevant documents in a search results list containing ten results on subjects' evaluations. Results demonstrate linear relationships between subjects' evaluations and the position of relevant documents in a search results list and the total number of relevant documents retrieved. Of the two, number of relevant documents retrieved was a stronger predictor of subjects' evaluation ratings and resulted in subjects using a greater range of evaluation scores.

We describe a new approach to information retrieval: algorithmic mediation for intentional, synchronous collaborative exploratory search. Using our system, two or more users with a common information need search together, simultaneously. The collaborative system provides tools, user interfaces and, most importantly, algorithmically-mediated retrieval to focus, enhance and augment the team's search and communication activities. Collaborative search outperformed post hoc merging of similarly instrumented single user runs. Algorithmic mediation improved both collaborative search (allowing a team of searchers to find relevant information more efficiently and effectively), and exploratory search (allowing the searchers to find relevant information that cannot be found while working individually).

No abstract available.

We describe a preliminary analysis of queries created by 81 users for 4 topics from the TREC Robust Track. Our goal was to explore the potential benefits of using queries created by multiple users on retrieval performance for difficult topics. We first examine the overlap in users' queries and the overlap in results with respect to different queries for the same topic. We then explore the potential benefits of combining users' queries in various ways. Our results provide some evidence that having access to multiple users' queries can improve retrieval for individual searchers and for difficult topics.

Using Saracevic's relevance types, we explore approaches to combining algorithm and cognitive relevance in a term relevance feedback scenario. Data collected from 21 users who provided relevance feedback about terms suggested by a system for 50 TREC HARD topics are used. The former type of feedback is considered as cognitive relevance and the latter type is considered as algorithm relevance. We construct retrieval runs using these two types of relevance feedback and experiment with ways of combining them with simple Boolean operators. Results show minimal differences in performance with respect to the different techniques.

No abstract available.

Several information organization, access, and filtering systems can benefit from different kind of document representations than those used in traditional Information Retrieval (IR). Topic Detection and Tracking (TDT) is an example of such an application. In this paper we demonstrate that named entities serve as better choices of units for document representation over all words. In order to test this hypothesis we study the effect of words-based and entity-based representations on Story Link Detection (SLD) - a core task in TDT research. The experiments on TDT corpora show that entity-based representations give significant improvements for SLD. We also propose a mechanism to expand the set of named entities used for document representation, which enhances the performance in some cases. We then take a step further and analyze the limitations of using only named entities for the document representation. Our studies and experiments indicate that adding additional topical terms can help in addressing such limitations.

ABSTRACT Although information retrieval research has always been concernedwith improving the effectiveness of search, in some applications, such as information analysis, a more specific requirement exists forhigh accuracy retrieval. This means that achieving high precision in the top document ranks is paramount. In this paper we presentwork aimed at achieving high accuracy in ad-hoc document retrieval by incorporating approaches from question answering (QA).We focus on getting the first relevant result as high as possible in the ranked list and argue that traditional precision and recall are notappropriate measures for evaluating this task. We instead use the mean reciprocal rank (MRR) of the first relevant result. We eval-uate three different methods for modifying queries to achieve high accuracy. The experiments done on TREC data provide support forthe approach of using MRR and incorporating QA techniques for getting high accuracy in ad-hoc retrieval task. Categories and Subject Descriptors H.3.4 [Information Storage and Retrieval]: Systems and Soft-ware--Performance evaluation (efficiency and effectiveness) ; H.3.3 [Information Storage and Retrieval]: Information Search andRetrieval--Query formulation