Current methods for evaluating large language models (LLMs) typically focus on high-level tasks such as text generation, without targeting a particular AI application. This approach is not sufficient for evaluating LLMs for Responsible AI dimensions like fairness, since protected attributes that are highly relevant in one application may be less relevant in another. In this work, we construct a dataset that is driven by a real-world application (generate a plain-text product description, given a list of product features), parameterized by fairness attributes intersected with gendered adjectives and product categories, yielding a rich set of labeled prompts. We show how to use the data to identify quality, veracity, safety, and fairness gaps in LLMs, contributing a proposal for LLM evaluation paired with a concrete resource for the research community.

In this paper, we investigate the problem of quantifying fairness in Retrieval-Augmented Generation (RAG) systems, particularly for complex cognitive tasks that go beyond factual question-answering. While RAG systems have demonstrated effectiveness in information extraction tasks, their fairness implications for cognitively complex tasks - including ideation, content creation, and analytical reasoning - remain under-explored. We propose a novel evaluation framework that extends IR fairness metrics by incorporating centrality-based measures to account for influence of retrieved documents on generated output beyond ranking. Our framework evaluates RAG systems across various cognitive dimensions using two ranking approaches: lexical (BM25) and dense (BGE), and language models of varying sizes. Our findings provide insights into: (1) the propagation of fairness disparities from retrieval to generation phases, and (2) the variation in system performance across different cognitive dimensions.

Immersive technologies such as virtual reality (VR) and head-mounted displays (HMD) have seen increased adoption in recent years. In this work, we study two factors that influence users' experience when shopping in VR through voice queries: (1) context alignment of the search environment and (2) the level of detail on the Search Engine Results Page (SERP). To this end, we developed a search system for VR and conducted a within-subject exploratory study (N=18) to understand the impact of the two experimental conditions. Our results suggest that both context alignment and SERP are important factors for information-seeking in VR, which present unique opportunities and challenges. More specifically, based on our findings, we suggest that search systems for VR must be able to: (1) provide cues for information-seeking in both the VR environment and SERP, (2) distribute attention between the VR environment and the search interface, (3) reduce distractions in the VR environment and (4) provide a ''sense of control'' to search in the VR environment.

In this paper, we demonstrate the Information Interactions in Virtual Reality (IIVR) system designed and implemented to study how users interact with abstract information objects in immersive virtual environments in the context of information retrieval. Virtual reality displays are quickly growing as social and personal computing media, and understanding user interactions in these immersive environments is imperative. As a step towards effective information retrieval in such emerging platforms, our system is central to upcoming studies to observe how users engage in information triaging tasks in Virtual Reality (VR). In these studies, we will observe the effects of (1) information layouts and (2) types of interactions in VR. We believe this early system motivates researchers in understanding and designing meaningful interactions for future VR information retrieval applications.

We describe a Wizard of Oz (WoZ) system, a WebApp, which we use to study how a conversational search system should take the initiative when engaging with users during collaborative search. This system integrates directly into Slack, a chat-messaging platform where users will collaborate. Through our system, the Wizard plays the role of a conversational search system that can search for information, send relevant web results, and message users. In our research, we study three Wizard conditions: bot\_info, bot\_dialog, and bot\_task, which differ in terms of how the Wizard can intervene in a conversation. The intervention modes follow the mixed-initiative framework by Chu-Carroll and Brown~\citechu1997tracking and provide us a foundation to study system initiative for conversational search. In this paper, we describe our design decisions and technical details on how we implemented the system.

Popular messaging platforms such as Slack have given rise to thousands of applications (or bots) that users can engage with individually or as a group. In this paper, we study the use of searchbots (i.e., bots that perform specific types of searches) during collaborative information-seeking tasks mediated through Slack. We report on a user study in which 27 pairs of participants were exposed to three searchbot conditions (a within-subjects design). In the first condition, participants completed the task by searching independently and coordinating through Slack (no searchbot). In the second condition, participants could only search inside of Slack using the searchbot. In the third condition, participants could both search inside of Slack using the searchbot and outside of Slack using their own independent search interfaces. We investigate four research questions focusing on the influence of the searchbot condition on outcomes associated with: (RQ1) participants' levels of workload, (RQ2) collaborative awareness, (RQ3) experiences interacting with the searchbot, and (RQ4) search behaviors. Our results suggest opportunities and challenges in designing searchbots to support collaborative search. On one hand, access to the searchbot resulted in more collaborative awareness, ease of coordination, and fewer duplicated searches. On the other hand, forcing participants to share the querying environment resulted in fewer overall queries, fewer query refinements by individuals, and greater levels of effort. We discuss the implications of our findings for designing effective searchbots to support collaborative search.

Popular messaging platforms such as Slack have given rise to hundreds of chatbots that users can engage with individually or as a group. We present a Wizard of Oz study on the use of searchbots (i.e., chatbots that perform specific types of searches) during collaborative information-seeking tasks. Specifically, we study searchbots that intervene dynamically and compare between two intervention types: (1) the searchbot presents questions to users to gather the information it needs to produce results, and (2) the searchbot monitors the conversation among the collaborators, infers the necessary information, and then displays search results with no additional input from the users. We investigate three research questions: (RQ1) What is the effect of a searchbot (and its intervention type) on participants» collaborative experience' (RQ2) What is the effect of a searchbot»s intervention type on participants» perceptions about the searchbot and level of engagement with the searchbot' and (RQ3) What are participants» impressions of a dynamic searchbot? Our results suggest that dynamic searchbots can enhance users» collaborative experience and that the intervention type does not greatly affect users» perceptions and level of engagement. Participants» impressions of the searchbot suggest unique opportunities and challenges for future work.

The popularity of messaging platforms such as Slack has given rise to thousands of different chatbots that users can engage with individually or as a group. The proposed dissertation research will investigate the use of searchbots (i.e., chatbots that perform specific search operations) during collaborative information-seeking tasks. Specifically, we will address the following research goals. RG1: Our first research goal will be to investigate the use of searchbots in a collaborative search scenario. The goal of collaborative search is to develop systems that help two or more people collaborate synchronously or asynchronously on information-seeking tasks. Collaborative search systems such as SearchTogether~\cite{Morris2007}, Coagmento~\cite{shah2010coagmento}, CollabSearch~\cite{Yue2012}, and ResultsSpace~\cite{Capra2012} allow users to share information, communicate asynchronously or in real-time, and provide interactive visualizations that raise awareness of each user's search activities, allowing users to learn from each other's search strategies and avoid duplicating work. Prior research shows that while people often search in pairs and in larger groups, they do so without the use of specialized search tools and instead coordinate via "out-of-channel" communication tools such as email, text messaging, phone, and social media~\cite{morris2008survey,Morris2013}. Our first goal will be to investigate the use of searchbots during real-time collaborative search tasks. Our interest in the use of searchbots for collaborative search echoes a suggestion made by Morris~\cite{Morris2013} to develop lightweight collaborative search tools over existing communication platforms.