Preference judgments have been established as an effective method for offline evaluation of information retrieval systems with advantages to graded or binary relevance judgments. Graded judgments assign each document a pre-defined grade level, while preference judgments involve assessing a pair of items presented side by side and indicating which is better. However, leveraging preference judgments may require a more extensive number of judgments, and there are limitations in terms of evaluation measures. In this study, we present a new preference judgment tool called JUDGO, designed for expert assessors and researchers. The tool is supported by a new heap-like preference judgment algorithm that assumes transitivity and allows for ties. An earlier version of the tool was employed by NIST to determine up to the top-10 best items for each of the 38 topics for the TREC 2022 Health Misinformation track, with over 2,200 judgments collected. The current version has been applied in a separate research study to collect almost 10,000 judgments, with multiple assessors completing each topic. The code and resources are available at https://judgo-system.github.io.

Despite recent progress on conversational systems, they still do not perform smoothly when faced with ambiguous requests. When questions are unclear, conversational systems should have the ability to ask clarifying questions, rather than assuming a particular interpretation or simply responding that they do not understand. While the research community has paid substantial attention to the problem of predicting query ambiguity in traditional search contexts, researchers have paid relatively little attention to predicting when this ambiguity is sufficient to warrant clarification in the context of conversational systems. In this paper, we propose an unsupervised method for predicting the need for clarification. This method is based on the measured coherency of results from an initial answer retrieval step, under the assumption that a less ambiguous query is more likely to retrieve more coherent results when compared to an ambiguous query. We build a graph from retrieved items based on their context similarity, treating measures of graph connectivity as indicators of ambiguity. We evaluate our approach on two open-domain conversational question answering datasets, ClariQ and AmbigNQ, comparing it with neural and non-neural baselines. Our unsupervised approach performs as well as supervised approaches while providing better generalization.

One of the most interesting topics in the scope of social network analysis is dynamic community detection, keeping track of communities' evolutions in a dynamic network. This article introduces a new Louvain-based dynamic community detection algorithm relied on the derived knowledge of the previous steps of the network evolution. The algorithm builds a compressed graph, where its supernodes represent the detected communities of the previous step and its superedges show the edges among the supernodes. The algorithm not only constructs the compressed graph with low computational complexity but also detects the communities through the integration of the Louvain algorithm into the graph. The efficiency of the proposed algorithms is widely investigated in this article. By doing so, several evaluations have been performed over three standard real-world data sets, namely Enron Email, Cit-HepTh, and Facebook data sets. The obtained results indicate the superiority of the proposed algorithm with respect to the execution time as an efficiency metric. Likewise, the results show the modularity of the proposed algorithm as another effectiveness metric compared with the other well-known related algorithms.