In recent years, gathering relevance judgments through non-topic originators has become an increasingly important problem in Information Retrieval. Relevance judgments can be used to measure the effectiveness of a system, and are often needed to build supervised learning models in learning-to-rank retrieval systems. The two most popular approaches to gathering bronze level judgments - where the judge is not the originator of the information need for which relevance is being assessed, and is not a topic expert - is through a controlled user study, or through crowdsourcing. However, judging comes at a cost (in time, and usually money) and the quality of the judgments can vary widely. In this work, we directly compare the reliability of judgments using three different types of bronze assessor groups. Our first group is a controlled Lab group; the second and third are two different crowdsourcing groups, CF-Document where assessors were free to judge any number of documents for a topic, and CF-Topic where judges were required to judge all of the documents from a single topic, in a manner similar to the Lab group. Our study shows that Lab assessors exhibit a higher level of agreement with a set of ground truth judgments than CF-Topic and CF-Document assessors. Inter-rater agreement rates show analogous trends. These finding suggests that in the absence of ground truth data, agreement between assessors can be used to reliably gauge the quality of relevance judgments gathered from secondary assessors, and that controlled user studies are more likely to produce reliable judgments despite being more costly.

The increasing use of location-based services has raised many issues of decision support and resource allocation. A crucial problem is how to solve queries of Group k-Nearest Neighbour (GkNN). A typical example of a GkNN query is finding one or many nearest meeting places for a group of people. Existing methods mostly rely on a centralised base station. However, mobile P2P systems offer many benefits, including self-organization, fault-tolerance and load-balancing. In this study, we propose and evaluate a novel P2P algorithm focusing on GkNN queries, in which mobile query objects and static objects of interest are of two different categories. The algorithm is evaluated in the MiXiM simulation framework with both real and synthetic datasets. The results show the practical feasibility of the P2P approach for solving GkNN queries for mobile networks.

The increasing use of mobile communications has raised many issues of decision support and resource allocation. A crucial problem is how to solve queries of Reverse Nearest Neighbor (RNN). An RNN query returns all objects that consider the query object as their nearest neighbor. Existing methods mostly rely on a centralized base station. However, mobile P2P systems offer many benefits, including self-organization, fault-tolerance and load-balancing. In this study, we propose two P2P algorithms focusing on bichromatic RNN queries, in which mobile query objects and static objects of interest are of two different categories, based on a boundary polygon around the mobile query object. The Exhaustive Search Algorithm makes use of all information from the peers to aim at high accuracy rate while the Optimized Search Algorithm reduces the number of queried peers. The algorithms are evaluated in MiXiM simulation framework with a real dataset. The results show the practical feasibility of the P2P approach in solving bichromatic RNN queries for mobile networks.

Together with the advance of positioning and mobile technologies, P2P query processing has attracted a growing interest number of location-aware applications such as answering kNN queries in Mobile Ad-hoc Networks. It not only overcomes drawbacks of centralized paradigms; for example single point of failure and bottle-neck issues, but more importantly harnesses power of peers' collaboration. In this research, we propose a pure mobile P2P query processing with novel searching and validating algorithms to find kNNs without support from the base station. The simulation results show that our paradigm can make use of data sharing from peers to save energy consumption on querying at least six times compared with centralized and hybrid paradigms in a reasonable mean latency of processing time for networks with high density of moving objects.