All pairs similarity search is the problem of finding all pairs of records that have a similarity score above the specified threshold. Many real-world systems like search engines, online social networks, and digital libraries frequently have to solve this problem for datasets having millions of records in a high dimensional space, which are often sparse. The challenge is to design algorithms with feasible time requirements. To meet this challenge, algorithms have been proposed based on the inverted index, which maps each dimension to a list of records with non-zero projection along that dimension. Common to these algorithms is a three-phase framework of data preprocessing, pairs matching, and indexing. Matching is the most time-consuming phase. Within this framework, we propose fast matching technique that uses the sparse nature of real-world data to effectively reduce the size of the search space through a systematic set of tighter filtering conditions and heuristic optimizations. We integrate our technique with the fastest-to-date algorithm in the field and achieve up to 6.5X speed-up on three large real-world datasets.

We address the problem of handling topic oriented tasks on the World Wide Web. Our aim is to find most relevant and important pages for broad-topic queries while searching in a small set of candidate pages. We present a link analysis based algorithm SelHITS which is an improvement over Kleinberg's HITS algorithm. We introduce concept of virtual links to exploit latent information in the hyperlinked environment. Selective expansion of the root set and novel ranking strategy are the distinguishing features of our approach. Selective expansion method avoids topic drift and provides results consistent with only one interpretation of the query. Experimental evaluation and user feedback show that our algorithm indeed distills the most relevant and important pages for broad-topic queries. Trends in user feedback suggests that there exists a uniform notion of quality of search results within users

We address the problem of answering broad-topic queries on the World Wide Web. We present a link based analysis algorithm SelHITS, which is an improvement over Kleinberg's HITS [2] algorithm. We introduce the concept of virtual links to exploit the latent information in the hyperlinked environment. We propose a novel approach to calculate hub and authority values. We also present a selective expansion method which avoids topic drift and provides results consistent with only one interpretation of the query, even if the query is ambiguous. Initial experimental evaluation and user feedback show that our algorithm indeed distills the most important and relevant pages for broad-topic queries. We also infer that there exists a uniform notion of quality of search results within users.