Recent work has identified that distillation can be used to create vector quantization based ANN indexes by learning the inverted file index and product quantization. The argued advantage of using a fixed teacher encoder for queries and documents is that the scores produced by the teacher can be used instead of the label judgements that are required when using traditional supervised learning, such as contrastive learning. However, current work only distills the teacher encoder outputs of dot products between quantized query embedddings and product quantized document embeddings. Our work combines the benefits of contrastive learning and distillation by using contrastive distillation whereby the teacher outputs contrastive scores that the student learns from. Our experimental results on MSMARCO passage retrieval and NQ open question answering datasets show that contrastive distillation improves over current state of the art for vector quantized dense retrieval.

Selecting an appropriate and relevant context forms an essential component for the efficacy of several information retrieval applications like Question Answering (QA) systems. The problem of Answer Sentence Selection (AS2) refers to the task of selecting sentences, from a larger text, that are relevant and contain the answer to users' queries. While there has been a lot of success in building AS2 systems trained on open-domain data (e.g., SQuAD, NQ), they do not generalize well in closed-domain settings, since domain adaptation can be challenging due to poor availability and annotation expense of domain-specific data. This paper proposes SEDAN, an effective self-learning framework to adapt AS2 models for domain-specific applications. We leverage large pre-trained language models to automatically generate domain-specific QA pairs for domain adaptation. We further fine-tune a pre-trained Sentence-BERT architecture to capture semantic relatedness between questions and answer sentences for AS2. Extensive experiments demonstrate the effectiveness of our proposed approach (over existing state-of-the-art AS2 baselines) on different Question Answering benchmark datasets.

Question Answering (QA) has become a foundational research area in Natural Language Understanding (NLU) with widespread applications in search, personal digital assistance, and conversational systems. Despite the success in open-domain question answering, existing extractive question answering models pre-trained using Wikipedia articles (e.g., SQuAD data) perform rather poorly in closed-domain and industrial scenarios. Further, a major limitation in adapting question answering systems to such contexts is the poor availability and the expensive annotation of domain-specific data. Thus, wide applicability of QA models are severely hampered in enterprise systems.In this paper, we aim to address the above challenges by introducing a novel QA framework, Qasar, using self-supervised learning for efficient domain adaptation. We show, for the first time, the advantage of fine-tuning pre-trained QA models for closed-domains by synthetically generated domain-specific questions and answers (from relevant documents) from large language models like T5. Further, we also propose a novel context retrieval component based on question-context semantic relatedness to further boost the accuracy of the Qasar QA framework. Experimental results show significant performance improvements on both open-and closed-domain QA datasets, while requiring no labelling efforts, which we believe will contribute to the ease of deployment of such systems in enterprise settings. The different modules of our framework (synthetic data generation, context retrieval, and question answering) can be fully reproduced by fine-tuning publicly available language models and QA models on SQuAD dataset as discussed in the paper.

With the advancements of Natural Language Understanding (NLU), diverse industrial applications like user intent classification, smart chatbots, sentiment analysis and question answering have be-come a primary paradigm. Transformers-based multi-lingual language models such as XLM have performed significantly well in diverse semantic understanding and classification tasks. However, fine-tuning such large pre-trained architectures is resource and compute intensive, limiting its wide adoption in enterprise environments.We present a novel efficient and light-weight frame-work based on sentence embeddings to obtain enhanced multi-lingual text representations for domain-specific NLU applications. Our framework combines the concepts of up-projection, alignment and meta-embeddings enhancing the textual semantic similarity knowledge of smaller sentence embedding architectures. Extensive experiments on diverse cross-lingual classification tasks showcase the proposed framework to be comparable to state-of-the-art large language models (in mono-lingual and zero-shot settings), even with lesser training and resource requirements.

Natural Language Understanding (NLU) has become a primary paradigm in enterprise settings for myriad industrial applications like user intent classification, smarter chatbots, sentiment analysis, and duplicate detection to name a few. With the advent of globalization, significant advancements have been recently achieved in transformers-based multi-lingual language models such as XLM and its variants for downstream multi-lingual sentence or short text classification tasks. However, fine-tuning such large pre-trained language models is highly resource-intensive as it assumes the adaptation of the full model, hampering its wide adoption in production grade applications due to the demanding computational and memory requirements.In this paper, we present a practical and efficient framework based on fusing various pre-trained sentence encoders leveraging the multi-lingual knowledge distillation approach. We demon-strate, for the first time, the practicality of utilizing such multi-lingual sentence embeddings for supervised learning tasks with a focus on sentence classification scenarios. We experimented our proposed framework on a wide range of open source classification datasets and exhibit very competitive performance compared to fine-tuning large pre-trained language models. We showcase that our light-weight framework provides the advantage of ease of training within minutes on a single CPU, competitive inference time, and robustness to parameter settings. In hope of facilitating and democratizing practical research focused on NLP, we are planning to release our code as well as a new pre-trained sentence embeddings for XLM-R-large model.

Approximate subgraph matching, an important primitive for many applications like question answering, community detection, and motif discovery, often involves large labeled graphs such as knowledge graphs, social networks, and protein sequences. Effective methods for extracting matching subgraphs, in terms of label and structural similarities to a query, should depict accuracy, computational efficiency, and robustness to noise. In this paper, we propose VerSaChI for finding the top-k most similar subgraphs based on 2-hop label and structural overlap similarity with the query. The similarity is characterized using Chebyshev's inequality to compute the chi-square statistical significance for measuring the degree of matching of the subgraphs. Experiments on real-life graph datasets showcase significant improvements in terms of accuracy compared to state-of-the-art methods, as well as robustness to noise.

Subgraph querying is one of the most important primitives in many applications. Although the field is well studied for deterministic graphs, in many situations, the graphs are probabilistic in nature. In this paper, we address the problem of subgraph querying in large probabilistic labeled graphs. We employ a novel algorithmic framework, called ChiSeL, that uses the idea of statistical significance for approximate subgraph matching on uncertain graphs that have uncertainty in edges. For each candidate matching vertex in the target graph that matches a query vertex, we compute its statistical significance using the chi-squared statistic. The search algorithm then proceeds in a greedy manner by exploring the vertex neighbors having the largest chi-square score. In addition to edge uncertainty, we also show how ChiSeL can handle uncertainty in labels and/or vertices. Experiments on large real-life graphs show the efficiency and effectiveness of our algorithm.

With the increasing impetus on globalization, workplace modernization, and employee welfare, modern organizations are focusing more and more resources in developing their pool of human capital. Employee attrition poses a major challenge for organizations - be it in terms of operational cost or loss of talent. We present CLARA (CLustering for Analysis and Remedial of Attrition), a deployed end-to-end system applying descriptive, predictive, and prescriptive analytics, providing as output actionable remedial actions to be used by HR departments to improve employee retention. We propose a coupling between clustering and frequent pattern based scoring measure to identify candidate employees at a high risk of attrition, and subsequently recommend suggestive actions to improve their retention. Using publicly available IBM human resource (HR) dataset, we show that CLARA demonstrates comparable performance (compared to state-of-the-art techniques) in identifying such “high risk” employees. We further validate the quality of the recommendations provided by the framework in reducing the overall rate of human capital loss, and discuss the real-life implementation of the framework within our organization. CLARA attains up to 65% precision, on IBM HR dataset, in predicting employee attrition and, thanks to the remedial actions, up to 22.5% reduction in the predictive attrition score for the top-5 identified employees.

The maximum clique extraction problem finds extensive application in diverse domains like community discovery in social networks, brain connectivity networks, motif discovery, gene expression in bioinformatics, anomaly detection, road networks and expert graphs. Since the problem is NP-hard, known algorithms for finding a maximum clique can be expensive for large real-life graphs. Current heuristics also fail to provide high accuracy and run-time efficiency for dense networks, quite common in the above domains. In this paper, we propose the ALTHEA heuristic to efficiently extract a maximum clique from a dense graph. We show that ALTHEA, based on chi-square statistical significance, is able to dramatically prune the search space for finding a maximum clique, thereby providing run-time efficiency. Further, experimental results on both real and synthetic graph datasets demonstrate that ALTHEA is highly accurate and robust in detecting a maximum clique.

With the advent of Big Data paradigm, filtering, retrieval, and linking of unstructured multi-modal data has become a necessity. Assigning topic labels to contents, that accurately capture the meaning and contextual information, is a fundamental problem in organizing unstructured data. The usage of manually-assigned tags for this purpose introduces inconsistencies because of different "surface forms". On the other hand, existing automated approaches either use hierarchical multi-label classification, or are unsupervised and rely on (undirected) graph measures leveraging taxonomies. While the former requires large training data set to learn the characteristics of each topic class, the latter lacks the flexibility to learn broad range of related topics and are less accurate.We propose a novel framework, ANNOTATE based on a small set of features and directed traversal of taxonomies to learn a broad spectrum of related topics using limited training data. We also show that our approach provides accurate labels for several domains without the need for re-training. For instance, the framework, trained on a small set of BBC news articles, exhibits close matches to user-generated tags for Quora documents. Experimental results, on the same model, for news classification and identifying aspects of Amazon product reviews, based on Amazon Mechanical Turk evaluation show our approach to be significantly better than state-of-the-art.We further present real-life case studies of our proposed framework for automatically tagging Quora posts, and topically segmenting, indexing and linking related YouTube videos (using our publicly available Chrome browser extension).

The rising need to harvest domain specific knowledge in several applications is largely limited by the ability to dynamically grow structured knowledge representations, due to the increasing emergence of new concepts and their semantic relationships with existing ones. Such enrichment of existing hierarchical knowledge sources with new information to better model the "changing world" presents two-fold challenges: (1) Detection of previously unknown entities or concepts, and (2) Insertion of the new concepts into the knowledge structure, respecting the semantic integrity of the created relationships. To this end we propose a novel framework, ETF, to enrich large-scale, generic taxonomies with new concepts from resources such as news and research publications. Our approach learns a high-dimensional embedding for the existing concepts of the taxonomy, as well as for the new concepts. During the insertion of a new concept, this embedding is used to identify semantically similar neighborhoods within the existing taxonomy. The potential parent-child relationships linking the new concepts to the existing ones are then predicted using a set of semantic and graph features. Extensive evaluation of ETF on large, real-world taxonomies of Wikipedia and WordNet showcase more than 5% F1-score improvements compared to state-of-the-art baselines. We further demonstrate that ETF can accurately categorize newly emerging concepts and question-answer pairs across different domains.

Labeled graphs provide a natural way of representing entities, relationships and structures within real datasets such as knowledge graphs and protein interactions. Applications such as question answering, semantic search, and motif discovery entail efficient approaches for subgraph matching involving both label and structural similarities. Given the NP-completeness of subgraph isomorphism and the presence of noise, approximate graph matching techniques are required to handle queries in a robust and real-time manner. This paper presents a novel technique to characterize the subgraph similarity based on statistical significance captured by chi-square statistic. The statistical significance model takes into account the background structure and label distribution in the neighborhood of vertices to obtain the best matching subgraph and, therefore, robustly handles partial label and structural mismatches. Based on the model, we propose two algorithms, VELSET and NAGA, that, given a query graph, return the top-k most similar subgraphs from a (large) database graph. While VELSET is more accurate and robust to noise, NAGA is faster and more applicable for scenarios with low label noise. Experiments on large real-life graph datasets depict significant improvements in terms of accuracy and running time in comparison to the state-of-the-art methods.

With increasing popularity of Netflix, Yahoo! Video, etc., interactive multimedia services such as video-on-demand (VoD) provide an interesting and rich field of research. The advent of smarter wireless devices has surged the need for such services through wireless connectivity. However, personalization of individual user needs, reducing latency, coupled with maintaining low operational costs provides a challenging problem. In this paper, we propose an efficient VoD system, for wireless mobile devices, based on a novel caching algorithm, Intelligent Network Caching Algorithm (INCA) using analytics-driven look ahead scheme for both prefetch and replacement policies to deliver higher performance. This enables enhanced Quality Of Experience (QoE) of users with limited infrastructural changes and low operational cost. Alongside, we develop theoretical formulation of the QoE optimization problem that lies at the intersection of MPC (Markov Predictive Control) and MDP (Markov Decision Process). Empirical analysis over realistic user video query logs demonstrate better cache hit rate and QoE with low prefetch bandwidth, compared to existing caching schemes.

The unparalleled growth and popularity of the Internet coupled with the advent of diverse modern applications such as search engines, on-line transactions, climate warning systems, etc., has catered to an unprecedented expanse in the volume of data stored world-wide. Efficient storage, management, and processing of such massively exponential amount of data has emerged as a central theme of research in this direction. Detection and removal of redundancies and duplicates in real-time from such multi-trillion record-set to bolster resource and compute efficiency constitutes a challenging area of study. The infeasibility of storing the entire data from potentially unbounded data streams, with the need for precise elimination of duplicates calls for intelligent approximate duplicate detection algorithms. The literature hosts numerous works based on the well-known probabilistic bitmap structure, Bloom Filter and its variants. In this paper we propose a novel data structure, Streaming Quotient Filter, (SQF) for efficient detection and removal of duplicates in data streams. SQF intelligently stores the signatures of elements arriving on a data stream, and along with an eviction policy provides near zero false positive and false negative rates. We show that the near optimal performance of SQF is achieved with a very low memory requirement, making it ideal for real-time memory-efficient de-duplication applications having an extremely low false positive and false negative tolerance rates. We present detailed theoretical analysis of the working of SQF, providing a guarantee on its performance. Empirically, we compare SQF to alternate methods and show that the proposed method is superior in terms of memory and accuracy compared to the existing solutions. We also discuss Dynamic SQF for evolving streams and the parallel implementation of SQF.

With the explosion of information stored world-wide, data intensive computing has emerged as a central area of research. Efficient management and processing of this massively exponential amount of data from diverse sources, such as telecommunication call data records, telescope imagery, online transaction records, web pages, stock markets, medical records (monitoring critical health conditions of patients), climate warning systems, etc., has become a necessity. Removing redundancy from such huge (multi-billion records) datasets results in resource and compute efficiency for downstream processing and constitutes an important area of study. "Intelligent compression" or deduplication in streaming scenarios, for precise identification and elimination of duplicates from the unbounded data stream is a greater challenge given the real-time nature of data arrival. Stable Bloom Filters (SBF) [13] address this problem to a certain extent. However, SBF suffers from a high false negative rate and slow convergence rate, thereby rendering it inefficient for applications with low false negative rate tolerance.