In this work, we highlight some interesting challenges faced when trying to build a large-scale commercial IR-based chatbot, Ruuh, for an emerging market like India which has unique characteristics such as high linguistic and cultural diversity, large section of young population and the second largest mobile market in the world. We set out to build a "human-like" AI agent which aspires to become the trusted friend of every Indian youth. To meet this objective, we realised that we need to think beyond the utilitarian notion of merely generating "relevant" responses and enable the agent to comprehend and meet a wider range of user social needs, like expressing happiness when user's favourite team wins, sharing a cute comment on showing the pictures of the user's pet and so on. The agent should also be well-versed with the informal language of the urban Indian youth which often includes slang and code-mixing across two or more languages (English and their native language). Finally, in order to be their trusted friend, the agent has to communicate with respect without offending their sentiments and emotions. Some of the above objectives pose significant research challenges in the areas of NLP, IR and AI. We take the audience through our journey of how we tackled some of the above challenges while building a large-scale commercial IR-based conversational agent. Our attempts to solve some of the above challenges have also resulted in some interesting research contributions in the form of publications and patents in the above areas. Our chat-bot currently has more than 1M users who have engaged in more than 70M conversations.

Trivia is any fact about an entity which is interesting due to its unusualness, uniqueness or unexpectedness. Trivia could be successfully employed to promote user engagement in various product experiences featuring the given entity. A Knowledge Graph (KG) is a semantic network which encodes various facts about entities and their relationships. In this paper, we propose a novel approach called DBpedia Trivia Miner (DTM) to automatically mine trivia for entities of a given domain in KGs. The essence of DTM lies in learning an Interestingness Model (IM), for a given domain, from human annotated training data provided in the form of interesting facts from the KG. The IM thus learnt is applied to extract trivia for other entities of the same domain in the KG. We propose two different approaches for learning the IM - a) A Convolutional Neural Network (CNN) based approach and b) Fusion Based CNN (F-CNN) approach which combines both hand-crafted and CNN features. Experiments across two different domains - Bollywood Actors and Music Artists reveal that CNN automatically learns features which are relevant to the task and shows competitive performance relative to hand-crafted feature based baselines whereas F-CNN significantly improves the performance over the baseline approaches which use hand-crafted features alone. Overall, DTM achieves an F1 score of 0.81 and 0.65 in Bollywood Actors and Music Artists domains respectively.

Community Question Answering (cQA) services like Yahoo! Answers 1 , Baidu Zhidao 2 , Quora 3 , StackOverflow 4 etc. provide a platform for interaction with experts and help users to obtain precise and accurate answers to their questions.The time lag between the user posting a question and receiving its answer could be reduced by retrieving similar historic questions from the cQA archives.The main challenge in this task is the "lexicosyntactic" gap between the current and the previous questions.In this paper, we propose a novel approach called "Siamese Convolutional Neural Network for cQA (SCQA)" to find the semantic similarity between the current and the archived questions.SCQA consist of twin convolutional neural networks with shared parameters and a contrastive loss function joining them.

Community Question Answering (cQA) forums have become a popular medium for soliciting direct answers to specific questions of users from experts or other experienced users on a given topic. However, for a given question, users sometimes have to sift through a large number of low-quality or irrelevant answers to find out the answer which satisfies their information need. To alleviate this, the problem of Answer Quality Prediction (AQP) aims to predict the quality of an answer posted in response to a forum question. Current AQP systems either learn models using - a) various hand-crafted features (HCF) or b) Deep Learning (DL) techniques which automatically learn the required feature representations. In this paper, we propose a novel approach for AQP known as - “Deep Feature Fusion Network (DFFN)” which combines the advantages of both hand-crafted features and deep learning based systems. Given a question-answer pair along with its metadata, the DFFN architecture independently - a) learns features from the Deep Neural Network (DNN) and b) computes hand-crafted features using various external resources and then combines them using a fully connected neural network trained to predict the final answer quality. DFFN is end-end differentiable and trained as a single system. We propose two different DFFN architectures which vary mainly in the way they model the input question/answer pair - DFFN-CNN uses a Convolutional Neural Network (CNN) and DFFN-BLNA uses a Bi-directional LSTM with Neural Attention (BLNA). Both these proposed variants of DFFN (DFFN-CNN and DFFN-BLNA) achieve state-of-the-art performance on the standard SemEval-2015 and SemEval-2016 benchmark datasets and outperforms baseline approaches which individually employ either HCF or DL based techniques alone.

In this paper, we present a novel approach to Pseudo-Relevance Feedback (PRF) called Multilingual PRF (MultiPRF). The key idea is to harness multilinguality. Given a query in a language, we take the help of another language to ameliorate the well known problems of PRF, viz. (a) The expansion terms from PRF are primarily based on co-occurrence relationships with query terms, and thus other terms which are lexically and semantically related, such as morphological variants and synonyms, are not explicitly captured, and (b) PRF is quite sensitive to the quality of the initially retrieved top k documents and is thus not robust. In MultiPRF, given a query in language L1, it is translated into language L2 and PRF is performed on a collection in language L2 and the resultant feedback model is translated from L2 back into L1. The final feedback model is obtained by combining the translated model with the original feedback model of the query in L1.

Today, parallel corpus-based systems dominate the transliteration landscape. But the resource-scarce languages do not enjoy the luxury of large parallel transliteration corpus. For these languages, rule-based transliteration is the only viable option. In this article, we show that by properly harnessing the monolingual resources in conjunction with manually created rule base, one can achieve reasonable transliteration performance. We achieve this performance by exploiting the power of Character Sequence Modeling (CSM), which requires only monolingual resources. We present the results of our rule-based system for Hindi to English, English to Hindi, and Persian to English transliteration tasks. We also perform extrinsic evaluation of transliteration systems in the context of Cross Lingual Information Retrieval. Another important contribution of our work is to explain the widely varying accuracy numbers reported in transliteration literature, in terms of the entropy of the language pairs and the datasets involved.