Task-Oriented Dialogue Systems (TODS) have been widely deployed for domain specific virtual assistants at contact centres to route customers' calls or deliver information needs of the customer in a conversational interaction. TODS employ natural language understanding components in order to map user commands to a set of pre-defined intents. However, Contact Centre users often fail to formulate their complex information needs in a single utterance which leads to formulating ambiguous user commands. This can negatively impact intent classification, and consequently customer satisfaction. To avoid feeding ambiguous user commands to the intent classifier of virtual assistants and help users in formulating their commands, we have implemented a solution that (1) identifies when a user is ambiguous and the virtual assistant should ask a clarification question, (2) disambiguates the user command and provides top-N most likely intents in a form of a clarification question. Our experimental result shows that our proposed intent disambiguation solution has a statistically significant improvement over a popularity based intent disambiguation model and an Intent Ranking Model of the Natural Language Understanding engine for a virtual assistant of a contact centre in terms of intent disambiguation accuracy and Mean Reciprocal Rank.

The design, evaluation and adaptation of conversational information systems are typically guided by ratings from third-party, i.e. non-user, annotators. Interfaces used in gathering such ratings are designed in an ad-hoc fashion as it has not yet been investigated which design yields high-quality ratings. This work describes how to design user interfaces for gathering high-quality ratings with third-party annotators. In a user study, we compare a base interface that consolidates best practices from literature, an interface with clear definitions and an interface in which tasks are separated visually. We find that these interfaces yield annotations of high quality and separation of tasks. We find no significant improvements in quality between UIs. This work can serve as a starting point for researchers and practitioners interested in collecting high-quality dialogue user satisfaction ratings using third-party annotators.

The Web contains many APIs that could be combined in countless ways to enable Intelligent Assistants to complete all sorts of tasks. We propose a method to automatically produce task completion flows from a collection of these APIs by combining them in a graph and automatically extracting paths from the graph for task completion. These paths chain together API calls and use the output of executed APIs as inputs to others. We automatically extract these paths from an API graph in response to a user query and then rank the paths by the likelihood of them leading to user satisfaction. We apply our approach for task completion in the email and calendar domains and show how it can be used to automatically create task completion flows.

Intelligent assistants are increasingly being used on smart speaker devices, such as Amazon Echo, Google Home, Apple Homepod, and Harmon Kardon Invoke with Cortana. Typically, user satisfaction measurement relies on user interaction signals, such as clicks and scroll movements, in order to determine if a user was satisfied. However, these signals do not exist for smart speakers, which creates a challenge for user satisfaction evaluation on these devices. In this paper, we propose a new signal, user intent, as a means to measure user satisfaction. We propose to use this signal to model user satisfaction in two ways: 1) by developing intent sensitive word embeddings and then using sequences of these intent sensitive query representations to measure user satisfaction; 2) by representing a user's interactions with a smart speaker as a sequence of user intents and thus using this sequence to identify user satisfaction. Our experimental results indicate that our proposed user satisfaction models based on the intent-sensitive query representations have statistically significant improvements over several baselines in terms of common classification evaluation metrics. In particular, our proposed task satisfaction prediction model based on intent-sensitive word embeddings has a 11.81% improvement over a generative model baseline and 6.63% improvement over a user satisfaction prediction model based on Skip-gram word embeddings in terms of the F1 metric. Our findings have implications for the evaluation of Intelligent Assistant systems.

Task and session identification is a key element of system evaluation and user behavior modeling in Intelligent Assistant (IA) systems. However, identifying task and sessions for IAs is challenging due to the multi-task nature of IAs and the differences in the ways they are used on different platforms, such as smart-phones, cars, and smart speakers. Furthermore, usage behavior may differ among users depending on their expertise with the system and the tasks they are interested in performing. In this study, we investigate how to identify tasks and sessions in IAs given these differences. To do this, we analyze data based on the interaction logs of two IAs integrated with smart-speakers. We fit Gaussian Mixture Models to estimate task and session boundaries and show how a model with 3 components models user interactivity time better than a model with 2 components. We then show how session boundaries differ for users depending on whether they are in a learning-phase or not. Finally, we study how user inter-activity times differs depending on the task that the user is trying to perform. Our findings show that there is no single task or session boundary that can be used for IA evaluation. Instead, these boundaries are influenced by the experience of the user and the task they are trying to perform. Our findings have implications for the study and evaluation of Intelligent Agent Systems.

Mobile devices and the internet of things blend our virtual online behavior with our actions in the real-world. The physical context creates numerous external factors that play a role in the user's online interactions, thus creating new external biases in the collected information interaction logs. Our general aim is to improve our understanding of onsite users' behavior, which allow us to create better online and onsite contextual suggestion systems. We focus on the cultural heritage domain and have collected onsite users' information interaction logs of visits in a museum. This prompts the question: How to understand users' behavior in order to be able to predict their onsite behaviors? Our main findings are the following: First, users behave differently in different onsite contextual situations. Second, there is a significant dependency between users' onsite behaviors and other users, who are interacting with next point of interests (POIs). Third, we have proposed a contextual Skip-Or-Stay behavior classifier based on four different pairwise contextual features that significantly improves a defined baseline based on all the considered evaluation metrics. Fourth, we have analyzed the importance of different contexts in the Skip-or-Stay behavior predictions.

There is a growing interest in using the Internet of Things (IoT) to create smart environments, which hold the promise to provide personalized experience based on the trail of user interactions with smart devices. We experiment with behavioral user models based on interactions with smart devices in a museum, and investigate the personalized recommendation of what to see after visiting an initial set of Point of Interests (POIs), a key problem in personalizing museum visits or tour guides. We have logged users' onsite physical information interactions of visits in a museum. Moreover, to have a better understanding of users' information interaction behaviors and their preferences, we have collected and studied query logs of a search engine of the same collection, and we have found similarities between users' online digital and onsite physical information interaction behaviors. We exploit user modeling based on users' different information interaction behaviors and experiment with a novel approach to a critical one-shot POI recommendation using deep neural multilayer perceptron based on explicitly given users' contextual information, and set-based extracted features using users' physical information interaction behaviors and similar users' digital information interaction behaviors. Experimental results indicates that our proposed behavioral user modeling, using both physical and digital user information interaction behaviors, improves the onsite POI recommendation baselines' performances in all common Information Retrieval evaluation metrics. Our proposed approach provides an effective way to achieve a high precision at rank 1 in onsite critical one-shot POI recommendation problem.

The existence of different biases in logged users' behavior makes it difficult to extract realistic topical and social information from users' interaction logs (e.g., query logs). To understand users' behavior and their interests in the cultural heritage domain, we have logged onsite user interaction logs of visits in a museum. This prompts the question on the reliability of the social information being gathered from the onsite logs: How does the position of museum objects affect users' behavior in the museum? How does order of visiting point of interests affect their dwell-time in front of each point of interest? How do different users' characteristics affect their behavior in the museum? In short, what are different kinds of biases that should be considered in the onsite logs? Our main findings are the following: First, there is a considerable position bias, which is due to the design of the exhibition and should be considered during extraction of social signals from the log. Second, there is a bias in the amount of time that users spend for interacting with the point of interests and the order of picking them to visit. This shows a fatigue on users' interactions while they are reaching to the end of the exhibition. Third, we find out some variations among the users' visit, which shows context is an important factor to consider while using onsite logs for different purposes.

Creating test collections for modern search tasks is increasingly more challenging due to the growing scale and dynamic nature of content, and need for richer contextualization of the statements of request. To address these issues, the TREC Contextual Suggestion Track explored an open test collection, where participants were allowed to submit any web page as a result for a personalized venue recommendation task. This prompts the question on the reusability of the resulting test collection: How does the open nature affect the pooling process? Can participants reliably evaluate variant runs with the resulting qrels? Can other teams evaluate new runs reliably? In short, does the set of pooled and judged documents effectively produce a post hoc test collection? Our main findings are the following: First, while there is a strongly significant rank correlation, the effect of pooling is notable and results in underestimation of performance, implying the evaluation of non-pooled systems should be done with great care. Second, we extensively analyze impacts of open corpus on the fraction of judged documents, explaining how low recall affects the reusability, and how the personalization and low pooling depth aggravate that problem. Third, we outline a potential solution by deriving a fixed corpus from open web submissions.

Expert finding problem in bibliographic networks has received increased interest in recent years. This problem concerns finding relevant researchers for a given topic. Motivated by the observation that rarely do all coauthors contribute to a paper equally, in this paper, we propose two discriminative methods for realizing leading authors contributing in a scientific publication. Specifically, we cast the problem of expert finding in a bibliographic network to find leading experts in a research group, which is easier to solve. We recognize three feature groups that can discriminate relevant experts from other authors of a document. Experimental results on a real dataset, and a synthetic one that is gathered from a Microsoft academic search engine, show that the proposed model significantly improves the performance of expert finding in terms of all common information retrieval evaluation metrics.

Expert finding in bibliographic networks has received increased interests in recent years. This task concerns with finding relevant researchers for a given topic. Motivated by the observation that rarely do all coauthors contribute to a paper equally, in this paper, we propose a discriminative method to realize leading authors contributing in a scientific publication. Specifically, we cast the problem of expert finding in a bibliographic network to find leading experts in a research group, which is easier to solve. According to some observations, we recognize three feature groups that can discriminate relevant and irrelevant experts. Experimental results on a real dataset, and an automatically generated one that is gathered from Microsoft academic search show that the proposed model significantly improves the performance of expert finding in terms of all common Information Retrieval evaluation metrics.