The rise of personal assistants has made question answering a very popular mechanism for user-system interaction. In Question Answering System, implicit feedbacks can be easily observed (user clicking in the link given by the QA system), but they are noisy. However, receiving an explicit feedback on the quality of the response just given is rare but more valuable. Motivated by a practical need in Question Answering System of processing these two types of rewards, this paper investigates and proposes a new stochastic multi-armed bandit model in which each action has a noisy reward and a sparse reward. We studied this problem in the contextual bandit settings, and proposed and analyzed efficient algorithms that are based on the LINUCB frameworks. Our algorithms are verified by empirical studies on various reward distributions and a real-world dataset and application.

In real-world applications when deploying Machine Learning (ML) models, initial model development includes close analysis of the model results and behavior by a data scientist. Once trained, however, models may need to be retrained with new data or updated to adhere to new rules or regulations. This presents two challenges. First, how to communicate how a model is making its decisions before and after retraining, and second how to support model editing to take into account new requirements. To address these needs, we built AIMEE (AI Model Explorer and Editor), a tool created to address these challenges by providing interactive methods to explain, visualize, and modify model decision boundaries using rules. Rules should benefit model builders by providing a layer of abstraction for understanding and manipulating the model and reduces the need to modify individual rows of data directly. To evaluate if this was the case, we conducted a pair of user studies totaling 23 participants to evaluate AIMEE's rules-based approach for model explainability and editing. We found that participants correctly interpreted rules and report on their perspectives of how rules are beneficial (and not), ways that rules could support collaboration, and provide a usability evaluation of the tool.

Machine learning technologies are increasingly being applied in many different domains in the real world. As autonomous machines and black-box algorithms begin making decisions previously entrusted to humans, great academic and public interest has been spurred to provide explanations that allow users to understand the decision-making process of the machine learning model. Besides explanations, Interactive Machine Learning (IML) seeks to leverage user feedback to iterate on an ML solution to correct errors and align decisions with those of the users. Despite the rise in explainable AI (XAI) and Interactive Machine Learning (IML) research, the links between interactivity, explanations, and trust have not been comprehensively studied in the machine learning literature. Thus, in this study, we develop and evaluate an explanation-driven interactive machine learning (XIML) system with the Tic-Tac-Toe game as a use case to understand how a XIML mechanism improves users’ satisfaction with the machine learning system. We explore different modalities to support user feedback through visual or rules-based corrections. Our online user study (n = 199) supports the hypothesis that allowing interactivity within this XIML system causes participants to be more satisfied with the system, while visual explanations play a less prominent (and somewhat unexpected) role. Finally, we leverage a user-centric evaluation framework to create a comprehensive structural model to clarify how subjective system aspects, which represent participants’ perceptions of the implemented interaction and visualization mechanisms, mediate the influence of these mechanisms on the system’s user experience.

AI solutions are heavily dependant on the quality and accuracy of the input training data, however the training data may not always fully reflect the most up-to-date policy landscape or may be missing business logic. The advances in explainability have opened the possibility of allowing users to interact with interpretable explanations of ML predictions in order to inject modifications or constraints that more accurately reflect current realities of the system. In this paper, we present a solution which leverages the predictive power of ML models while allowing the user to specify modifications to decision boundaries. Our interactive overlay approach achieves this goal without requiring model retraining, making it appropriate for systems that need to apply instant changes to their decision making. We demonstrate that user feedback rules can be layered with the ML predictions to provide immediate changes which in turn supports learning with less data.

We consider the problem of distinguishing two machine learning (ML) models built for the same task in a human-interpretable way. As models can fail or succeed in different ways, classical accuracy metrics may mask crucial qualitative differences. This problem arises in a few contexts. In business applications with periodically retrained models, an updated model may deviate from its predecessor for some segments without a change in overall accuracy. In automated ML systems, where several ML pipelines are generated, the top pipelines have comparable accuracy but may have more subtle differences. We present a method for interpretable comparison of binary classification models by approximating them with Boolean decision rules. We introduce stabilization conditions that allow for the two rule sets to be more directly comparable. A method is proposed to compare two rule sets based on their statistical and semantic similarity by solving assignment problems and highlighting changes. An empirical evaluation on several benchmark datasets illustrates the insights that may be obtained and shows that artificially induced changes can be reliably recovered by our method.

Recommender systems are used with increasing frequency in a wide variety of domains ranging from e- commerce to tourism, healthcare and online learning. However, the interaction with these systems generally tends to be limited to shallow feedback, such as providing ratings or filtering. Allowing users to interact with the recommender systems in a conversational environment brings opportunities in which the preferences can effectively be elicited from the users while the users can feel more in control of the whole process. However, when the existing non-interactive recommender systems are considered, it may not be easy to build an interactive layer directly on top of them. This is because there is already a great deal of modelling and work invested in the underlying algorithm and the system itself. Enabling interaction could mean rebuilding the whole solution from scratch, as the current design may not be able to consume preferences and information learnt from the user interaction online. In this paper, we propose the Interactive Recommender Framework, which converts non-interactive recommender solutions to conversational recommenders. We demonstrate how Interactive Recommender Framework can successfully enable interactivity on top of non-interactive recommender systems by integrating it into two different recommender algorithms from literature, and validate our solution through offline simulation experiments and online user studies.

Chatbots or conversational recommenders have gained increasing popularity as a new paradigm for Recommender Systems (RS). Prior work on RS showed that providing explanations can improve transparency and trust, which are critical for the adoption of RS. Their interactive and engaging nature makes conversational recommenders a natural platform to not only provide recommendations but also justify the recommendations through explanations. The recent surge of interest inexplainable AI enables diverse styles of justification, and also invites questions on how styles of justification impact user perception. In this article, we explore the effect of “why” justifications and “why not” justifications on users’ perceptions of explainability and trust. We developed and tested a movie-recommendation chatbot that provides users with different types of justifications for the recommended items. Our online experiment ( n = 310) demonstrates that the “why” justifications (but not the “why not” justifications) have a significant impact on users’ perception of the conversational recommender. Particularly, “why” justifications increase users’ perception of system transparency, which impacts perceived control, trusting beliefs and in turn influences users’ willingness to depend on the system’s advice. Finally, we discuss the design implications for decision-assisting chatbots.

Career goals represent a special case for recommender systems and require considering both short and long term goals. Recommendations must represent a trade off between relevance to the user, achievability and aspirational goals to move the user forward in their career. Users may have different motivations and concerns when looking for a new long term goal, so involving the user in the recommender process becomes all the more important than in other domains. Additionally, the cost to the user of making a bad decision is much higher than investing two hours in watching a movie they don't like or listening to an unappealing song. As a result, we feel career recommendations is a unique opportunity to truly engage the user in an interactive recommender as we believe they will invest the cognitive load. In this paper, we present an interactive career goal recommender framework that leverages the power of dialogue to allow the user interactively improve the recommendations and bring their own preferences to the system. The underlying recommendation algorithm is a novel solution that suggests both short and long term goals through utilizing the sequential patterns extracted from career trajectories that are enhanced with features of the supporting user profiles. The effectiveness of the proposed solution is demonstrated with extensive experiments on two real world data sets.

Recent research focuses beyond recommendation accuracy, towards human factors that influence the acceptance of recommendations, such as user satisfaction, trust, transparency and sense of control. We present a generic interactive recommender framework that can add interaction functionalities to non-interactive recommender systems. We take advantage of dialogue systems to interact with the user and we design a middleware layer to provide the interaction functions, such as providing explanations for the recommendations, managing users' preferences learnt from dialogue, preference elicitation and refining recommendations based on learnt preferences.

Sellers work together as a team on sales opportunities, using their expertise in different roles to increase the probability of a win. These roles include managing the relationship with the client, overall architecture support or deep knowledge of a particular product depending on the seller's expertise, and the current opportunity requirements. Forming the right team for an incoming opportunity is vital and depends on several factors including understanding the required roles for the opportunity and assigning the right person to fulfill these roles, taking into consideration the seller's social network. In this paper, we present the Opportunity Team Builder solution, which supports sellers in this work by dividing the process into the following sub-tasks; identifying the required roles for the opportunity based on the products that the client is interested in, recommending the best people to fulfill these roles, and providing a win probability figure to guide users in team formation. This supports the sellers in forming the bestfitting team for current opportunity dynamics. Each task in the solution is implemented as a model using historical data from previous sales opportunities. Models work in coordination with each other to ultimately maximize the probability of win over loss. The solution not only recommends the best person to join a team taking into account a combination of inferred skills and social relationships, but also the predicted impact the person can have on the overall performance of the team. We present how the whole solution is realized with an intelligent user interface enabling interaction with the user throughout the team formation process. Substantial experiments with real world data show that win/loss prediction is performed accurately and the Opportunity Team Builder solution can recommend teams that achieve a higher win probability.

This paper presents efficient data structures and a pruning technique in order to improve the efficiency of high utility sequential pattern mining. CRoM (Cumulated Rest of Match) based upper bound, which is a tight upper bound on the utility of the candidates is proposed in order to perform more conservative pruning before candidate pattern generation in comparison to the existing techniques. In addition, an efficient algorithm, HuspExt (High Utility Sequential Pattern Extraction), is presented which calculates the utilities of the child patterns based on that of the parents'. Substantial experiments on both synthetic and real datasets from different domains show that, the solution efficiently discovers high utility sequential patterns under low thresholds.

High utility sequential pattern mining has been considered as an important research problem and a number of relevant algorithms have been proposed for this topic. The main challenge of high utility sequential pattern mining is that, the search space is large and the efficiency of the solutions is directly affected by the degree at which they can eliminate the candidate patterns. Therefore, the efficiency of any high utility sequential pattern mining solution depends on its ability to reduce this big search space, and as a result, lower the computational complexity of calculating the utilities of the candidate patterns. In this paper, we propose efficient data structures and pruning technique which is based on Cumulated Rest of Match (CRoM) based upper bound. CRoM, by defining a tighter upper bound on the utility of the candidates, allows more conservative pruning before candidate pattern generation in comparison to the existing techniques. In addition, we have developed an efficient algorithm, High Utility Sequential Pattern Extraction (HuspExt), which calculates the utilities of the child patterns based on that of the parents'. Substantial experiments on both synthetic and real datasets from different domains show that, the proposed solution efficiently discovers high utility sequential patterns from large scale datasets with different data characteristics, under low utility thresholds.