Users can adapt contemporary natural language interfaces (NLIs) by teaching the NLIs how to handle new natural language (NL) inputs. One promising approach is interactive task learning (ITL), which enables users to teach new NL inputs for multi-modal systems. While recent advances enable users to teach the syntactic and semantic level of the NL inputs through ITL, NLIs are still not able to learn how to consider the context, such as the current state of the graphical user interface (GUI). To address this challenge, we designed MALACHITE through three formative studies. MALACHITE enables users to successfully teach NL inputs on a semantic and syntactic level leveraging the GUI screen of a data visualization tool. With two evaluative studies, we provide evidence that with MALACHITE ’s suggestions, users significantly improve their accuracy by a factor of 2.3 in teaching GUI-dependent NL inputs in contrast to those without MALACHITE ’s suggestions.

Users are increasingly empowered to personalize natural language interfaces (NLIs) by teaching how to handle new natural language (NL) inputs. However, our formative study found that when teaching new NL inputs, users require assistance in clarifying ambiguities that arise and want insight into which parts of the input the NLI understands. In this paper we introduce ONYX, an intelligent agent that interactively learns new NL inputs by combining NL programming and programming-by-demonstration, also known as multi-modal interactive task learning. To address the aforementioned challenges, ONYX provides suggestions on how ONYX could handle new NL inputs based on previously learned concepts or user-defined procedures, and poses follow-up questions to clarify ambiguities in user demonstrations, using visual and textual aids to clarify the connections. Our evaluation shows that users provided with ONYX’s new features achieved significantly higher accuracy in teaching new NL inputs (median: 93.3%) in contrast to those without (median: 73.3%).