The ability of Voice User Interface (VUI) to understand how users will express their commands naturally and intuitively is an essential component of user experience, especially when the user is interacting with the VUI for the first time. Designing an automated method for testing the usability of VUI is a challenge for two reasons. First, there are many different ways for a user to express the same intention, e.g. “play some music”, ””put some music on”, etc., that is difficult to determine in advance. Second, many VUI apps today typically rely on the platform service provider (e.g. Amazon, Google, etc.) to perform many of the speech recognition and natural language processing tasks, and these services are provided as a blackbox. Consequently, it is difficult for the app developer to obtain information about errors and user feedback. In this paper, we propose a framework, VORI, to systematically evaluate the interactability of VUI, as well as a new metric for quantifying the interactability of a VUI. We use VORI to analyze 127 applications on Alexa by sending over 82,931 commands. Our analysis results highlight that 41.7% of apps only accept strict input that has to exactly match the developer’s predefined sample commands with an interactability score of 20% or less. This suggests developers should consider a better interactability strategy in the design of VUIs, and more research is needed to further explore the design space to improve the interactability.

Mobile augmented reality systems are becoming increasingly common and powerful, with applications in such domains as healthcare, manufacturing, education, and more. This rise in popularity is thanks in part to the functionalities offered by commercially available vision libraries such as ARCore, Vuforia, and Google’s ML Kit; however, these libraries also give rise to the possibility of a hidden operations threat , that is, the ability of a malicious or incompetent application developer to conduct additional vision operations behind the scenes of an otherwise honest AR application without alerting the end-user. In this article, we present the privacy risks associated with the hidden operations threat and propose a framework for application development and runtime permissions targeted specifically at preventing the execution of hidden operations. We follow this with a set of experimental results, exploring the feasibility and utility of our system in differentiating between user-expectation-compliant and non-compliant AR applications during runtime testing, for which preliminary results demonstrate accuracy of up to 71%. We conclude with a discussion of open problems in the areas of software testing and privacy standards in mobile AR systems.