While Alexa can perform over 100,000 skills, its capability covers only a fraction of what is possible on the web. Individuals need and want to automate a long tail of web-based tasks which often involve visiting different websites and require programming concepts such as function composition, conditional, and iterative evaluation. This paper presents DIYA (Do-It-Yourself Assistant), a new system that empowers users to create personalized web-based virtual assistant skills that require the full generality of composable control constructs, without having to learn a formal programming language.

This paper presents Brassau, a graphical virtual assistant that converts natural language commands into GUIs. A virtual assistant with a GUI has the following benefits compared to text or speech based virtual assistants: users can monitor multiple queries simultaneously, it is easy to re-run complex commands, and user can adjust settings using multiple modes of interaction. Brassau introduces a novel template-based approach that leverages a large corpus of images to make GUIs visually diverse and interesting. Brassau matches a command from the user to an image to create a GUI. This approach decouples the commands from GUIs and allows for reuse of GUIs across multiple commands. In our evaluation, users prefer the widgets produced by Brassau over plain GUIs.

This paper presents the architecture of Almond, an open, crowdsourced, privacy-preserving and programmable virtual assistant for online services and the Internet of Things (IoT). Included in Almond is Thingpedia, a crowdsourced public knowledge base of natural language interfaces and open APIs. Our proposal addresses four challenges in virtual assistant technology: generality, interoperability, privacy, and usability. Generality is addressed by crowdsourcing Thingpedia, while interoperability is provided by ThingTalk, a high-level domain-specific language that connects multiple devices or services via open APIs. For privacy, user credentials and user data are managed by our open-source ThingSystem, which can be run on personal phones or home servers. Finally, we address usability by providing a natural language interface, whose capability can be extended via training with the help of a menu-driven interface.

In this paper we build and evaluate a live-programming system for digital artists to create artwork within a virtual reality environment. Using the large display space that virtual reality provides, we develop an interaction for artists to see parallel evaluations of their art. A study of the system with ten participants demonstrated that parallel editing and execution is accessible to designers and that designers can leverage these techniques to survey more options faster.

This paper presents the design of an optical fiber proximity sensor for haptic exploration with a robotic finger. The sensor uses emitter and receiver optical fiber pairs to measure the intensity of light reflected off surrounding objects in a 2-D workspace. We present the design and construction a 32-point sensor array mounted within a 36 mm diameter finger and describe software techniques to process data acquired by an inexpensive web-cam. We experimentally characterize the sensor performance and demonstrate applications for haptic exploration such as pre-contact velocity reduction and non-contact contour following based on object curvature.