With the advancement of Artificial Intelligence technology to make smart devices, understanding how humans develop trust in virtual agents is emerging as a critical research field. Through our research, we report on a novel methodology to investigate user’s trust in auditory assistance in a Virtual Reality (VR) based search task, under both high and low cognitive load and under varying levels of agent accuracy. We collected physiological sensor data such as electroencephalography (EEG), galvanic skin response (GSR), and heart-rate variability (HRV), subjective data through questionnaire such as System Trust Scale (STS), Subjective Mental Effort Questionnaire (SMEQ) and NASA-TLX. We also collected a behavioral measure of trust (congruency of users’ head motion in response to valid/ invalid verbal advice from the agent). Our results indicate that our custom VR environment enables researchers to measure and understand human trust in virtual agents using the matrices, and both cognitive load and agent accuracy play an important role in trust formation. We discuss the implications of the research and directions for future work.

Human trust is a psycho-physiological state that is difficult to measure, yet is becoming increasingly important for the design of human-computer interactions. This paper explores if human trust can be measured using physiological measures when interacting with a computer interface, and how it correlates with cognitive load. In this work, we present a pilot study in Virtual Reality (VR) that uses a multi-sensory approach of Electroencephalography (EEG), galvanic skin response (GSR), and Heart Rate Variability (HRV) to measure trust with a virtual agent and explore the correlation between trust and cognitive load. The goal of this study is twofold; 1) to determine the relationship between biosignals, or physiological signals with trust and cognitive load, and 2) to introduce a pilot study in VR based on cognitive load level to evaluate trust. Even though we could not report any significant main effect or interaction of cognitive load and trust from the physiological signal, we found that in low cognitive load tasks, EEG alpha band power reflects trustworthiness on the agent. Moreover, cognitive load of the user decreases when the agent is accurate regardless of task’s cognitive load. This could be possible because of small sample size, tasks not stressful enough to induce high cognitive load due to lab study and comfortable environment or timestamp synchronisation error due to fusing data from various physiological sensors with different sample rate.

Human–computer interaction systems that bypass manual control can be beneficial for many use cases, including users with severe motor disability. We investigated pupillometry (inferring mental activity via dilations of the pupil) as an interaction method because it is noninvasive, easy to analyse, and increasingly available for practical development. In 3 experiments we investigated the efficacy of using pupillometry to detect imaginary motor movements of the hand. In Experiment 1 we demonstrated that, on average, the pupillary response is greater when the participant is imagining a hand-grasping motion, as compared with the control condition. In Experiment 2 we investigated how imaginary hand-grasping affects the pupillary response over time. In Experiment 3 we employed a simple classifier to demonstrate single-trial detection of imagined motor events using pupillometry. Using the mean pupil diameter of a single trial, accuracy rates as high as 71.25%, were achieved. Implications for the development of a pupillometry-based switch and future directions are discussed.

This paper describes the continued development of a system to provide early assessment of speech development issues in children and better triaging to professional services. Whilst corpora of children's speech are increasingly available, recognition of disordered children's speech is still a data-scarce task. Transfer learning methods have been shown to be effective at leveraging out-of-domain data to improve ASR performance in similar data-scarce applications. This paper combines transfer learning, with previously developed methods for constrained decoding based on expert speech pathology knowledge and knowledge of the target text. Results of this study show that transfer learning with out-of-domain adult speech can improve phoneme recognition for disordered children's speech. Specifically, a Deep Neural Network (DNN) trained on adult speech and fine-tuned on a corpus of disordered children's speech reduced the phoneme error rate (PER) of a DNN trained on a children's corpus from 16.3% to 14.2%. Furthermore, this fine-tuned DNN also improved the performance of a Hierarchal Neural Network based acoustic model previously used by the system with a PER of 19.3%. We close with a discussion of our planned future developments of the system.

A proof of concept system is developed to provide a broad assessment of speech development issues in children. It has been designed to enable non-experts to complete an initial screening of children's speech with the aim of reducing the workload on Speech Language Pathology services. The system was composed of an acoustic model trained by neural networks with split temporal context features and a constrained HMM-encoded with the knowledge of Speech Language Pathologists. Results demonstrated the system was able to improve PER by 33% compared with standard HMM decoders, with a minimum PER of 19.03% achieved. Identification of Phonological Error Patterns with up to 94% accuracy was achieved despite utilizing only a small corpus of disordered speech from Australian children. These results indicate the proposed system is viable and the direction of further development are outlined in the paper. Copyright

In this research, we explore how different types of augmented gesture communication cues can be used under different view sharing techniques in a remote collaboration system. In a pilot study, we compared four conditions: (1) Pointers on Still Image, (2) Pointers on Live Video, (3) Annotation on Still Image, and (4) Annotation on Live Video. Through this study, we found three results. First, users collaborate more efficiently using annotation cues than pointer cues for communicating object position and orientation information. Second, live video becomes more important when quick feedback is needed. Third, the type of gesture cue has more influence on performance and user preference than the type of view sharing method.

In this paper we present a novel interaction framework for augmented reality, and demonstrate its application in an interactive AR exposure treatment system for the fear of spiders. We use data from the Microsoft Kinect to track and model real world objects in the AR environment, enabling realistic interaction between them and virtual content. Objects are tracked in three dimensions using the Iterative Closest Point algorithm and a point cloud model of the objects is incrementally developed. The approximate motion and shape of each object in the scene serve as inputs to the AR application. Very few restrictions are placed on the types of objects that can be used. In particular, we do not require objects to be marked in a certain way in order to be recognized, facilitating natural interaction. To demonstrate our interaction framework we present an AR exposure treatment system where virtual spiders can walk up, around, or behind real objects and can be carried, prodded and occluded by the user. We also discuss improvements we are making to the interaction framework and its potential for use in other applications.

We investigate 360° panoramas as overviews to support users in the task of locating objects in the surrounding environment. Panoramas are typically visualized as rectangular photographs, but this does not provide clear cues for physical directions in the environment. In this paper, we conduct a series of studies with three different shapes: Frontal, Top-Down and Bird's Eye; the last two shapes are chosen because they provide a clearer representation of the spatial mapping between panorama and environment. Our results show that good readability of the panorama is most important and that a clear representation of the spatial mapping plays a secondary role. This paper is the first to provide understanding on how users exploit 360° panoramic over-views to locate objects in the surrounding environment and how different design factors can affect user performance.

Creating entertaining and educational books not only requires providing visually stimulating content but also means for students to interact, create, and express themselves. In this paper we present a new type of mixed-reality book experience, which augments an educational coloring book with user-generated three dimensional content. We explore a “pop-up book” metaphor and describe a process by which children's drawing and coloring is used as input to generate and change the appearance of the book content. Our system is based on natural feature tracking and image processing techniques that can be easily exploited for other AR publishing applications.

Augmented Reality combines real world and virtual information in interactive visualizations. Since phones started integrating GPS, compass and accelerometer, several Augmented Reality browsers for phones have hit the market. These are applications that access large amounts of geo-referenced information from online sources and present it at corresponding physical locations, superimposed onto a live video stream. However, Augmented Reality is constrained by the camera's field of view and restricted to first- person views, limiting the amount of overview that users can gain. We present two zooming interfaces that compensate for these constraints by enabling users to smoothly zoom between the Augmented Reality view and (1) an egocentric panoramic view of 360°, and (2) an exocentric top-down view. We present the results from two studies that show how in most search tasks our zooming interfaces are faster and require less panning than an overlay- based tool, scaling better as the amount of information grows.

An assumption behind new interface approaches that employ physical means of interaction is that these can leverage users’ prior knowledge from the real world, making them intuitive or ‘natural’ to use. This paper presents a user study of Tangible Augmented Reality, which shows that physical input tools can invite a wide variety of interaction behaviours and raise unmatched expectations about how to interact. Children played with interactive sequences in an augmented book using physical paddles to control the main characters. Our analysis focuses on how knowledge and skills that children have from the physical world succeed or fail to apply in the interaction with this application. We found that children expected the digital augmentations to behave and react analogous to physical 3D objects, encouraged by the ability to act in 3D space and the (digital) visual feedback. The affordances of the paddles as physical interaction devices invited actions that the system could not detect or interpret. In effect, children often struggled to understand what it was in their actions that made the system react.

In this paper we present the results of our long term development of a mixed reality book. Most previous work in the area has focused on the technology of augmented reality books, such as providing registration using fiducial markers. In this work, however, we focused on exploring the design and development process of this type of application in a broader sense. We studied the semantics of a mixed reality book, the design space and the user experience with this type of interface.

In this paper we present the process of designing and developing a fully functional and interactive AR handheld device. Based on a human-centered approach we describe the results of our exploration; from the contextual usage to the different rapid prototyped models and the evaluation activities. We also discuss some recommendations for how to use similar methods for developing future new AR physical interfaces.

We have observed children reading an augmented book aimed at early literacy education. We explored how children aged six to seven experience and interact with these novel instructional media. We here focus on issues arising from the tangibility of interface elements, the integration of physical and digital elements, on-screen and paper elements, and of text and interactive sequences.

This paper describes an augmented reality (AR) multimodal interface that uses speech and paddle gestures for interaction. The application allows users to intuitively arrange virtual furniture in a virtual room using a combination of speech and gestures from a real paddle. Unlike other multimodal AR applications, the multimodal fusion is based on the combination of time-based and semantic techniques to disambiguate a users speech and gesture input. We describe our AR multimodal interface architecture and discuss how the multimodal inputs are semantically integrated into a single interpretation by considering the input time stamps, the object properties, and the user context.