We present Friendly Face - a virtual agent designed to reduce public speaking anxiety by standing within an audience. The agent senses the speaker's behavior during an oral presentation and provides emotional and instrumental support. The system unobtrusively tracks the motion, speech, and prosody of the presenter and provides an intuitive interface to give supportive feedback whenever the presenter looks at the agent, attentive listening behavior through agent gaze and backchannel listening behavior, and time and topic cueing based on real-time analysis of speech content compared to presentation slide contents. An evaluation of Friendly Face agent with a functionally equivalent control system demonstrated that the agent system led to significant reductions in public speaking anxiety compared to a control condition, assessed both objectively with physiological measures and validated self-report instruments.

Public speaking is an essential task for success in many careers, yet fear of public speaking makes this an undesirable activity for most people and negatively affects the quality of many presentations. Cognitive behavioral therapy is an effective tool for helping people overcome social anxieties disorders, including public speaking anxiety. However, most people do not engage in this therapy for public speaking anxiety, due to time constraints and other barriers. We present a virtual coach that uses cognitive behavioral therapy techniques to help presenters restructure irrational thoughts associated with public speaking anxiety. The design of the virtual coach was informed by an analysis of a corpus of cognitive restructuring examples generated by a clinical psychologist. In a between-subjects experiment comparing the virtual coach to a control condition, the virtual coach was shown to be significantly better at reducing thoughts associated with speech anxiety and improving a presentation experience for speakers.

This paper proposes using face as a touch surface and employing hand-over-face (HOF) gestures as a novel input modality for interaction with smartphones, especially when touch input is limited. We contribute InterFace, a general system framework that enables the HOF input modality using advanced computer vision techniques. As an examplar of the usage of this framework, we demonstrate the feasibility and usefulness of HOF with an Android application for improving single-user and group selfie-taking experience through providing appearance customization in real-time. In a within-subjects study comparing HOF against touch input for single-user interaction, we found that HOF input led to significant improvements in accuracy and perceived workload, and was preferred by the participants. Qualitative results of an observational study also demonstrated the potential of HOF input modality to improve the user experience in multi-user interactions. Based on the lessons learned from our studies, we propose a set of potential applications of HOF to support smartphone interaction. We envision that the affordances provided by the this modality can expand the mobile interaction vocabulary and facilitate scenarios where touch input is limited or even not possible.

Current slideware, such as PowerPoint, reinforces the delivery of linear oral presentations. In settings such as question answering sessions or review lectures, more extemporaneous and dynamic presentations are required. An intelligent system that can automatically identify and display the slides most related to the presenter's speech, allows for more speaker flexibility in sequencing their presentation. We present Quester, a system that enables fast access to relevant presentation content during a question answering session and supports nonlinear presentations led by the speaker. Given the slides contents and notes, the system ranks presentation slides based on semantic closeness to spoken utterances, displays the most related slides, and highlights the corresponding content keywords in slide notes. The design of our system was informed by findings from interviews with expert presenters and analysis of recordings of lectures and conference presentations. In a within-subjects study comparing our dynamic support system with a static slide navigation system during a question answering session, presenters expressed a strong preference for our system and answered the questions more efficiently using our system.

When deployed on mobile devices, virtual agents have the potential to deliver advice regarding medical conditions, as well as provide a ubiquitous channel for health education and behavior change for a variety of chronic health conditions. We describe design guidelines for mobile agent dialogues to support chronic disease management, a general-purpose smartphone-based architecture for a conversational virtual agent that simulates face-to-face health counseling conversations with patients, and an initial agent implementation that provides counseling to patients with the chronic heart condition atrial fibrillation in conjunction with a mobile heart rhythm monitor that is attached to the back of the phone. Preliminary results from a randomized trial with 120 patients with atrial fibrillation indicate that the agent results in significant improvements in self-reported quality of life relative to a standard of care control group.

Prior studies of public speaking behavior have demonstrated that public speaking anxiety monotonically decreases with the number of co-presenters giving an oral presentation and increases with the size of the audience. However, speaker "hand off" behavior---the verbal and nonverbal cues used to transition from one speaker to another---and its effect on speaker anxiety and presentation quality has not been systematically studied. In this work we report on two empirical studies of speaker hand-off behavior used during human co-presentations. We find that the cues used for hand-offs during prepared and rehearsed presentations differ significantly from the cues observed in face-to-face conversational turn-taking. We describe two systems that leverage automatic recognition of these verbal and nonverbal cues to drive hand-offs during co-presentations with a life-sized virtual agent.

Longitudinal agent-based interventions only work if people continue using them on a regular basis, thus identifying users who are at risk of disengaging from these applications is important for retention and efficacy. We develop machine learning models that predict long-term user engagement in three longitudinal virtual agent-based health interventions. We achieve accuracies of 74% to 90% in predicting user dropout in a given prediction period of the intervention based on the user's past interactions with the agent. Our models contain features related to session frequency and duration, health behavior, and user-agent dialogue content. We find that the features most predictive of dropout include number of user utterances, percent of user utterances that are questions, and the percent of user health behavior goals met during the observation period. Ramifications for the design of virtual agents for longitudinal applications are discussed.

The fear of forgetting what to say next is a common problem in oral presentation delivery. To maintain good content coverage and reduce anxiety, many presenters use written notes in conjunction with presentation slides. However, excessive use of notes during delivery can lead to disengagement with the audience and low quality presentations. We designed IntelliPrompter, a speech-based note display system that automatically tracks a presenter’s coverage of each slide’s content and dynamically adjusts the note display interface to highlight the most likely next topic to present. We developed two versions of our intelligent teleprompters using Google Glass and computer screen displays. The design of our system was informed by findings from 36 interviews with presenters and analysis of a presentation note corpus. In a within-subjects study comparing our dynamic screen-based and Google Glass note display interfaces with a static note system, presenters and independent judges expressed a strong preference for the dynamic screen-based system.

Most persons with spinal cord injury (SCI) require training and support for self-care management to help prevent the development of serious secondary conditions after hospital discharge. We have designed a virtual coach system, in which an animated character engages users in simulated face-to-face conversation to provide health education and motivate healthy behavior. We conducted an exploratory study with nine participants who have SCI to examine the acceptance and attitudes towards our system. Results of the study show that participants are highly receptive of the virtual coach technology and recognize it as an effective medium to promote self-care.

The quality of most professional oral presentations is often poor, owing to a number of factors, including public speaking anxiety. We present DynamicDuo, a system that uses an automated, life-sized, animated agent to help inexperienced speakers deliver their presentations in front of an audience. The design of the system was informed by an analysis of TED talks given by two human presenters to identify the most common dual-presentation formats and transition behaviors used. In a within-subjects study (N=12) comparing co-presenting with DynamicDuo against solo-presenting with conventional presentation software, we demonstrated that our system led to significant improvements in public speaking anxiety and speaking confidence for non-native English speakers. Judges who viewed videotapes of these presentations rated those with DynamicDuo significantly higher on speech quality and overall presentation quality for all presenters.

Phoneme-based text entry provides an alternative typing method for nonspeaking individuals who often experience difficulties in orthographic spelling. In this paper, we investigate the application of rate enhancement strategies to improve the user performance of phoneme-based text entry systems. We have developed a phoneme-based predictive typing system, which employs statistical language modeling techniques to dynamically reduce the phoneme search space and offer accurate word predictions. Results of a case study with a nonspeaking participant demonstrated that our rate enhancement strategies led to improved text entry speed and error rates.

Rehearsal is a critical component of preparing to give an oral presentation, yet it is frequently abbreviated, performed in ways that are inefficient or ineffective, or simply omitted. We conducted an exploratory study to understand the relationship between the theory and practice of presentation rehearsal, classifying our qualitative results into five themes to motivate more structured rehearsal support deeply integrated in slide presentation software. In a within-subject study (N=12) comparing against participants' existing rehearsal practices, we found that our resulting PitchPerfect system significantly improved overall presentation quality and content coverage as well as provided greater support for content mastery, time management, and confidence building.

The high incidence of literacy deficits among people with severe speech impairments (SSI) has been well documented. Without literacy skills, people with SSI are unable to effectively use orthographic-based communication systems to generate novel linguistic items in spontaneous conversation. To address this problem, phoneme-based communication systems have been proposed which enable users to create spoken output from phoneme sequences. In this paper, we investigate whether prediction techniques can be employed to improve the usability of such systems. We have developed iSCAN, a phoneme-based predictive communication system, which offers phoneme prediction and phoneme-based word prediction. A pilot study with 16 able-bodied participants showed that our predictive methods led to a 108.4% increase in phoneme entry speed and a 79.0% reduction in phoneme error rate. The benefits of the predictive methods were also demonstrated in a case study with a cerebral palsied participant. Moreover, results of a comparative evaluation conducted with the same participant after 16 sessions using iSCAN indicated that our system outperformed an orthographic-based predictive communication device that the participant has used for over 4 years.

The present study investigates the effectiveness of a computer-based intervention to support adults with severe speech and physical impairments (SSPI) in developing their phonological awareness, an essential contributory factor to literacy acquisition. Three participants with SSPI undertook seven intervention sessions during which they were asked to play a training game on an iPad. The game was designed to enable learners to practice their phonological awareness skills independently with minimal instruction from human instructors. Preliminary results of post-intervention assessments demonstrate general positive effects of the intervention upon the phonological awareness and literacy skills of the participants. These results support the use of mainstream technologies to aid learning for individuals with disabilities.