Advances in AI have enabled ESL learners to practice speaking through conversational systems. However, most tools rely on explicit correction, which can interrupt the conversation and undermine confidence. Grounded in second language acquisition and motivational psychology, we present AI Twin, a system that rephrases learner utterances into more fluent English and delivers them in the learner’s voice. Embodying a more confident and proficient version of the learner, AI Twin reinforces motivation through alignment with their aspirational Ideal L2 Self. Also, its use of implicit feedback through rephrasing preserves conversational flow and fosters an emotionally supportive environment. In a within-subject study with 20 adult ESL learners, we compared AI Twin with explicit correction and a non-personalized rephrasing agent. Results show that AI Twin elicited higher emotional engagement, with participants describing the experience as more motivating. These findings highlight the potential of self-representative AI for personalized, psychologically grounded support in ESL learning.

As digital tools increasingly mediate mental health care, self-clone chatbots can offer a uniquely novel approach to intra-personal exploration and self-derived support. Trained to replicate users’ conversational patterns, self-clones allow users to talk to themselves through their digital replicas. Despite the promises, these systems may carry risks around identity confusion, negative reinforcement, and blurred user agency. Through interviews with 16 mental health professionals and 6 general users, we aim to uncover tensions and design opportunities in this emerging space to guide responsible self-clone design. Our analysis produces a design framework organized around three priorities: (1) defining goals and grounding the approach in existing therapeutic models, (2) design dimensions including the self-clone persona and user-clone relationship dynamics, and (3) considerations for minimizing potential emotional and ethical harms. This framework contributes an interdisciplinary foundation for designing self-clone chatbots as AI-mediated self-interaction tools that are emotionally and ethically attuned in mental health contexts.

Peer agents can supplement real-time collaborative learning in asynchronous online courses. Constructive Controversy (CC) theory suggests that humans deepen their understanding of a topic by confronting and resolving controversies. This study explores whether CC’s benefits apply to LLM-based peer agents, focusing on the impact of agents’ disputatious behaviors and disclosure of agents’ behavior designs on the learning process. In our mixed-method study (n=144), we compare LLMs that follow detailed CC guidelines (regulated) to those guided by broader goals (unregulated) and examine the effects of disclosing the agents’ design to users (transparent vs. opaque). Findings show that learners’ values influence their agent interaction: those valuing control appreciate unregulated agents’ willingness to cease push-back upon request, while those valuing intellectual challenges favor regulated agents for stimulating creativity. Additionally, design transparency lowers learners’ perception of agents’ abilities. Our findings lay the foundation for designing effective collaborative peer agents in isolated educational settings.

AI compliance is becoming increasingly critical as AI systems grow more powerful and pervasive. Yet the rapid expansion of AI policies creates substantial burdens for resource-constrained practitioners lacking policy expertise. Existing approaches typically address one policy at a time, making multi-policy compliance costly. We present PASTA, a scalable compliance tool integrating four innovations: (1) a comprehensive model-card format supporting descriptive inputs across development stages; (2) a policy normalization scheme; (3) an efficient LLM-powered pairwise evaluation engine with cost-saving strategies; and (4) an interface delivering interpretable evaluations via compliance heatmaps and actionable recommendations. Expert evaluation shows PASTA’s judgments closely align with human experts (ρ ≥.626). The system evaluates five major policies in under two minutes at approximately $3. A user study (N = 12) confirms practitioners found outputs easy-to-understand and actionable, introducing a novel framework for scalable automated AI governance.

Social media clones are AI-powered social delegates of ourselves created using our personal data. As our identities and online personas intertwine, these technologies have the potential to greatly enhance our social media experience. If mismanaged however, these clones may also pose new risks to our social reputation and online relationships. To set the foundation for a productive and responsible integration, we set out to understand how social media clones will impact our online behavior and interactions. We conducted a series of semi-structured interviews introducing eight speculative clone concepts to 32 social media users through a design workbook. Applying existing work in AI-mediated communication in the context of social media, we found that although clones can offer convenience and comfort, they can also threaten the user’s authenticity and increase distrust within the online community. As a result users tend to behave more like their clones to mitigate discrepancies and interaction breakdowns. These findings are discussed through the lens of past literature in identity and impression management to highlight challenges in the adoption of social media clones by the general public, and propose design considerations for their successful integration into social media platforms.

Peer Reviewed

In a rapidly evolving UX/UI design landscape marked by technological advancements and shifts toward hybrid work, understanding the implications of these changes on software prototyping practices is crucial. This study investigates the influence of evolving work practices, tool advancements, and designers’ attitudes on prototyping practices and design processes in the contemporary software industry. Based on in-depth interviews with 10 practitioners and educators, we explore the factors contributing to the preference for digital-first prototypes and the diminishing appeal of low-fidelity prototyping methods. Our findings reveal how digital prototypes outshine physical counterparts in hybrid work, the role of all-in-one digital tools in centralizing designers’ workflows and encouraging high-fidelity prototyping, corporate preferences for visually appealing prototypes, and the impact of designers’ educational backgrounds, generational differences, and professional maturity. This research offers valuable insights to inform decision-making and strategies for design practitioners, educators, and organizations in adapting to current and future prototyping practices.

University students engage in a substantial amount of multitasking in online classes despite being aware of its negative impacts on their learning. Depending on the learner’s goals, in-class multitasking can be a positive strategic behavior to increase productivity. In a formative pilot study (N=10), we established the structure and scope for our design by exploring students’ motivations, perceptions, and challenges in in-class multitasking and identified several promising design elements. Our design facilitates multitasking in online synchronous classes by providing a novel bichronous (blending of synchronous and asynchronous) learning environment manifested in Time-Turner that enables asynchronous guided accelerated viewing of past content during synchronous classes. A summative evaluation of our prototype showed significant improvement in learning outcomes when multitasking (N=20). Furthermore, 95% of users found Time-Turner helpful and expressed interest in having it in their online classes. Our findings show the great potential of supporting positive multitasking in synchronous online classes.

Intergenerational communication can enhance well-being and family cohesion, but stereotypes and low empathy can be barriers to achieving effective communication. VR perspective-taking is a potential approach that is known to enhance understanding and empathy toward others by allowing a user to take another’s viewpoint. In this study, we introduce LegacySphere, a novel VR perspective-taking experience leveraging the combination of embodiment, role-play, and storytelling. To explore LegacySphere’s design and impact, we conducted an observational study involving five dyads with a one-generation gap. We found that LegacySphere promotes empathetic and reflexive intergenerational dialogue. Specifically, avatar embodiment encourages what we term “relationship cushioning,” fostering a trustful, open environment for genuine communications. The blending of real and embodied identities prompts insightful questions, merging both perspectives. The experience also nurtures a sense of unity and stimulates reflections on aging. Our work highlights the potential of immersive technologies for enhancing empathetic intergenerational relationships.

Exposure to diverse perspectives is helpful for bursting the filter bubble in online public video platforms. The recent advancement of Large Language Models (LLMs) illuminates the potential of creating a debate chatbot that prompts users to critically examine their stances on a topic formed by watching videos. However, whether the viewer is influenced by the chatbot may depend on its persona. In this paper, we investigated the effect of two relevant persona attributes - social identity and rhetorical styles - on critical thinking. In a mixed-methods study (n=36), we found that chatbots with outgroup (vs. ingroup) identity (t(33)=-2.33, p=0.03) and persuasive (vs. eristic) rhetoric (t(44)=1.98, p=0.05) induced critical thinking most effectively, making participants re-examine their arguments. However, participants’ stances remain largely unaffected, likely due to the chatbot’s lack of contextual knowledge and human touch. Our paper provides empirical groundwork for designing chatbot persona for remedying filter bubbles in online communities.

This paper explores the design challenges that arise in supporting online freelancers to navigate relationships with clients. Prior studies have shown that current platform designs can lead to worker precarity in freelancer-client relationships, such as power imbalances, information asymmetry, and labor abuse. To envision alternative designs that empower workers in managing their relationships with clients, we engaged 22 Upwork freelancers in participatory speculative design activities. Through this co-design process, we identified design tensions that constrain design options as a result of conflicting values and priorities that could only be balanced and compromised rather than completely resolved. Six design tensions were identified in the context of designing for four different phases of freelancing. We observed three patterns in these tensions: 1) the freelancers' need for client involvement in their tasks and career growth, which conflicted with their skepticism that clients had sufficient incentives to be involved; 2) that there was often no viable balancing option for some tensions, but they could be addressed through changes in the platform's incentive structure; and 3) some tensions occurred not only between freelancers and clients, but also within the freelancer community. We present three approaches for addressing these design tensions and discuss how this research can support more equitable and healthy freelancer-client relationships.

Videos are convenient resources for asynchronous learning, but they lack interpersonal interactions found in synchronous classrooms. Due to missed social connectedness, the isolated video-based learners experience low emotional, behavioral, and cognitive engagement. This work presents "Scripted Vicarious Dialogues" (SVD), a technique for engaging students in a pseudo-social experience of witnessing scripted dialogues between virtual characters (teaching assistants and students) around a video. We conducted a participatory design study to derive design guidelines for SVD. The findings indicate the need to distinguish the virtual components and to give students control of the dialogue’s pace. We then implemented an interactive prototype of SVD and evaluated it (N=40) against a non-social, direct-learning baseline. The results show that the preference for SVD versus the baseline is polarized (25 of 40 preferred SVD; no neutral preferences), and those who preferred SVD had significantly higher emotional and behavioral engagement with SVD compared to the baseline.

As the use of hand gestures becomes increasingly prevalent in virtual reality (VR) applications, prototyping Compound Freehand Interactions (CFIs) effectively and efficiently has become a critical need in the design process. Compound Freehand Interaction (CFI) is a sequence of freehand interactions where each sub-interaction in the sequence conditions the next. Despite the need for interactive prototypes of CFI in the early design stage, creating them is effortful and remains a challenge for designers since it requires a highly technical workflow that involves programming the recognizers, system responses and conditionals for each sub-interaction. To bridge this gap, we present GestureCanvas, a freehand interaction-based immersive prototyping system that enables a rapid, end-to-end, and code-free workflow for designing, testing, refining, and subsequently deploying CFI by leveraging the three pillars of interaction models: event-driven state machine, trigger-action authoring, and programming by demonstration. The design of GestureCanvas includes three novel design elements — (i) appropriating the multimodal recording of freehand interaction into a CFI authoring workspace called Design Canvas, (ii) semi-automatic identification of the input trigger logic from demonstration to reduce the manual effort of setting up triggers for each sub-interaction, (iii) on the fly testing for independently validating the input conditionals in-situ. We validate the workflow enabled by GestureCanvas through an interview study with professional designers and evaluate its usability through a user study with non-experts. Our work lays the foundation for advancing research on immersive prototyping systems allowing even highly complex gestures to be easily prototyped and tested within VR environments.

Design guidelines serve as an important conceptual tool to guide designers of interactive applications with well-established principles and heuristics. Consulting domain experts is a common way to develop guidelines. However, experts are often not easily accessible, and their time can be expensive. This problem poses challenges in developing comprehensive and practical guidelines. We propose a new guideline development method that uses online public videos as the basis for capturing diverse patterns of design goals and interaction primitives. In a case study focusing on AR-based assembly instructions, we apply our novel Identify-Rationalize pipeline, which distills design patterns from videos featuring AR-based assembly instructions (N=146) into a set of guidelines that cover a wide range of design considerations. The evaluation conducted with 16 AR designers indicated that the pipeline is useful for generating comprehensive guidelines. We conclude by discussing the transferability and practicality of our method.

Digitally replicating the appearance and behaviour of individuals is becoming feasible with recent advancements in deep-learning technologies such as interactive deepfake applications, voice conversion, and virtual actors. Interactive applications of such agents, termed AI clones, pose risks related to impression management, identity abuse, and unhealthy dependencies. Identifying concerns AI clones will generate is a prerequisite to establishing the basis of discourse around how this technology will impact a source individual's selfhood and interpersonal relationships. We presented 20 participants of diverse ages and backgrounds with 8 speculative scenarios to explore their perception towards the concept of AI clones. We found that (1. doppelganger-phobia) the abusive potential of AI clones to exploit and displace the identity of an individual elicits negative emotional reactions; (2. identity fragmentation) creating replicas of a living individual threatens their cohesive self-perception and unique individuality; and (3. living memories) interacting with a clone of someone with whom the user has an existing relationship poses risks of misrepresenting the individual or developing over-attachment to the clone. These findings provide an avenue to discuss preliminary ethical implications, respect for identity and authenticity, and design recommendations for creating AI clones.

Modern large-scale language models approach the quality of human-level writing. This promises the advent of AI writing companions performing AI-led writing under human control, surpassing traditional writing tools limited to revision and ideation supports. However, human-AI co-writing may endanger writers’ control, autonomy, and ownership by overstepping co-creative boundaries. Our design workbook study with 7 hobbyists and 13 professional writers elicited three sets of primary barriers to the adoption of human-AI co-writing. Storywriters desire retaining control over writing rather than letting AI take the lead when they (1) prioritize emotional values in turning ideas into words over the productivity of AI-generated writing; (2) have high self-confidence and distrust AI in challenging sub-tasks (e.g., creating characters and dialogue); and (3) expect the AI control mechanism to mismatch their writing strategies. We lay the groundwork for AI companions that respect storywriters’ personal values and writing methods.

Women make up approximately half of the workforce in ride-hailing, food delivery, and home service platforms in North America. While studies have reported that gig workers face bias, harassment, and a gender pay gap, we have limited understanding of women’s perspectives of these issues and their coping mechanisms. We interviewed 20 women gig workers to hear their unique experiences with these challenges. We found that gig platforms are gender-agnostic, meaning they do not acknowledge women’s experiences and the value they bring. By not enforcing anti-harassment policies in design, gig platforms also leave women workers vulnerable to bias and harassment. Due to the lack of support for immediate actions and in fear of losing access to work, women workers “brush off” harassment. In addition, the platforms’ dispatching and recommendation mechanisms do not acknowledge women’s contributions in perceived safety for customers and social support for peer workers.

This work investigates the practices and challenges of voice user interface (VUI) designers. Existing VUI design guidelines recommend that designers strive for natural human-agent conversation. However, the literature leaves a critical gap regarding how designers pursue naturalness in VUIs and what their struggles are in doing so. Bridging this gap is necessary for identifying designers’ needs and supporting them. Our interviews with 20 VUI designers identified 12 ways that designers characterize and approach naturalness in VUIs. We categorized these characteristics into three groupings based on the types of conversational context that each characteristic contributes to: Social, Transactional, and Core. Our results contribute new findings on designers’ challenges, such as a design dilemma in augmenting task-oriented VUIs with social conversations, difficulties in writing for spoken language, lack of proper tool support for imbuing synthesized voice with expressivity, and implications for developing design tools and guidelines.

In human-computer interaction (HCI) studies, bias in the gender representation of participants can jeopardize the generalizability of findings, perpetuate bias in data driven practices, and make new technologies dangerous for underrepresented groups. Key to progress towards inclusive and equitable gender practices is diagnosing the current status of bias and identifying where it comes from. In this mixed-methods study, we interviewed 13 HCI researchers to identify the potential bias factors, defined a systematic data collection procedure for meta-analysis of participant gender data, and created a participant gender dataset from 1,147 CHI papers. Our analysis provided empirical evidence for the underrepresentation of women, the invisibility of non-binary participants, deteriorating representation of women in MTurk studies, and characteristics of research topics prone to bias. Based on these findings, we make concrete suggestions for promoting inclusive community culture and equitable research practices in HCI.

Shadowing, i.e., listening to recorded native speech and simultaneously vocalizing the words, is a popular language-learning technique that is known to improve listening skills. However, despite strong evidence for its efficacy as a listening exercise, existing shadowing systems do not adequately support listening-focused practice, especially in self-regulated learning environments with no external feedback. To bridge this gap, we introduce Computer-Assisted Shadowing Trainer (CAST), a shadowing system that makes self-regulation easy and effective through four novel design elements — (i) in-the-moment highlights for tracking and visualizing progress, (ii) contextual blurring for inducing self-reflection on misheard words, (iii) self-listening comparators for post-practice self-evaluation, and (iv) adjustable pause-handles for self-paced practice. We base CAST on a formative user study (N=15) that provides fresh empirical grounds on the needs and challenges of shadowers. We validate our design through a summative evaluation (N=12) that shows learners can successfully self-regulate their shadowing practice with CAST while retaining focus on listening.

Pull Requests (PRs) are a frequently used method for proposing changes to source code repositories. When discussing proposed changes in a PR discussion, stakeholders often reference a wide variety of information objects for establishing shared awareness and common ground. Previous work has not considered how the referential behavior impacts collaborative software development via PRs. This knowledge gap is the major barrier in evaluating the current support for referencing in PRs and improving them. We conducted an explorative analysis of \textasciitilde7K references, collected from 450 public PRs on GitHub, and constructed taxonomies of referent types and expressions. Using our annotated dataset, we identified several patterns in the use of references. Referencing source code elements was prevalent but the authoring interface lacks support for it. Three classes of contextual factors influence referencing behaviors: referent type, discussion thread, and project attributes. Referencing patterns may indicate PR outcomes (e.g., merged PRs frequently reference issues, users, and tests). We conclude with design implications to support more effective referencing in PR discussion interfaces.

Throughout the COVID-19 pandemic, older adults have been encouraged to stay indoors and isolated, leading to potential disruptions in their social activities and interpersonal relationships. This interview study ($N=24$) provides a close examination of older adults' communication technology adoption and usage in light of the pandemic. Our interviews revealed that the pandemic motivated many older adults to learn new technology and become more tech-savvy in an effort to stay connected with others. However, older adults also reported challenges related to the pandemic that were major impediments to technology adoption. These were: (1) lack of access to in-person technology support under physical distancing mandates, (2) lack of opportunities for online participation due to negative age stereotypes and assumptions, and (3) increased apprehension to seek help from family members and friends who were suffering from pandemic-related stresses. This study extends technology adoption literature and contributes an up-to-date examination of the "grey digital divide" (the gap between older adults who use technology and those who do not). Our findings demonstrate that despite the rapidly increasing number of tech-savvy seniors, a digital divide not only persists, but has been exacerbated by the transition to virtual-only offerings. We reveal the challenges and coping strategies of older adults who remain separated from technology and propose actionable solutions to increase digital access during the COVID-19 pandemic and beyond.

Virtual reality drawing applications let users draw 3D shapes using brushes that form ribbon shaped, or ruled-surface, strokes. Each ribbon is uniquely defined by its user-specified ruling length, path, and the ruling directions at each point along this path. Existing brushes use the trajectory of a handheld controller in 3D space as the ribbon path, and compute the ruling directions using a fixed mapping from a specific controller coordinate-frame axis. This fixed mapping forces users to rotate the controller and thus their wrists to change ribbon normal or ruling directions, and requires substantial physical effort to draw even medium complexity ribbons. Since human ability to rotate their wrists continuously is heavily restricted, the space of ribbon geometries users can comfortably draw using these brushes is limited. These brushes can be unpredictable, producing ribbons with unexpectedly varying width or flipped and wobbly normals in response to seemingly natural hand gestures. Our AdaptiBrush ribbon brush system dramatically extends the space of ribbon geometries users can comfortably draw while enabling them to accurately predict the ribbon shape that a given hand motion produces. We achieve this by introducing a novel adaptive ruling direction computation method, enabling users to easily change ribbon ruling and normal orientation using predominantly translational controller, and thus wrist, motion. We facilitate ease-of-use by computing predictable ruling directions that smoothly change in both world and controller coordinate systems, and facilitate ease-of-learning by prioritizing ruling directions which are well-aligned with one of the controller coordinate system axes. Our comparative user studies confirm that our more general and predictable ruling computation leads to significant improvements in brush usability and effectiveness compared to all prior brushes; in a head to head comparison users preferred AdaptiBrush over the next-best brush by a margin of 2 to 1.

Listening to text using read-aloud applications is a popular way for people to consume content when their visual attention is situationally impaired (e.g., commuting, walking, tired eyes). However, due to the linear nature of audio, such apps do not support skimming---a non-linear, rapid form of reading---essential for quickly grasping the gist and organization of difficult texts, like academic or professional documents. To support auditory skimming for situational impairments, we (1) identified the user needs and challenges in auditory skimming through a formative study (N=20), (2) derived the concept of "eyes-reduced" skimming that blends auditory and visual modes of reading, inspired by how participants mixed visual and non-visual interactions, (3) generated a set of design guidelines for eyes-reduced skimming, and (4) designed and evaluated a novel audio skimming app that embodies the guidelines. Our in-situ preliminary observation study (N=6) suggested that participants were positive about our design and were able to auditorily skim documents. We discuss design implications for eyes-reduced reading, read-aloud apps, and text-to-speech engines.

Video-based learning (VBL) is widespread; however, there are numerous challenges when teaching and learning with video. For instructors, creating effective instructional videos takes considerable time and effort. For students, watching videos can be a passive learning activity. Artificial intelligence (AI) has the potential to improve the VBL experience for students and teachers. This half-day workshop will bring together multi-disciplinary researchers and practitioners to collaboratively envision the future of VBL enhanced by AI. This workshop will be comprised of a group discussion followed by a presentation session. The goal of the workshop is to facilitate the cross-pollination of design ideas and critical assessments of AI approaches to VBL.

As videos become increasingly ubiquitous, so is video-based commenting. To contextualize comments, people often reference specific audio/visual content within video. However, the literature falls short of explaining the types of video content people refer to, how they establish references and identify referents, how video characteristics (e.g., genre) impact referencing behaviors, and how references impact social engagement. We present a taxonomy for classifying video references by referent type and temporal specificity. Using our taxonomy, we analyzed 2.5K references with quotations and timestamps collected from public YouTube comments. We found: 1) people reference intervals of video more frequently than time-points, 2) visual entities are referenced more often than sounds, and 3) comments with quotes are more likely to receive replies but not more "likes". We discuss the need for in-situ dereferencing user interfaces, illustrate design concepts for typed referencing features, and provide a dataset for future studies.

While video becomes increasingly prevalent in educational settings, current research has yet to investigate what feedback instructors need regarding their students' engagement and learning despite video technologies being equipped to provide viewing analytics and collect student feedback. In this paper we investigate instructors' requirements from video analytics. We used a Grounded Theory Approach and interviewed 16 instructors who teach using video to determine the advantages for using video in their teaching and the different requirements for analytics and feedback in their existing practice. Based on our analysis of the interviews, we found three categories of information that instructors want to inform their teaching. Instructors are looking to see if their students have watched their videos, how much they understood in those videos, and how useful the videos are to the students. These categories provide the foundations and design implications for instructor-centric educational video analytics interfaces.

Studies on collaborative virtual environments (CVEs) have suggested capture and later replay of multimodal interactions (e.g., speech, body language, and scene manipulations), which we refer to as multimodal recordings, as an effective medium for time-distributed collaborators to discuss and review 3D content in an immersive, expressive, and asynchronous way. However, there exist gaps of empirical knowledge in understanding how this multimodal asynchronous VR collaboration (MAVRC) context impacts social behaviors in mediated-communication, workspace awareness in cooperative work, and user requirements for authoring and consuming multimedia recording. This study aims to address these gaps by conceptualizing MAVRC as a type of CSCW and by understanding the challenges and design considerations of MAVRC systems. To this end, we conducted an exploratory need-finding study where participants (N = 15) used an experimental MAVRC system to complete a representative spatial task in an asynchronously collaborative setting, involving both consumption and production of multimodal recordings. Qualitative analysis of interview and observation data from the study revealed unique, core design challenges of MAVRC in: (1) coordinating proxemic behaviors between asynchronous collaborators, (2) providing traceability and change awareness across different versions of 3D scenes, (3) accommodating viewpoint control to maintain workspace awareness, and (4) supporting navigation and editing of multimodal recordings. We discuss design implications, ideate on potential design solutions, and conclude the paper with a set of design recommendations for MAVRC systems.

Video is an increasingly popular medium for education. Motivated by the problem of video as a one-way medium, this paper investigates the ways in which learners» active interaction with video materials contributes to active learning. In this study, we examine active viewing behaviors, specifically seeking and highlighting within videos, which may suggest greater levels of participation and learning. We deployed a system designed for active viewing to an undergraduate class for a semester. The analysis of online activity traces and interview data provided novel findings on video highlighting behavior in educational contexts.

Video-based learning is most effective when students are engaged with video content; however, the literature has yet to identify students' viewing behaviors and ground them in theory. This paper addresses this need by introducing a framework of active viewing, which is situated in an established model of active learning to describe students' behaviors while learning from video. We conducted a field study with 460 undergraduates in an Applied Science course using a video player designed for active viewing to evaluate how students engage in passive and active video-based learning. The concept of active viewing, and the role of interactive, constructive, active, and passive behaviors in video-based learning, can be implemented in the design and evaluation of video players.

Speech commenting systems have been shown to facilitate asynchronous online communication from educational discussion to writing feedback. However, the production of speech comments introduces several challenges to users, including overcoming self-consciousness and time consuming editing. In this paper, we introduce TypeTalker, a speech commenting interface that presents speech as a synthesized generic voice to reduce speaker self-consciousness, while retaining the expressivity of the original speech with natural breaks and co-expressive gestures. TypeTalker streamlines speech editing through a simple textbox that respects temporal alignment across edits. A comparative evaluation shows that TypeTalker reduces speech anxiety during live-recording, and offers easier and more effective speech editing facilities than the previous state-of-the-art interface technique. A follow-up study on recipient perceptions of the produced comments suggests that while TypeTalker's generic voice may be traded-off with a loss of personal touch, it can also enhance the clarity of speech by refining the original speech's speed and accent.

In asynchronous online discussions, users actively reference visual materials (e.g., video, document) to provide supporting evidence and additional context. However, creating and comprehending complex references can be challenging, especially when there are multiple referents to refer, or when a referent is highly specific (e.g., specific sentences in a paper rather than the paper as a whole). To identify users' challenges in making references with multiple and specific referents while using existing discussion tools, we conducted an observational study and a preliminary interview. Based on the design lessons, we built Korero, a discussion interface that aims to facilitate complex referencing actions. For evaluation, we compared Korero against conventional interfaces in two user studies with referencing tasks of different referential difficulty. We found that Korero not only significantly reduces the time and effort in making references with multiple and specific referents, but also shows potential in increasing users' engagement with the discussion and referent materials.

Voice communication adds nuance and expressivity to virtual discussions, but its one-shot nature tends to discourage collaborators from utilizing it. However, text-based interfaces have made voice editing much easier, especially with recent advancements enabling live, time-aligned speech transcription. We introduce SimpleSpeech, an easy-to-use platform for asynchronous audio communication (AAC) with lightweight tools for inserting content, adjusting pauses, and correcting transcript errors. Qualitative and quantitative results suggest that novice audio producers, such as high school students, experience decreased mental workload when using SimpleSpeech to produce audio messages than without editing. The linguistic formality in SimpleSpeech messages was also studied, and found to form a middle ground between oral and written media. Our findings on editable voice messages show new implications for the optimal design and use cases of AAC systems.

New multi-modal annotation tools hold the promise of bringing the benefits of face-to-face contact to remote, asynchronous interactions. One such system, RichReview++, incorporates new techniques to improve access to the embedded multimedia commentary and allows users to annotate with new modalities, like deictic gestures. We conducted a series of field deployments of RichReview++ to characterize how these features benefit students using them for activities in the university classroom. Our first deployment investigated the use of multi-modal annotations as a way for instructors to provide feedback on student term papers. Our second deployment used annotations to support peer discussion about assigned readings in a graduate-level course. We found that presenting voice comments as interactive waveforms seems to facilitate students' consumption of the instructor's voice comments. We also found that gestural annotations clarify voice and give annotators a quick and lightweight way to alter the scope of their voice comments. Based on these results, we suggest ways to best leverage multi-modal annotation tools in education environments.

The digitization of educational course content has proved to be problematic for math instructors due to the lack of quality feedback tools that can accommodate the commenter to efficiently express math formulae and convey descriptions about complex ideas contextualized in situ. This paper proposes that RichReview, a document annotation system which creates inking, voice and deictic gestures on top of the student's submitted work, is a possible formative math feedback solution, because it enables face-to-face like commentary within the contexts of the document at hand. A preliminary qualitative evaluation study conducted while having students receive RichReview feedback showed promise to our approach to enhance the quality of feedback, with the implication that incorporating multi-modal feedback into workflows can be an effective method to address elements of feedback submissions lacking in coursework that has moved online.

The orientation and repositioning of physical artefacts (such as paper documents) to afford shared viewing of content, or to steer the attention of others to specific details, is known as micro-mobility. But the role of grasp in micro-mobility has rarely been considered, much less sensed by devices. We therefore employ capacitive grip sensing and inertial motion to explore the design space of combined grasp + micro-mobility by considering three classes of technique in the context of active reading. Single user, single device techniques support grip-influenced behaviors such as bookmarking a page with a finger, but combine this with physical embodiment to allow flipping back to a previous location. Multiple user, single device techniques, such as passing a tablet to another user or working side-by-side on a single device, add fresh nuances of expression to co-located collaboration. And single user, multiple device techniques afford facile cross-referencing of content across devices. Founded on observations of grasp and micro-mobility, these techniques open up new possibilities for both individual and collaborative interaction with electronic documents.

This paper introduces a novel document annotation system that aims to enable the kinds of rich communication that usually only occur in face-to-face meetings. Our system, RichReview, lets users create annotations on top of digital documents using three main modalities: freeform inking, voice for narration, and deictic gestures in support of voice. RichReview uses novel visual representations and time-synchronization between modalities to simplify annotation access and navigation. Moreover, RichReview's versatile support for multi-modal annotations enables users to mix and interweave different modalities in threaded conversations. A formative evaluation demonstrates early promise for the system finding support for voice, pointing, and the combination of both to be especially valuable. In addition, initial findings point to the ways in which both content and social context affect modality choice.

Having insufficient space for making annotations is a problem that afflicts both paper and digital documents. We introduce the TextTearing technique for in situ expansion of inter-line whitespace and pair it with a lightweight interaction for margin expansion as a way to address this problem. The full system leverages the dynamism of digital documents and employs a bimanual design that combines the precision of pen with the fluidity of touch. Our evaluation found that a simpler unimanual variant of TextTearing was preferred over direct annotation and margin-only expansion. Direct annotation in naturally occurring whitespace was least preferred.