Imagine being able to send a personalized embodied agent to meetings you are unable to attend. This paper explores the idea of a Ditto —an agent that visually resembles a person, sounds like them, possesses knowledge about them, and can represent them in meetings. This paper reports on results from two empirical investigations: 1) focus group sessions with six groups (n=24) and 2) a Wizard of Oz (WOz) study with 10 groups (n=39) recruited from within a large technology company. Results from the focus group sessions provide insights on what contexts are appropriate for Dittos, and issues around social acceptability and representation risk. The focus group results also provide feedback on visual design characteristics for Dittos. In the WOz study, teams participated in meetings with two different embodied agents: a Ditto and a Delegate (an agent which did not resemble the absent person). Insights from this research demonstrate the impact these embodied agents can have in meetings and highlight that Dittos in particular show promise in evoking feelings of presence and trust, as well as informing decision making. These results also highlight issues related to relationship dynamics such as maintaining social etiquette, managing one's professional reputation, and upholding accountability. Overall, our investigation provides early evidence that Dittos could be beneficial to represent users when they are unable to be present but also outlines many factors that need to be carefully considered to successfully realize this vision.

Hybrid meetings limit inclusion for remote participants. The Hybridge experimental system provides different interfaces for remote and room endpoints, focusing on improving inclusion via shared spatiality and remote agency. In-room participants see remotes on displays around a table, and remotes see video integrated into a digital twin. Remotes can choose where to appear and from where they view the room. We tested Hybridge in a within-subjects study of group survival tasks. An in-person condition was followed by a counterbalanced order of hybrid traditional videoconferencing ("Gallery") and Hybridge. We found that co-presence and agency differences between in-room and remotes were alleviated in Hybridge but remained in Gallery. Physical presence for remotes was higher in Hybridge than Gallery. Conversation flow was better in Hybridge than Gallery, but ease of awareness was not different. We argue that asymmetry should be embraced when designing hybrid meeting systems, with inclusivity achieved by tailoring features for the needs of different endpoints.

Virtual environments (VEs) afford similar interactions to those in physical environments: individuals can navigate and manipulate objects. Yet, a prerequisite for these interactions is being able to view the environment. Despite the existence of numerous scene-viewing techniques (i.e., interaction techniques that facilitate the visual perception of virtual scenes), there is no guidance to help designers choose which techniques to implement. We propose a scene taxonomy based on the visual structure and task within a VE by drawing on literature from cognitive psychology and computer vision, as well as virtual reality (VR) applications. We demonstrate how the taxonomy can be used by applying it to an accessibility problem, namely limited head mobility. We used the taxonomy to classify existing scene-viewing techniques and generate three new techniques that do not require head movement. In our evaluation of the techniques with 16 participants, we discovered that participants identified tradeoffs in design considerations such as accessibility, realism, and spatial awareness, that would influence whether they would use the new techniques. Our results demonstrate the potential of the scene taxonomy to help designers reason about the relationships between VR interactions, tasks, and environments.

Presenters often screen-share slides over remote meetings as visual aids. Screen reader users however often do not have adequate ways to access slide content during presentations. To inform the design of more accessible interfaces, we ran a formative design workshop to elicit what screen reader users value during live screen-shared presentations, which are Prioritize Key Information, Reduce Cognitive Load, Provide Exploration Independence, Minimize Control Effort, and Encode Spatial Information. Based on these values, we created two prototypes that users experienced in a follow-up design probe. These interactions provide users control over what, when, and how slide elements are read out and introduce spatial audio separation between the presenter’s voice and screen reader audio. Participants found that a laptop-based prototype designed for the first four values significantly improved the accessibility of screen-shared content over existing tools, even with a presenter who was perceived to present materials accessibly. The importance of concurrent exploration is highlighted by participants’ frequent and diverse use of access to consume information complementary to the presenter, align focus, and understand information using alternative means. A phone-based prototype that encoded spatial information and greater exploration independence at the cost of increased cognitive load received mixed feedback, illustrating the balance between values that interaction design needs to strike. We conclude with insights and considerations for how the identified values and corresponding interactions can be used to improve the accessibility of screen-shared presentations.

With the shift to hybrid meetings in work spaces, there is an increasing need to create a more inclusive hybrid meeting experience where people meeting together in a room interact with those joining remotely. This paper describes a design exploration, implementation, and evaluation of Perspectives, a novel hybrid meeting system that aimed to create an inclusive and equitable space for hybrid meetings. Perspectives digitally composites everyone into a virtual room so that each person has a unique but spatially consistent viewpoint into the meeting. The user study compared Perspectives with three commercially available UX designs for hybrid meetings: Gallery, Together Mode, and Front Row. Results from this study revealed key benefits of Perspectives, including supporting natural interactions, creating a strong sense of co-presence, and reducing cognitive load. Results from the study also helped iterate on the design principles of Perspectives, which offer important insights on supporting hybrid meetings.

People with limited mobility often use multiple devices when interacting with computing systems, but little is known about the impact these multi-modal configurations have on daily computing use. A deeper understanding of the practices, preferences, obstacles, and workarounds associated with accessible multi-modal input can uncover opportunities to create more accessible computer applications and hardware. We explored how people with limited mobility use multi-modality through a three-part investigation grounded in the context of video games. First, we surveyed 43 people to learn about their preferred devices and configurations. Next, we conducted semi-structured interviews with 14 participants to understand their experiences and challenges with using, configuring, and discovering input setups. Lastly, we performed a systematic review of 74 YouTube videos to illustrate and categorize input setups and adaptations in-situ. We conclude with a discussion on how our findings can inform future accessibility research for current and emerging computing technologies.

Virtual reality (VR) leverages human sight, hearing and touch senses to convey virtual experiences. For d/Deaf and hard of hearing (DHH) people, information conveyed through sound may not be accessible. To help with future design of accessible VR sound representations for DHH users, this paper contributes a consistent language and structure for representing sounds in VR. Using two studies, we report on the design and evaluation of a novel taxonomy for VR sounds. Study 1 included interviews with 10 VR sound designers to develop our taxonomy along two dimensions: sound source and intent. To evaluate this taxonomy, we conducted another study (Study 2) where eight HCI researchers used our taxonomy to document sounds in 33 VR apps. We found that our taxonomy was able to successfully categorize nearly all sounds (265/267) in these apps. We also uncovered additional insights for designing accessible visual and haptic-based sound substitutes for DHH users.

We propose Nearmi, a framework that enables designers to create customizable and accessible point-of-interest (POI) techniques in virtual reality (VR) for people with limited mobility. Designers can use Nearmi by creating and combining instances of its four components—representation, display, selection, and transition. These components enable users to gain awareness of POIs in virtual environments, and automatically re-orient the virtual camera toward a selected POI. We conducted a video elicitation study where 17 participants with limited mobility provided feedback on different Nearmi implementations. Although participants generally weighed the same design considerations when discussing their preferences, their choices reflected tradeoffs in accessibility, realism, spatial awareness, comfort, and familiarity with the interaction. Our findings highlight the need for accessible and customizable VR interaction techniques, as well as design considerations for building and evaluating these techniques.

Virtual reality (VR) leverages sight, hearing, and touch senses to convey virtual experiences. For d/Deaf and hard of hearing (DHH) people, however, information conveyed through sound may not be accessible. While prior work has explored making every day sounds accessible to DHH users, the context of VR is, as yet, unexplored. In this paper, we provide a first comprehensive investigation of sound accessibility in VR. Our primary contributions include a design space for developing visual and haptic substitutes of VR sounds to support DHH users and prototypes illustrating several points within the design space. We also characterize sound accessibility in commonly used VR apps and discuss findings from early evaluations of our prototypes with 11 DHH users and 4 VR developers.

Through an iterative design process using Wizard of Oz (WOz) prototypes, we designed a video calling application for people with Autism Spectrum Disorder. Our Video Calling for Autism prototype provided an Expressiveness Mirror that gave feedback to autistic people on how their facial expressions might be interpreted by their neurotypical conversation partners. This feedback was in the form of emojis representing six emotions and a bar indicating the amount of overall expressiveness demonstrated by the user. However, when we built a working prototype and conducted a user study with autistic participants, their negative feedback caused us to reconsider how our design process led to a prototype that they did not find useful. We reflect on the design challenges around developing AI technology for an autistic user population, how Wizard of Oz prototypes can be overly optimistic in representing AI-driven prototypes, how autistic research participants can respond differently to user experience prototypes of varying fidelity, and how designing for people with diverse abilities needs to include that population in the development process.

Mobile videoconferencing is increasingly being used to bring remote friends or family along to an activity happening outside the home, such as shopping or visiting a tourist attraction. We explored how including contextual information of the event, in addition to audio and video of the person at the event, impacts the shared experience. We studied three kinds of information: a map showing the position of the person at the activity, a second live video showing what was in front of that person, and periodic high quality images showing what was in front of the person. We carried out a field study with twelve pairs of participants, where one participant (the nomad) was at a self-selected activity while the other (the couch potato) joined the activity from our living room lab. The study results show that including contextual information significantly improved connectedness and the sense of presence for both participants. Each type of contextual information offered unique benefits. The map was used for orientation and to provide directions, the live video for "do you see this" moments and to maintain a sense of liveliness, and the periodic images for "did you see that" moments and to see greater detail. Together they led to smooth view negotiation, activity input from the couch potato, and high levels of engagement.

Video communication is moving beyond face-to-face discussions on desktop computers to sharing experiences out in the real world. We explored how mobile video could enable distributed family members to share experiences wherever they occurred - kids' sporting events, birthday parties, etc. We investigated how people used two technology probes to share activities outside the home: an iPad running Skype and our Experiences2Go prototype composed of a networked slate and a camcorder on a tripod. We observed their use in the field with nine families and explored the impact that their mobility, optical zoom, and multiple view features had on sharing the experience. We identified four sets of stakeholders in sharing experiences, the variety of sharing scenarios enabled, and reactions to the features that each probe offered, leading to design considerations for future mobile shared experience systems.

Task and reference spaces are important communication channels for remote collaboration. However, all existing systems for sharing these spaces have an inherent weakness: they cannot share arbitrary physical and digital objects on arbitrary surfaces. We present IllumiShare, a new cost-effective, light-weight device that solves this issue. It both shares physical and digital objects on arbitrary surfaces and provides rich referential awareness. To evaluate IllumiShare, we studied pairs of children playing remotely. They used IllumiShare to share the task-reference space and Skype Video to share the person space. The study results show that IllumiShare shared the play space in a natural and seamless way. We also found that children preferred having both spaces compared to having only one. Moreover, we found that removing the task-reference space caused stronger negative disruptions to the play task and engagement level than removing the person space. Similarly, we found that adding the task-reference space resulted in stronger positive disruptions.

Two important performance metrics in collaborative systems are local and remote response times. Previous analytical and simulation work has shown that these response times depend on three important factors: processing architecture, communication architecture, and scheduling of tasks dictated by these two architectures. We show that it is possible to create a system that improves response times by dynamically adjusting these three system parameters in response to changes to collaboration parameters such as new users joining and network delays changing. We present practical approaches for collecting collaboration parameters, computing multicast overlays, applying analytical models of previous work, preserving coupling semantics during optimizations, and keeping overheads low. Simulations and experiments show that the system improves performance in practical scenarios.

We conducted a study comparing avatar conferencing with video and audio conferencing for work scenarios. We studied nine four-person teams using a within-subjects design that measured users' perceptions and preferences across the conferencing conditions. Video was rated highest in all measures. Avatar and Audio were rated similarly, except for sociability, where Avatar was rated higher than Audio, and realism, where Avatar was rated lower than Audio. While users appreciated how avatar conferencing brought them together in a common virtual space, they found the cartoon avatars to be inappropriate for a professional discussion. As a result, participants preferred Video the most and Avatar the least for a business meeting. Lower ratings for the avatar condition were partly due to users' frustrations when the avatar system did not track them perfectly. When assuming a "perfect" system, preference for Avatar increased significantly while preference for Audio and Video remained unchanged.

People sometimes miss small parts of meetings and need to quickly catch up without disrupting the rest of the meeting. We developed an Accelerated Instant Replay (AIR) Conferencing system for videoconferencing that enables users to catch up on missed content while the meeting is ongoing. AIR can replay parts of the conference using four different modalities: audio, video, conversation transcript, and shared workspace. We performed two studies to evaluate the system. The first study explored the benefit of AIR catch-up during a live meeting. The results showed that when the full videoconference was reviewed (i.e., all four modalities) at an accelerated rate, users were able to correctly recall a similar amount of information as when listening live. To better understand the benefit of full review, a follow-up study more closely examined the benefits of each of the individual modalities. The results show that users (a) preferred using audio along with any other modality to using audio alone, (b) were most confident and performed best when audio was reviewed with all other modalities, (c) compared to audio-only, had better recall of facts and explanations when reviewing audio together with the shared workspace and transcript modalities, respectively, and (d) performed similarly with audio-only and audio with video review.

The performance of a collaborative system depends on how two mandatory collaborative tasks, processing and transmission of user commands, are scheduled. We have developed multiple policies for scheduling these tasks on computers that have (a) one processing element on the network interface card and (b) one or more processing cores on the CPU. To compare these policies, we have a developed a formal analytical model that predicts their performance. It shows that the optimal scheduling policy depends on several factors including the number of cores that is available. We have implemented a system that supports all of the policies and performed experiments to validate the formal model. This system is a component of a self-optimizing scheduler we have developed that improves response times by automatically choosing the scheduling policy based on number of cores and other factors.

When people attend meetings they may miss parts of the discussion if they, for example, step out to take a phone call, go to the bathroom, or have a momentary lapse in concentration. As a result, they may need to catch up on what they missed upon returning to the meeting. Asking other attendees for a recap is often disruptive. To avoid such disruptions, we have developed an Accelerated Instant Replay (AIR) Conferencing system for videoconferencing that enables participants to privately catch up to an ongoing meeting. We explored several mechanisms where the meeting content is replayed at an accelerated rate so that the participants can catch up to the live discussion reasonably quickly.

A collaborative system must perform both processing and transmission tasks. We present a policy for scheduling these tasks on a single core that is inspired by studies of human perception and the real-time systems field. It lazily delays the execution of the processing task if the delay cannot be noticed by humans. We use simulations and formal analysis to compare this policy with previous scheduling policies. We show that the policy trades-off an unnoticeable degradation in performance of some users for a much larger noticeable improvement in performance of others.

We have found that viewing recorded meetings using traditional meeting viewers whose interfaces consist of an automatic speaker and a fixed context view does not provide sufficient information and control to the users. In particular, a survey of users who watch meeting recordings on a regular basis revealed that it is also useful to provide (1) speaker-related information, including who the speaker is talking to, looking at, and being interrupted by, and (2) more control of the interface, including changing the relative sizes of the speaker and context views and navigating within the context view. We present a 3D interface prototype designed specifically to meet these requirements when viewing recorded meetings. We describe in detail the results of a user study comparing the effectiveness of the new and traditional style interfaces with respect to these requirements. Based on this study, we present a set of guidelines for future interfaces.

While multicast has been studied extensively in many domains such as content streaming and file sharing, there is little research applying it to synchronous collaborations involving shared access to a distributed object. Based on several cases of real-world collaborations involving instant messaging, distributed lectures, and computationally-intensive collaborative game playing, we show that compared to traditional centralized and replicated collaboration architectures, a new bi-architecture collaboration system model with multicasting support can improve response, feedthrough, and task completion times. In addition, we show that to optimize performance, the set of traditionally considered factors, consisting of network delays and transmission costs, must be expanded to include several new factors, such as processing costs, scheduling policies, and think times. In one or more of the real-world collaborations we consider, we show that multicast (a) can increase feedthrough times if processing costs and scheduling policies are not considered and (b) may degrade or improve task completion times depending on the cost of computing the multicast overlay.

We evaluate response times, in N-user collaborations, of the popular centralized (client-server) and replicated (peer-to-peer) architectures, and a hybrid architecture in which each replica serves a cluster of nearby clients. Our work consists of definitions of aspects of these architectures that have previously been unspecified but must be resolved for the analysis, a formal evaluation model, and a set of experiments. The experiments are used to define the parameters of and validate the formal analysis. In addition, they compare the performances, under the three architectures, of existing data-centric, logic-centric, and stateless shared components. We show that under realistic conditions, a small number of users, high intra-cluster network delays, and large output processing and transmission costs favor the replicated architecture, large input size favors the centralized architecture, high inter-cluster network delays favor the hybrid architecture, and high input processing and transmission costs, low think times, asymmetric processing powers, and logic-intensive applications favor both the centralized and hybrid architectures. We use our validated formal model to make useful predictions about the performance of the three kinds of architectures under realistic scenarios we could not create in lab experiments.