Early mathematical activities often combine symbolic representations of numbers with physical manipulatives (e.g., numicons, dice, cards) that help children represent quantities. However, children with mathematical difficulties often struggle to relate abstract representations (number symbols) to their physical counterparts (manipulatives). In close collaboration with mathematics education experts, we explored how Augmented Reality (AR) can complement existing educational materials rather than replace them. We designed an AR system that augments existing exercises and conducted a pilot study with 10 children solving addition tasks. Our observations suggest that children enjoyed and successfully used the AR augmentations to complete the tasks. However, the AR system also created undesired side-effect such as an overreliance on the virtual augmentations. We discuss design implications for how AR systems can better complement established educational materials and support the learning of calculation-based arithmetic strategies.

When distributed mixed reality (MR) systems map physical spaces to enable co-presence for local and remote collaborators, they can unintentionally disrupt the sociocultural norms that give actions their meaning. For instance, a participant sitting at their own desk may be rendered as occupying their collaborator’s desk, inadvertently signalling an invasion of personal space. This paper examines the design tension between spatial information and sociocultural norms through a qualitative counterfactual cards activity with 20 participants, probing how they navigate these trade-offs across different collaborative contexts. Our findings show similarities between Expectancy Violations Theory (EVT) and the factors participants assess when they decide to uphold accurate spatial information or sociocultural norms during collaboration in MR. However, there were some departures from EVT, which can be used to inform the development of MR-specific theories in the future.

Dyscalculia is a specific learning difficulty that affects the ability to understand and work with numbers, often requiring specialized support strategies to develop foundational math skills. The exercises within these support strategies aim to reduce cognitive load by engaging multiple modalities—such as physically experiencing, verbally reasoning, and visually illustrating mathematical concepts. However, most current exercises are paper-based or de- livered through apps detached from the physical world, even when incorporating real-world elements like finger counting or handling currency. This work-in-progress explores how Augmented Reality (AR) can enhance these support strategies by embedding them in dynamic and interactive physical environments. We present a technical design exploration featuring three prototype applications that support students with dyscalculia (grades 1–9) through embodied and situated mathematical training. Our contribution is an early investigation into how AR can reimagine educational strategies by integrating physical manipulatives with virtual augmentation, creating more engaging math experiences for children with dyscalculia.

Prototyping multimodal interaction in XR (Augmented/Virtual Reality) is challenging for non-technical designers. Moreover, existing XR prototyping tools do not support concurrent multimodal interaction and lack an overall view of the interaction design of the experience. We created Mucho , a novel no-coding immersive tool for prototyping interactive multimodal XR experiences. Mucho supports three modes: In recording, designers can demonstrate input examples to populate a timeline representation with events. In playback, designers can create a video prototype based on the timeline while adding system actions at the corresponding timeframes. In live mode, Mucho creates a state machine to test the interactive experience in runtime. We present an evaluation by demonstration to illustrate how Mucho can prototype a diverse set of examples including modalities such as hand gestures, proximity, speech, and gaze. Finally, we discuss the limitations of our approach and outline future directions to support immersive prototyping of multimodal XR experiences.

The whiteboard is essential for collaborative work. To preserve its physicality in remote collaboration, Mixed Reality (MR) can blend real whiteboards across distributed spaces. Going beyond reality, MR can further enable interactions like panning and zooming in a virtually reconfigurable infinite whiteboard. However, this reconfigurability conflicts with the sense of physicality. To address this tension, we introduce Blended Whiteboard, a remote collaborative MR system enabling reconfigurable surface blending across distributed physical whiteboards. Blended Whiteboard supports a unique collaboration style, where users can sketch on their local whiteboards but also reconfigure the blended space to facilitate transitions between loosely and tightly coupled work. We describe design principles inspired by proxemics; supporting users in changing between facing each other and being side-by-side, and switching between navigating the whiteboard synchronously and independently. Our work shows exciting benefits and challenges of combining physicality and reconfigurability in the design of distributed MR whiteboards.

Although video is extremely useful for expressing interaction, post-hoc editing makes it impractical for rapid prototyping. We present an early-stage video-based design method — “editing-in-the-camera” — where title cards guide video capture and label video clips. This method lets designers easily create video prototypes that can be discussed within the same design session, without further editing. We present VideoClipper, a mobile app that incorporates this method by transforming sequences of title cards into an interactive storyboard that designers can shoot into directly. VideoClipper offers simple special effects to better illustrate user interaction with paper prototypes, including ghosting and stop-motion animation. We also present Collaborative VideoClipper, which was created during the COVID-19 pandemic to support multi-user, multi-device rapid prototyping with remote participants. We describe evaluation results of both applications and describe our experiences in diverse educational and professional settings, including brainstorming, interviewing, video prototyping, user studies and participatory design workshops.

Collaborating across dissimilar, distributed spaces presents numerous challenges for computer-aided spatial communication. Mixed reality (MR) can blend selected surfaces, allowing collaborators to work in blended f-formations (facing formations), even when their workstations are physically misaligned. Since collaboration often involves more than just participant pairs, this research examines how we might scale MR experiences for large-group collaboration. To do so, this study recruited collaboration designers (CDs) to evaluate and reimagine MR for large-scale collaboration. These CDs were engaged in a four-part user study that involved a technology probe, a semi-structured interview, a speculative low-fidelity prototyping activity and a validation session. The outcomes of this paper contribute (1) a set of collaboration design principles to inspire future computer-supported collaborative work, (2) eight collaboration patterns for blended f-formations and collaboration at scale and (3) theoretical implications for f-formations and space-place relationships. As a result, this work creates a blueprint for scaling collaboration across distributed spaces.

Mixed Reality allows for distributed meetings where people’s local physical spaces are virtually aligned into blended interaction spaces. In many cases, people’s physical rooms are dissimilar, making it challenging to design a coherent blended space. We introduce the concept of Partially Blended Realities (PBR) — using Mixed Reality to support remote collaborators in partially aligning their physical spaces. As physical surfaces are central in collaborative work, PBR supports users in transitioning between different configurations of tables and whiteboard surfaces. In this paper, we 1) describe the design space of PBR, 2) present RealityBlender to explore interaction techniques for how users may configure and transition between blended spaces, and 3) provide insights from a study on how users experience transitions in a remote collaboration task. With this work, we demonstrate new potential for using partial solutions to tackle the alignment problem of dissimilar spaces in distributed Mixed Reality meetings.

Programming by Demonstration (PbD) is a well-known technique that allows non-programmers to describe interactivity by performing examples of the expected behavior, but it has not been extensively explored for AR. We present Rapido, a novel early-stage prototyping tool to create fully interactive mobile AR prototypes from non-interactive video prototypes using PbD. In Rapido, designers use a mobile AR device to record a video prototype to capture context, sketch assets, and demonstrate interactions. They can demonstrate touch inputs, animation paths, and rules to, e.g., have a sketch follow the focus area of the device or the user’s world-space touches. Simultaneously, a live website visualizes an editable overview of all the demonstrated examples and infers a state machine of the user flow. Our key contribution is a method that enables designers to turn a video prototype into an executable state machine through PbD. The designer switches between these representations to interactively refine the final interactive prototype. We illustrate the power of Rapido’s approach by prototyping the main interactions of three popular AR mobile applications.

Designers have limited tools to prototype AR experiences rapidly. Can lightweight, immediate tools let designers prototype dynamic AR interactions while capturing the nuances of a 3D experience? We interviewed three AR experts and identified several recurring issues in AR design: creating and positioning 3D assets, handling the changing user position, and orchestrating multiple animations. We introduce PROJECT PRONTO, a tablet-based video prototyping system that combines 2D video with 3D manipulation. PRONTO supports four intertwined activities: capturing 3D spatial information alongside a video scenario, positioning and sketching 2D drawings in a 3D world, and enacting animations with physical interactions. An observational study with professional designers shows that participants can use PRONTO to prototype diverse AR experiences. All participants performed two tasks: replicating a sample non-trivial AR experience and prototyping their open-ended designs. All participants completed the replication task and found PRONTO easy to use. Most participants found that PRONTO encourages more exploration of designs than their current practices.

Professional designers and developers often struggle when transitioning between the design and implementation of an interactive system. We conducted three studies that focused on the design of custom interactions to understand the mismatches between their processes, tools, and representations. We found that current practices induce unnecessary rework and cause discrepancies between design and implementation. We identified three recurring types of breakdowns: omitting critical details, ignoring edge cases, and disregarding technical limitations. We propose four design principles to create tools that mitigate these problems: Provide multiple viewpoints , maintain a single source of truth , reveal the invisible, and support design by enaction . We applied these principles to create E NACT , a live environment for prototyping touch-based interactions. We conducted two studies to assess E NACT and to compare designer–developer collaboration with E NACT versus current tools. Results suggest that E NACT helps participants detect more edge cases, increases designers’ participation and provides new opportunities for co-creation.

Video prototypes help capture and communicate interaction with paper prototypes in the early stages of design. However, designers sometimes find it tedious to create stop-motion videos for continuous interactions and to re-shoot clips as the design evolves. We introduce Montage, a proof-of-concept implementation of a computer-assisted process for video prototyping. Montage lets designers progressively augment video prototypes with digital sketches, facilitating the creation, reuse and exploration of dynamic interactions. Montage uses chroma keying to decouple the prototyped interface from its context of use, letting designers reuse or change them independently. We describe how Montage enhances video prototyping by combining video with digital animated sketches, encourages the exploration of different contexts of use, and supports prototyping of different interaction styles.

Professional interaction designers and software developers have different trainings and skills, yet they need to closely collaborate to create interactive systems. We conducted three studies to understand the mismatches between their processes, tools and representations. Based on 16 interviews, we found that current practices induce unnecessary rework and cause discrepancies between the original design and the implementation. We identified three key design breakdowns where designers omitted critical details, ignored the presence of edge cases or disregarded technical limitations. We next observed two face-to-face meetings between designers and developers. We found that early involvement of the developer helped to mitigate potential design breakdowns but new ones emerged as the project unfolded. Finally, we ran a participatory design session with two designer/developer pairs. Both pairs had difficulty representing and communicating pre-existing interactions. Creating complete interaction descriptions required iterating from individual examples to rule-based representations. We conclude with implications for designing collaborative tools that facilitate the designer's ability to express and the developer's ability to implement complex interactive systems.