We present Editing Reality, a mixed reality system that enables in-situ co-creation with generative AI directly within physical environments. Rather than treating generation as a one-shot command, the system supports embodied and iterative creation through speech, sketching, and direct manipulation, allowing users to generate, modify, erase, and retexture real-anchored virtual and reconstructed scene elements in place. Using a Research Through Design approach, we investigate how co-creation unfolds through iterative system development, a formative workshop, and expert review. From this process, we articulate a set of designerly framings that characterize in-situ co-creation as a negotiated, spatial, and temporal practice shaped by previews, accumulation, waiting, embodied evaluation, and learning the system as a spatial actor. We instantiate these ideas in a working system and report expert feedback highlighting both its creative potential and its design implications. Our work contributes a conceptual lens for understanding generative AI in mixed reality not as a one-shot automation tool, but as part of an embodied, situated creative process.

Physical computing engages learners in computational thinking through building interest-driven physical interactive projects. However, traditional electronic approaches require deep understanding of circuits and embedded systems — a challenge for young learners before meaningful engagement with computational concepts. This pictorial demo- nstrates an alternative, computer vision (CV) approach using fiducial markers and craft materials, through a workshop model where learners design custom controllers for video games. With learner stories from workshops with learners aged 10-15, and emerging themes of computational, material, and social moments, we illustrate how CV-driven physical comp-uting can be an effective, low-barrier, introduction to computing that preserves the creative learning benefits of physical computing.

Tangible User Interfaces (TUIs) that integrate digital information with physical interaction require specialized hardware and complex calibration, limiting their adoption in portable or mobile display systems. This paper introduces ArUcoTUI, a computer vision (CV) toolkit for prototyping tangible interactions on portable screens, leveraging standard cameras and the OpenCV library. ArUcoTUI uses ArUco fiducial markers to detect physical inputs. The software toolkit offers streamlined calibration, a signal processing pipeline, and a client application that translates tangible input into structured events for use in HCI applications. Using a conventional camera in a top-down setting with a flat-panel display, we demonstrate how this toolkit supports the development of interactive surface TUIs with advanced features, including 3D spatial interaction, multi-device interaction, and actuated tangibles within applications. We describe the software implementation, which utilizes accessible hardware to support the development of these tangible interactions. We provide the results of a preliminary evaluation with users, including design implications and suggestions for future research and development.

Outdated Computer Vision (CV) toolkits for Tangible User Interfaces (TUI) have led to fragmented practices, diminished reproducibility, and reduced community support. This paper examines the past, present, and future trajectory of CV-TUI toolkits. Our scoping review of ACM literature (N=120) reveals a divergence between applications using the limited interactions of established toolkits like ReacTIVision and the fragmented, bespoke systems built for complex interactions, highlighting the need for advanced toolkits that enable accessible making. We found how ReacTIVision-based projects were primarily used in stationary tabletop settings with standard token events, while non-ReacTIVision projects explored unique interaction techniques in diverse configurations. Reflecting on these insights, we offer six suggestions for building the next-generation CV-TUI toolkits to support rapid prototyping of tangible interactions. This study provides the TUI community with an updated perspective to inform future research.

Character animation remains challenging for novices and children despite advances in digital tools. While recent tangible interfaces have lowered barriers by enabling creators to animate their drawings on paper, they are limited to preset animation sequences and support for only human-like characters. We present Rig-a-Doodle, a tangible kit and web application for open-ended character rigging animation, where creators can draw any character and construct a custom physical rig using everyday materials to animate it. This work-in-progress contributes a system of tangible interaction to animate hand-drawn characters by direct physical manipulation of custom rigs in real-time. We share findings from a preliminary workshop with adults to explore the kinds of expressive animation the kit enables, discover issues with interaction, and source ideas for future directions.

Building on the growing interest in technology-supported dance practice, neural imaging offers novel opportunities to reveal dancers’ internal states and expand the possibilities for augmented, embodied interaction. Despite advances in social neuroscience, the exploration of dance through brain imaging remains limited by technical challenges. To overcome these barriers, we developed and validated a real-time vibrotactile biofeedback system based on inter-brain coupling (IBC) measures from tango dancers using a mobile, synchronous multi-brain EEG system. We first conducted an empirical study recording synchronized EEG and motion data to test whether behavioral synchronization enhances inter-brain coupling. Insights from this study informed the design of our tangible neurofeedback system, which experienced dancers evaluated. Our findings support the Synchronicity Hypothesis of Dance and demonstrate how embodied technologies can enhance collective dance practice. This work introduces a novel methodological and interaction paradigm, bridging neural measurement with wearable feedback for socially situated embodied experiences.

Traditional carpentry faces a critical shortage of skilled workers due to limited opportunities for potential apprentices to access onsite woodworking experience. Through expert interview, we learned the importance of hammering sound to judge the precision in Kigumi assembly, as master carpenters rely on differences between soft sound and sharp sound without relying on visual. This paper presents Sound of Kigumi (SoK), a playful VR system for inexperienced users to casually experience sound sensory skills through loop of hammering and chiseling. In SoK, users listen to hammering sound in relation to tightness, assess the precision of their work, and return to chiseling for further adjustments. Furthermore, SoK implements pseudo-haptic feedback by visually modifying hammering resistance based on chiseling progress. Expert evaluation indicated SoK replicates the hammering process and serves as effective introductory tool, and user feedback confirmed SoK provides an immersive woodworking experience and effective Kigumi learning.

We present Lighting the Reef, an interactive installation that uses modular 3D paper circuits to explore ecological fragility. Participants build coral structures from foldable paper blocks with copper tape and low-voltage components. When connections align, coral modules glow, metaphorically expressing the energy exchange between coral and zooxanthellae, the symbiotic algae crucial to coral metabolism. Pollution modules add resistance that dims light or interrupts the current entirely, mirroring environmental disruption. We position Lighting the Reef as a Research through Design case that articulates fragility as an interaction aesthetic and ecological metaphor. We reflect on how modular circuitry, material constraints, and embodied play make precarity tangible. We also report workshops with 15 participants that discussed themes of care, collapse, and interdependence. We contribute insights into designing for fragility with modular circuits, ecological storytelling through tangible interaction, and accessible and reproducible designs for participatory sustainability education.

Mobile video calls are widely used to share information about real-world objects and environments with remote collaborators. While these calls provide valuable visual context in real time, the experience of interacting with people and moving around a space is significantly reduced when compared to co-located conversations. Recent work has demonstrated the potential of Mobile Augmented Reality applications to enable more spatial forms of collaboration across distance. To better understand the dynamics of mobile AR collaboration and how this medium compares against the status quo, we conducted a comparative structured observation study to analyze people’s perception of space and interaction with remote collaborators across mobile video calls and AR-based calls. Fourteen pairs of participants completed a spatial collaboration task using each medium. Through a mixed-methods analysis of session videos, transcripts, motion logs, post-task exercises, and interviews, we highlight how the choice of medium influences the roles and responsibilities that collaborators take on and the construction of a shared language for coordination. We discuss the importance of spatial reasoning with one’s body, how video calls help participants “be on the same page” more directly, and how AR calls enable both onsite and remote collaborators to engage with the space and each other in ways that resemble in-person interaction. Our study offers a nuanced view of the benefits and limitations of both mediums, and we conclude with a discussion of design implications for future systems that integrate mobile video and AR to better support spatial collaboration in its many forms.

Large language models (LLMs) have enabled the automatic generation of step-by-step augmented reality (AR) instructions for a wide range of physical tasks. However, existing LLM-based AR guidance often lacks rich visual augmentations to effectively embed instructions into spatial context for a better user understanding. We present Guided Reality, a fully automated AR system that generates embedded and dynamic visual guidance based on step-by-step instructions. Our system integrates LLMs and vision models to: 1) generate multi-step instructions from user queries, 2) identify appropriate types of visual guidance, 3) extract spatial information about key interaction points in the real world, and 4) embed visual guidance in physical space to support task execution. Drawing from a corpus of user manuals, we define five categories of visual guidance and propose an identification strategy based on the current step. We evaluate the system through a user study (N=16), completing real-world tasks and exploring the system in the wild. Additionally, four instructors shared insights on how Guided Reality could be integrated into their training workflows.

EdBoard is an educational breadboard designed to facilitate early learning of physical computing in children. We designed EdBoard's curriculum based on constructionist pedagogy, and our goal is to create an engaging and interactive learning environment for children. The EdBoard curriculum comprises three components: story booklets, circuit cards, and paper builders. These elements together foster an engaging and interactive learning experience for children. These booklets provide children with a narrative context that encourages pretend play. Circuit cards can be inserted into the EdBoard, helping children understand the positions of different components within the circuit. Paper builders empower children to create functional and interactive artifacts without the EdBoard. These hands-on activities reinforce children's understanding of physical computing concepts. During the demo, we will guide the audience through the EdBoard's curriculum design, providing insights into how these elements come together to facilitate learning. Attendees will have the opportunity to experience EdBoard as an educational tool. Furthermore, we invite open discussions on the potential of the EdBoard platform to enhance K-12 computer science education. We believe that EdBoard could play an important role in making computer science education more accessible and engaging for children.

Driven by the vision of future responsive environments, where everyday surroundings can perceive human behaviors and respond through intelligent robotic actuation, we propose Wizard of Props (WoP): a human-centered design workflow for creating expressive, implicit, and meaningful interactions. This collaborative experience prototyping approach integrates full-scale physical props with Mixed Reality (MR) to support ideation, prototyping, and rapid testing of responsive environments. We present two design explorations that showcase our investigations of diverse design solutions based on varying technology resources, contextual considerations, and target audiences. Design Exploration One focuses on mixed environment building, where we observe fluid prototyping methods. In Design Exploration Two, we explore how novice designers approach WoP, and illustrate their design ideas and behaviors. Our findings reveal that WoP complements conventional design methods, enabling intuitive body-storming, supporting flexible prototyping fidelity, and fostering expressive environment-human interactions through in-situ improvisational performance.

In this demonstration, we present a system for creating and viewing spatial memories using mobile devices. The concept of interacting with three-dimensional recordings of people and places has been explored using depth cameras and Augmented and Virtual Reality headsets, but these methods require specialized hardware that has not yet reached wider availability. Recent developments in mobile camera technology have presented new capabilities which we leverage in our application for Spatial Mobile Memories. The application captures visual and depth information, together with the orientation and movement of the mobile device. This is then combined into a single standard image or video file. Users can share these files via personal messaging channels, and replay them in 2D and 3D across devices using the same application. The goal of this project is to develop open and widely-accessible systems for the creation and viewing of spatial memories that leverage the ubiquity and familiarity of mobile devices.

Interaction is critical for data analysis and sensemaking. However, designing interactive physicalizations is challenging as it requires cross-disciplinary knowledge in visualization, fabrication, and electronics. Interactive physicalizations are typically produced in an unstructured manner, resulting in unique solutions for a specific dataset, problem, or interaction that cannot be easily extended or adapted to new scenarios or future physicalizations. To mitigate these challenges, we introduce a computational design pipeline to 3D print network physicalizations with integrated sensing capabilities. Networks are ubiquitous, yet their complex geometry also requires significant engineering considerations to provide intuitive, effective interactions for exploration. Using our pipeline, designers can readily produce network physicalizations supporting selection-the most critical atomic operation for interaction-by touch through capacitive sensing and computational inference. Our computational design pipeline introduces a new design paradigm by concurrently considering the form and interactivity of a physicalization into one cohesive fabrication workflow. We evaluate our approach using (i) computational evaluations, (ii) three usage scenarios focusing on general visualization tasks, and (iii) expert interviews. The design paradigm introduced by our pipeline can lower barriers to physicalization research, creation, and adoption.

Computational thinking has been identified as an important skill for children to learn in the 21st century, and many innovative kits and tools have been developed to integrate it into children’s learning. Yet, most solutions require the use of devices like computers or other expensive hardware, thus being inaccessible to low-income schools and communities. We present Cartoonimator, a low-cost, paper-based computational kit for children to create animations and engage with computational thinking. Cartoonimator requires only paper and a smartphone to use, offering an affordable learning experience. Children can draw the scenes and characters for their animation on the paper, which is printed with computer vision markers. We developed the mobile web app to provide an interface to capture keyframes and compile them into animations. In this paper, we describe the implementation and workflow of Cartoonimator, its deployment with children at a local STEAM event, and a planned evaluation for the kit.

Data visualization literacy (DVL) is increasingly important in navigating today’s world. Young students are now required to develop data visualization skills (e.g., identifying patterns, collecting, organizing, and analyzing data). Past work focuses on cultivating children’s DVL through playful and gamified approaches. However, these past solutions require hardware (e.g., smartphones) that is not readily replicable within standard classrooms. In addition, the extent of embodied interactions found in past solutions is fairly limited to large-scale interfaces. To address these challenges, we explore how to cultivate children’s DVL through embodied learning that is more apt for a classroom environment. Using a standard Google Chromebook, we designed an interface that utilizes the tracking of a fiducial marker on a wooden arrow to allow a student to author their own bar graphs. We conducted Wizard of Oz tests (n = 3) to gather feedback on the functionality of our design. This project aims to display the benefits of giving students a mode to develop their own visualizations and broaden how they can interact with data.

The electronics-centered approach to physical computing presents challenges when designers build tangible interactive systems due to its inherent emphasis on circuitry and electronic components. To explore an alternative physical computing approach we have developed a computer vision (CV) based system that uses a webcam, computer, and printed fiducial markers to create functional tangible interfaces. Through a series of design studios, we probed how designers build tangible interfaces with this CV-driven approach. In this paper, we apply the annotated portfolio method to reflect on the fifteen outcomes from these studios. We observed that CV markers offer versatile materiality for tangible interactions, afford the use of democratic materials for interface construction, and engage designers in embodied debugging with their own vision as a proxy for CV. By sharing our insights, we inform other designers and educators who seek alternative ways to facilitate physical computing and tangible interaction design.

In-person human interaction relies on our spatial perception of each other and our surroundings. Current remote communication tools partially address each of these aspects. Video calls convey real user representations but without spatial interactions. Augmented and Virtual Reality (AR/VR) experiences are immersive and spatial but often use virtual environments and characters instead of real-life representations. Bridging these gaps, we introduce DualStream, a system for synchronous mobile AR remote communication that captures, streams, and displays spatial representations of users and their surroundings. DualStream supports transitions between user and environment representations with different levels of visuospatial fidelity, as well as the creation of persistent shared spaces using environment snapshots. We demonstrate how DualStream can enable spatial communication in real-world contexts, and support the creation of blended spaces for collaboration. A formative evaluation of DualStream revealed that users valued the ability to interact spatially and move between representations, and could see DualStream fitting into their own remote communication practices in the near future. Drawing from these findings, we discuss new opportunities for designing more widely accessible spatial communication tools, centered around the mobile phone.

Socially assistive robots play an effective role in children's therapy and education. Robots engage children and provide interaction that is free of the potential judgment of human peers and adults. Research in socially assistive robots for children generally focuses on therapeutic and educational outcomes for those children, informed by a vision of autonomous robots. This perspective ignores therapists and educators, who operate these robots in practice. Through nine interviews with individuals who have used robots to deliver socially assistive services to neurodivergent children, we (1) define a dual-cycle model of therapy that helps capture the domain expert view of therapy, (2) identify six core themes of teleoperator needs and patterns across these themes, (3) provide high-level guidelines and detailed recommendations for designing teleoperated socially assistive robot systems, and (4) outline a vision of robot-assisted therapy informed by these guidelines and recommendations that centers teleoperators of socially assistive robots in practice.

Fostering data visualization literacy (DVL) as part of childhood education could lead to a more data literate society. However, most work in DVL for children relies on a more formal educational context (i.e., a teacher-led approach) that limits children's engagement with data to classroom-based environments and, consequently, children's ability to ask questions about and explore data on topics they find personally meaningful. We explore how a curiosity-driven, child-led approach can provide more agency to children when they are authoring data visualizations. This paper explores how informal learning with crafting physicalizations through play and curiosity may foster increased literacy and engagement with data. Employing a constructionist approach, we designed a do-it-yourself toolkit made out of everyday materials (e.g., paper, cardboard, mirrors) that enables children to create, customize, and personalize three different interactive visualizations (bar, line, pie). We used the toolkit as a design probe in a series of in-person workshops with 5 children (6 to 11-year-olds) and interviews with 5 educators. Our observations reveal that the toolkit helped children creatively engage and interact with visualizations. Children with prior knowledge of data visualization reported the toolkit serving as more of an authoring tool that they envision using in their daily lives, while children with little to no experience found the toolkit as an engaging introduction to data visualization. Our study demonstrates the potential of using the constructionist approach to cultivate children's DVL through curiosity and play.

Tinycade is a platform designed to help game designers build their own mini arcade games by hand. With this platform, one can craft functioning game controllers out of everyday materials such as cardboard and toothpicks. By utilizing computer vision markers, we can create a variety of inputs without a single wire. In this pictorial, we discuss the functionality of Tinycade and showcase three games that demonstrate the variety of controls possible with this platform.

Designing a data physicalization requires a myriad of different considerations. Despite the cross-disciplinary nature of these considerations, research currently lacks a synthesis across the different communities data physicalization sits upon, including their approaches, theories, and even terminologies. To bridge these communities synergistically, we present a design space that describes and analyzes physicalizations according to three facets: context (end-user considerations), structure (the physical structure of the artifact), and interactions (interactions with both the artifact and data). We construct this design space through a systematic review of 47 physicalizations and analyze the interrelationships of key factors when designing a physicalization. This design space cross-pollinates knowledge from relevant HCI communities, providing a cohesive overview of what designers should consider when creating a data physicalization while suggesting new design possibilities. We analyze the design decisions present in current physicalizations, discuss emerging trends, and identify underlying open challenges.

Augmented Reality (AR), with its ability to make people feel like they are in the same space as friends from across the world, is an ideal medium for the purpose of Networked Musical Collaboration. Most conventional systems that enable networked musical collaboration minimize network latency by focusing on the transfer of auditory information at the expense of visual feedback. Studies into human perception have shown that sensory integration of audio and visual stimuli can take place even when there is a slight delay between the two signals. We studied the way changes in network latency effect participants’ auditory and visual perception in latency detection, latency tolerance and attention focus; in this paper, we explore the trade-off between the presence of AR visuals and the minimization of latency. Twenty-four participants were asked to play a hand drum and collaborate with a prerecorded remote musician rendered as an avatar in AR. Multiple trials involving different levels of audio-visual latency were conducted. We then analyzed the subjective responses of the participants together with the recorded musical information from each session. Results indicate a minimum noticeable delay value–defined as the highest amount of delay that can be experienced before two stimulated senses are perceived as separate events–between 160 milliseconds (ms) and 320 ms. Players also reported that a delay between sound and an accompanying avatar animation became less tolerable at 320 ms of delay, but was never completely intolerable, even up to 1200 ms of delay. We conclude that players begin to notice delay at about 320 ms and most players can tolerate large delays between sound and animation.

Fostering data literacy has largely been the domain of formal educational systems and export-oriented tools. Informal educational approaches, such as games or family activities, may overcome barriers to engaging with data by fostering data literacy through casual engagement. This work in progress explores how informal learning through creation and play with interactive data representations (physicalizations) can foster increased literacy and engagement with data. We outline a series of DIY paper charts using AR markers and everyday materials to help children interact and explore data through the creative process of making.

The Jam Station is a collaborative musical experience (CME), with embedded sensors to detect, evaluate, and reinforce collaboration among players of all skill levels. The system is also gamified, consisting of four instruments that face a centrally located, vertically oriented, display that visualizes various game states. Playing any of the instruments affect the game as musical collaboration is assessed and visualized on the display. The display also includes a “progress bar’’ which fills up the more players collaborate musically, and to “win’’ the game, the progress bar must be full, initiating a multi-color light show for the players. By creating an environment focused primarily on collaboration, we aim to support collaborative music-making for musicians of all skill levels.

This pictorial presents a design investigation at the intersection of paper and computer vision for tangible interfaces. Through this exploration, we uncovered various characteristics of paper that connect tangible interactions with concealing and revealing printed fiducial markers for detection-particularly through the affordances of paper craft and fiber. We illustrate a variety of paper structures that construct and deconstruct fiducial markers. We also demonstrate how these structures enable untethered functional physical inputs, such as push buttons and sliders. We showcase four proposals that extend these material insights into tangible interface applications, including interactive data physicalizations and functional paper prototypes. Furthermore, we continue the legacy of pictorials by exposing fabrication drawings for others to engage with this work at a more practical level.

RoomShift is a room-scale dynamic haptic environment for virtual reality, using a small swarm of robots that can move furniture. RoomShift consists of nine shape-changing robots: Roombas with mechanical scissor lifts. These robots drive beneath a piece of furniture to lift, move and place it. By augmenting virtual scenes with physical objects, users can sit on, lean against, place and otherwise interact with furniture with their whole body; just as in the real world. When the virtual scene changes or users navigate within it, the swarm of robots dynamically reconfigures the physical environment to match the virtual content. We describe the hardware and software implementation, applications in virtual tours and architectural design and interaction techniques.

This pictorial presents our material-driven inquiry into carbon-coated paper and kirigami structures. We investigated two variations of this paper and their affordances for tangible interaction; particularly their electrical, haptic, and visual aspects when shaped into three-dimensional forms through cutting, folding, and bending. Through this exploration, we uncovered distinct affordances between the two paper types for sensing folds and bends, due to differences in their material compositions. From these insights, we propose three applications that showcase the possibilities of this material for tangible interaction design. In addition, we leverage the pictorial format to expose working design schematics for others to take up their own explorations.

Recent innovations in fashion and smart textiles have contributed new visions of wearable computing, addressing the body through the cultural and social self. In this work, we draw on speculative design, maker technologies, and zoomorphism to explore how wearables might support sociability, and present Hooze, a fashion accessory that entices touch through its zoomorphic qualities and visual appearance. We describe our design and prototyping process, and reflect on how Hooze inspires transformative designs of wearables.

We present Mechamagnets, a technique for facilitating the design and fabrication of haptic and functional inputs for physical interfaces. This technique consists of a set of 3D printed spatial constraints which facilitate different physical movements, as well as unpowered haptic profiles created by embedding static magnets in 3D printed parts. We propose the Mechamagnets taxonomy to map the design space of this technique for designers and makers. Furthermore, we leverage the use of magnets by instrumenting these objects with linear Hall effect sensors to create functional digital inputs. We showcase Mechamagnets with a series of novel physical interfaces made with this technique.

We introduce shape-changing swarm robots. A swarm of self-transformable robots can both individually and collectively change their configuration to display information, actuate objects, act as tangible controllers, visualize data, and provide physical affordances. ShapeBots is a concept prototype of shape-changing swarm robots. Each robot can change its shape by leveraging small linear actuators that are thin (2.5 cm) and highly extendable (up to 20cm) in both horizontal and vertical directions. The modular design of each actuator enables various shapes and geometries of self-transformation. We illustrate potential application scenarios and discuss how this type of interface opens up possibilities for the future of ubiquitous and distributed shape-changing interfaces.

In the same way that we experience the real-world through a range of senses, experiencing a virtual environment through multiple sensory modalities may augment both our presence within a scenario and our reaction to it. In this paper, we present Season Traveller, a multisensory virtual reality (VR) narration of a journey through four seasons within a mystical realm. By adding olfactory and haptic (thermal and wind) stimuli, we extend traditional audio-visual VR technologies to achieve enhanced sensory engagement within interactive experiences. Using both subjective measures of presence and elicited physiological responses, we evaluated the impact of different modalities on the virtual experience. Our results indicate that 1) the addition of any singular modality improves sense of presence with respect to traditional audio-visual experiences and 2) providing a combination of these modalities produces a further significant enhancement over the aforementioned improvements. Furthermore, insights into participants' psychophysiology were extrapolated from electrodermal activity (EDA) and heart rate (HR) measurements during each of the VR experiences.

This paper presents Mechamagnets, a technique to rapidly prototype tactile mechanisms for tangible interfaces. We demonstrate how to embed different passive tactile mechanisms in physical systems through a combination of magnets and digitally fabricated parts. We also discuss how DIY materials and 3-axis magnetometers can instrument Mechamagnet interfaces into functional prototypes.

This demonstration presents ``Tainted', a first-person 3D adventure game that enables players to experience a traditional horror folklore using visual-auditory-olfactory interactions. The system consists of a computer to play the game and a smell-emitting module that contains jasmine, pineapple, mango, and banana scents. The information provided by the additional smell stimuli delivers an immersive gaming experience by providing game narrative and signaling predefined game conditions (e.g. imminent threat). Smell stimuli may also help to enhance player's cognition and memory, as well as to invoke emotional responses such as fear and curiosity.

Laser cutting is widely used by industrial designers and mechanical engineers as a rapid modeling tool. However, designing and fabricating laser cut assemblies can be a complex and tedious process, especially for novice designers. Through our research, we developed Joinery, a parametric joint generation tool for laser cut assemblies. Through Joinery, designers simply define connections between parts of an assembly, while the system generates the joints. Joinery supports fabrication-aware design through six different joint profiles that cater to different material and design needs. In this paper, we illustrate the use of Joinery as a creativity support tool in an industrial design process, and present several artifacts resulting from the tool. In addition, we discuss our findings from deploying this system in a college-level industrial design class.

In this paper, we present and evaluate Ambiotherm, a wearable accessory for Head Mounted Displays (HMD) that provides thermal and wind stimuli to simulate real-world environmental conditions, such as ambient temperatures and wind conditions, to enhance the sense of presence in Virtual Reality (VR). Ambiotherm consists of a Ambient Temperature Module that is attached to the user's neck, a Wind Simulation Module focused towards the user's face, and a Control Module utilizing Bluetooth communication. We demonstrate Ambiotherm with two VR environments, a hot desert, and a snowy mountain, to showcase the different types of simulated environmental conditions. We conduct several studies to 1) address design factors of the system and 2) evaluate Ambiotherm's effect on factors related to a user's sense of presence. Our findings show that the addition of wind and thermal stimuli significantly improves sensory and realism factors, contributing towards an enhanced sense of presence when compared to traditional VR experiences.

Similar to the concept of a cocktail or mocktail, we present Vocktail (a.k.a. Virtual Cocktail) - an interactive drinking utensil that digitally simulates multisensory flavor experiences. The Vocktail system utilizes three common sensory modalities, taste, smell, and visual (color), to create virtual flavors and augment the existing flavors of a beverage. The system is coupled with a mobile application that enables users to create customized virtual flavor sensations by configuring each of the stimuli via Bluetooth. The system consists of a cocktail glass that is seamlessly fused into a 3D printed structure, which holds the electronic control module, three scent cartridges, and three micro air-pumps. When a user drinks from the system, the visual (RGB light projected on the beverage), taste (electrical stimulation at the tip of the tongue), and smell stimuli (emitted by micro air-pumps) are combined to create a virtual flavor sensation, thus altering the flavor of the beverage. In summary, this paper discusses 1) technical details of the Vocktail system and 2) user experiments that investigate the influences of these multimodal stimuli on the perception of virtual flavors in terms of five primary tastes (i.e. salty, sweet, bitter, sour, and umami). Our results suggest that the combination of these stimuli delivers richer flavor experiences, as compared to separately simulating individual modalities, and indicates that the types of pairings that can be formed between smell and electric taste stimuli.

This paper presents a novel methodology to digitally share the flavor experience of a glass of lemonade (or other similar beverages) remotely. The proposed method utilizes a sensor to capture valuable information (primarily, the color and the corresponding pH value) of the lemonade and a customized tumbler to virtually simulate these properties using plain water. Thus, the system consists of three main components: 1) the lemonade sensor, 2) the communication protocol, and 3) a customized tumbler, acting as the lemonade simulator. Initially, the sensor captures the color and the pH value of the lemonade and encodes this information based on an established communication protocol for wireless transmission. On receiving the information from the sensor, the lemonade simulator overlays the color of the drink on plain water using an RGB Light Emitting Diode (LED) and simulates sour taste sensations on the user's tongue via electrical stimulation. An experimental study was conducted to evaluate this novel approach of digitally teleporting a glass of lemonade: 1) to assess the pre-taste perceptions based on the user's visual perceptions of the colors (real vs. virtual lemonade) and 2) to assess the taste sensations (real vs. virtual lemonade). By simulating the experience of drinking a glass of lemonade through the digital reconstruction of the beverage's main visual and taste factors, the results from these experiments will be able to justify the feasibility of teleporting a glass of lemonade using this novel methodology.

Among the five primary senses, the sense of taste is the least explored as a form of digital media applied in Human--Computer Interface. This article presents an experimental instrument, the Digital Lollipop, for digitally simulating the sensation of taste (gustation) by utilizing electrical stimulation on the human tongue. The system is capable of manipulating the properties of electric currents (magnitude, frequency, and polarity) to formulate different stimuli. To evaluate the effectiveness of this method, the system was experimentally tested in two studies. The first experiment was conducted using separate regions of the human tongue to record occurrences of basic taste sensations and their respective intensity levels. The results indicate occurrences of sour, salty, bitter, and sweet sensations from different regions of the tongue. One of the major discoveries of this experiment was that the sweet taste emerges via an inverse-current mechanism, which deserves further research in the future. The second study was conducted to compare natural and artificial (virtual) sour taste sensations and examine the possibility of effectively controlling the artificial sour taste at three intensity levels (mild, medium, and strong). The proposed method is attractive since it does not require any chemical solutions and facilitates further research opportunities in several directions including human--computer interaction, virtual reality, food and beverage, as well as medicine.

The Penan people of Malaysian Borneo were traditionally nomads of the rainforest. They would leave messages in the jungle for each other by shaping natural objects into language tokens and arranging these symbols in specific ways -- much like words in a sentence. With settlement, the language is being lost as it is not being used by the younger generation. We report here, a tangible system designed to help the Penan preserve their unique object writing language. The key features of the system are that: its tangibles are made of real objects; it works in the wild; and new tangibles can be fabricated and added to the system by the users. Our evaluations show that the system is engaging and encourages intergenerational knowledge transfer and thus has the potential to help preserve this language.

Flavor is often a pleasurable sensory perception we experience daily while eating and drinking. However, the sensation of flavor is rarely considered in the age of digital communication mainly due to the unavailability of flavors as a digitally controllable media. This paper introduces a digital instrument (Digital Flavor Synthesizing device), which actuates taste (electrical and thermal stimulation) and smell sensations (controlled scent emitting) together to simulate different flavors digitally. A preliminary user experiment is conducted to study the effectiveness of this method with predefined five different flavor stimuli. Experimental results show that the users were effectively able to identify different flavors such as minty, spicy, and lemony. Moreover, we outline several challenges ahead along with future possibilities of this technology. In summary, our work demonstrates a novel controllable instrument for flavor simulation, which will be valuable in multimodal interactive systems for rendering virtual flavors digitally.

In a piano lesson, a student often imitates the teacher--s playing in terms of speed, dynamics, and fingering. While this learning model leverages one's visual and even audial perception for emulation, it still lacks an important component of piano playing -- the tactile sensation. We seek to convey the tactile sensations of the teacher's keystrokes and then signal the student's corresponding fingers. We implemented an instrumented fingerless glove called Tactile Teacher to detect finger taps on hard surfaces. Since finger taps generate acoustic signals and cause vibrations, we embedded three vibration sensors on the glove and use machine learning algorithms to analyze the data from the sensors. After a brief training procedure, this prototype can accurately identify single finger tap in a very good performance at above 89% accuracy, and two finger taps resulted in accuracy around 85%.

Digital games as education tools for children have been studied in the past. However, the use of digital games in clinical environments such as for children's healthcare is still rare in the research community. This paper reports on the development of a digital tablet game called "Dodo's catching adventure" which examines the use of games in visual colordeficiency screening for young children. A user study was conducted at a National Eye Centre. Results of the study show that the digital game demonstrates sensitivity and specificity on RedGreen color deficiency detection, and is comparable to the two gold standards in color deficiency tests, namely the Ishihara and Farnsworth D15. Furthermore, children found the game to be more enjoyable than the Ishihara test. This provides evidence for the feasibility of using such games as diagnosis tools for early childhood health conditions.

This paper presents 'DoDo's Catching Adventure,' a new color vision deficient screening test for young children. Early detection of color blindness among children is useful for parents and teachers to better understand children's needs, to overcome difficulties in learning, and for life and career planning. Unfortunately, current color screening tests are not designed for young children; most require more advanced verbal or cognitive skills. DoDo game has taken a new approach by embedding game elements into a color vision screening test. A user study conducted at Singapore National Eye Centre on twenty-eight children, identified fourteen as Red-Green deficient subjects as did by Ishihara screening test, showed that DoDo was adequately effective in identifying Red-Green color vision deficiency and comparable to two current gold standard colorblind tests, Ishihara and D15.

The age of ubiquitous/pervasive/ambient computing is upon us. We see more and more connected objects and devices embedded in everyday life. Design and Human-Computer Interaction are crucial components of information technologies that color our experience. As designers and technologists, we have the unique opportunity to imagine, design and create interesting, intelligent and interactive technologies for a smart living environment. A smart living environment is responsive, reconfigurable and transformable, embedded with sensors and actuators, to support everyday happy, healthy living with things that think, spaces that sense and places that play. Specifically, we see opportunity for investigating creative design computing to consider the built environment as an interface. We aim to engage people in playful, creative ways in which computing technologies embedded in the built environment (e.g., objects, furniture, building, and space) can support everyday happy healthy living. We build wellness technologies in different scales (e.g., in human-centered view: hand, body and environment), for various applications (e.g., to support emotional, physical, and intellectual wellness) to unlock human potential and augmenting human capabilities through digital design innovations. Opportunities exist for integrating multidisciplinary perspectives together to create innovation with impacts. For example, the Mobile Music Touch, a light-weight fingerless instrumented glove provides passive haptic learning of piano playing but also works as an effective rehabilitation tool. The ClockMe system, not only converts traditional pencil-and-paper Clock Drawing Test on a digital tablet for automatic scoring and analysis for detection of Alzheimer's disease and related disorders but also provides more information of the drawing behavior such as drawing sequence and pressure. The Digital Box and Block Test employs image processing to automatically detect and record a clinically validated post-stroke rehabilitation assessment for in-home use and a tangible gaming system to increase patient motivation. The Dodo game provides pictures of cute animals for color matching, and serves as a clinical screening test to detect color deficiency in young children. The Taste + bottle and spoon provide stimulation to alter the sense of taste without any chemical flavoring using electric pluses and color LED lights. It could encourage water drinking or enhance taste for people with restrictive diets or diminished taste sensations. The Sensorendipity is a real-time smartphone-based web-enabled sensor platform for designers to create sensor-based applications such as monitoring activities, in exergaming or safe driving. Now is an exciting time to engage in creative design computing, to implement physically and computationally enhanced environment, to explore experience media, to build prototypes, towards a smart living environment. Advancing technology offers new ways to solve problems, discover opportunities, and create new objects and experience that delight our senses and improve the way we live and work. Let's begin with the spark of creativity and enthusiasm and follow up with design and computational thinking towards the goal of creating unique technology for everyone.

In recent years, digital media technologies expand horizons into non-traditional sensory modalities, for instance, the sense of taste. This demonstration presents `taste+,' which digitally improves the taste sensations of food and beverages without additional flavoring ingredients. It primarily utilizes weak and controlled electrical pulses on the human tongue to enhance sourness, saltiness, and bitterness of food or beverages. Taste+ consists of two prototype utensils, a bottle and a spoon. Both utensils have embedded electronic control modules to achieve enhanced taste sensations. These modules apply controlled electrical pulses on the tongue through silver electrodes attached to the mouth pieces. Furthermore, different superimposed colors symbolize distinct taste sensations, lemon green represents sour, ocean blue is salty, red for red wine bitter. The initial experimental results suggested that sourness and saltiness are the main sensations that could be evoked while bitterness has comparatively mild responses. Furthermore, we also observed several mixed sensations such as salty-sour together.

In this paper, we describe an interactive drinking platform, FunRasa, which digitally expands one's drinking experience. The system uses two methods: electrical stimulation on user's tongue and superimpose virtual color onto the actual drink, to expand the taste sensations of the drink. The system consists of a glass cup, an electronic platform with RGB LEDs, and a specially designed straw interface with two silver electrodes. When a user uses the straw, his tongue touches the silver electrodes (both lower and upper surfaces of the tip of the tongue) and is thus electrically stimulated when drinking. The user has the freedom to change the virtual color of the drink along with the magnitude of the current using two mechanical dials. Furthermore, we present several initial discussions on the user experience through a workshop that we conducted and several future usage scenarios of this technology.

We present the beginnings of a Cognitive Theory of Creativity Support aimed specifically at understanding novices and their needs. Our theory identifies unique difficulties novices face and reasons that may keep them from engaging in creative endeavors, such as fear of failure, time commitment, and lack of skill. To test our theory, we use it to analyze existing creativity support tools from multiple domains. We also describe the design and initial implementation of a creativity support tool based on our theory. The creativity support tool, called StorySketch, is designed to empower storytellers without graphical skills to engage in visual storytelling.

We present the concept and implementation of a sketching game called Sketch Master. Sketch Master is a game designed to help players learn and practice drawing from memory. The architecture of the tool and its various game modes are presented. Additionally, we describe how functions in Sketch Master serve as a research instrument to collect exploratory data about the relation between perception, memory, and sketching.

This paper proposes a new model of perception called Perceptual Logic and applies it to the domain of art to understand artistic style. We describe style in terms of affordances, or ways in which an artist can interact with and contribute to an artwork. Different types of Perceptual Logic are found to influence the perceived affordances of an artwork. We present a computational collaborative art program called PERLOGICART that uses a computational model of Perceptual Logic to learn an artist's style through collaboration. The research is conducted using a practice-based method --- we are working on building an interactive tool to support the making of artworks and the understanding of the creative process at the same time. PERLOGICART is a compelling interactive artwork as well as a valuable research tool that records and categorizes the creative process in a systematic manner.

We are interested in the usability of haptic hand gestures in emotional online communication. To investigate how different emotions are associated with different tactile hand gestures, we built a tactile user interface device to record parameters of hand gestures exerted on it such as the hand gesture intensity, temporal frequency, spatial frequency and pattern correlation to be used as the source of information to access emotions. We observed the behavioral aspects of tactile hand gestures used for emotional interaction through the sensory input device and analyzed the data using a model-based analysis tool. The obtained gestures are categorized into the groups defined by the tactile signal parameters. Based on the findings of this research, we show a possibility that the tactile hand gesture-to-emotion map can be used to enable affective on-line communication. We are applying the relationships between tactile hand gestures and emotions to emotional online communication devices.

Young children often feel anxious when visiting the doctors. This paper presents mediPuppet, an interactive companion with an exploratory procedure map designed to help children feel more relaxed and comfortable during their medical procedure. The ultimate goal of this study is to transform an intimidating stressful situation into a joyful exploring game for children during hospital visits.

The TEI Work-in-Progress workshop invited submissions about designing, making, studying, exploring and experiencing of projects on tangible, embedded and embodied interaction. The workshop aimed to provoke, intrigue, and inspire the TEI community by sharing work-in-progress practice, late breaking innovation, cutting-edge research, and controversial and compelling work. It provided a venue for eliciting feedback and fostering discussions and collaborations among TEI colleagues. The TEI Work-in-Progress workshop provided an opportunity for members of the TEI community with common and diverse interests to meet in the context of a focused and interactive discussion and a showcase for exciting new work that is still at an early stage.

This paper presents computational support for the Clock Drawing Test (CDT), a simple cognitive dysfunction diagnosis tool. Despite its popularity of current use, the CDT has been administered the same way for the past three decades as a pencil-and-paper test. By making a computerized drawing test, we can understand the process of clock drawing construction rather than simply analyzing the final drawing. In this paper, we will first introduce our computerized Clock Drawing Test, the ClockReader. Then, we will describe two automated data-capturing methods with specific data examples. The automated data analysis can provide neuropsychologists with useful qualitative information, such as the process of drawing, as well as patients' planning strategies.

This paper presents a handheld mobile game, Grocery Hunter that encourages children to take on healthy eating habits. Children can use a pocket PC to play the Grocery Hunter game to learn about food nutrition and healthy food choices. Childhood obesity in the United States has already reached epidemic proportions. The best way to help children attain and maintain healthy weight is through physical activity and nutritious eating. Our design addresses nutrition directly by teaching children healthy eating habits using an interactive game in the grocery store.

Mobile Music Touch (MMT) helps teach users to play piano melodies while they perform other tasks. MMT is a lightweight, wireless haptic music instruction system consisting of fingerless gloves and a mobile Bluetooth enabled computing device, such as a mobile phone. Passages to be learned are loaded into the mobile phone and are played repeatedly while the user performs other tasks. As each note of the music plays, vibrators on each finger in the gloves activate, indicating which finger is used to play each note. We present two studies on the efficacy of MMT. The first measures 16 subjects' ability to play a passage after using MMT for 30 minutes while performing a reading comprehension test. The MMT system was significantly more effective than a control condition where the passage was played repeatedly but the subjects' fingers were not vibrated. The second study compares the amount of time required for 10 subjects to replay short, randomly generated passages using passive training versus active training. Participants with no piano experience could repeat the passages after passive training while subjects with piano experience often could not.

We present a novel play interface called Blow Painting that enables people to create leaf collage paintings on a digital canvas by blowing at a toy windmill. The toy windmill detects the blowing action and direction with its embedded microphone and rotation sensor. Four kinds of leaf "stencils" (Red Maple, Sugar Maple, Hickory and White Oak) are embedded in the transparent RFID sheets to be placed under the windmill for the play interaction. We describe the motivation, system components, an informal pilot test, and discuss directions for future work.

This paper presents Tangible Tracking Table, an interactive tabletop display, and Optical Chess, a strategy game. We discuss the design and implementation of both systems and report our evaluation game play sessions with young adults, with a special focus on how the Tangible Tracking Table enhances interaction over a point-and-click interface.

Children and adolescents are at an age where they are beginning to gain autonomy over choosing the foods they eat, yet may not have adequate support or information to make informed choices. This paper describes the design of a heuristic-based health game called MunchCrunch to help this age group learn more about healthy and unhealthy foods to develop balanced eating habits.

We developed Escape Machine, a puzzle game in which children control the behavior of characters in a maze by manipulating a tangible state machine built with Posey, our computationally-enhanced hub-and-strut construction kit. We observed children ages nine to eleven playing the game in several sessions. The qualitative results of this observation validate the promise of Posey and Escape Machine to engage children in manipulating algorithmic specifications for behavior.

Prior knowledge and domain expertise are key to successful design and creativity. New digital workspaces should support the management of ever-increasing amounts of digital information and design artifacts as well as support collaboration during the creative process. In this paper, we present the findings of an observational study describing how expert designers organize and utilize information in their design workspaces throughout the initial creative phases of problem-structuring and high level design. We describe how these findings could inform further study in mixed reality creative environments.

We present two empirical studies of two pairs of students collaborating on two small product design sessions in both face-to-face and distributed settings while using computer-mediated communication technologies and a collaborative virtual environment. The study shows that teams spent more time working together when using programs that support shared sketching capabilities or shared viewing of 3D objects.

Blow Painting is a novel play interaction that enables children to blow at a toy windmill to create collage paintings of leaves on a digital canvas. The windmill is embedded with a microphone (to detect the blowing action) and a rotation sensor (to detect the blowing direction). Four kinds of leaves (Red Maple, Sugar Maple, Hickory and White Oak) are embedded inside transparent RFID sheets that can be placed under the windmill to provide different leaf "stencils" for the Blow Paining game.

In this paper, we describe the design, execution, and results of a study of the effects of alcohol consumption on creativity. We are specifically interested in myths surrounding alcohol and creativity; one's view of self as a creative being; and the effects of alcohol on inhibition and perception of creativity.

We present an activity we developed to demonstrate bottom-up form construction, the human hive. Participants team up to construct a hive structure from large interlocking cardboard blocks. Each participant is given a visual rule that describes where new cells should be added to the hive. The design of these rules guides the form of the structure that emerges from this uncoordinated activity. Bottom-up, distributed methods for specifying physical forms and behaviors are central to the emerging field of architectural robotics that deals with designing objects composed of self-reconfiguring materials.

This project studied the influence of kinesthetic intelligences on creativity in young children. To understand this relationship preschoolers were observed in their daycare setting during story-time over a period of four sessions. During the sessions observations focused on the children's ability to propose narratives when the teacher used different storytelling methods. The children's responses were videotaped and coded later for analysis. The coding was used to calculate the baseline Ideational Fluency of the children. Total fluency was calculated by adding the number of popular and original responses. Behavior patterns also emerged from the study. Children generated not only more responses, but a greater variation in narratives when they were allowed to interact with tangible characters and act out behaviors. Also, children generated more responses when they were given 3D stimuli than when given 2D stimuli. The degree of familiarity with the scenario also influenced the number of responses. The more familiar the children were with the scenario, their ability to contribute to the narrative improved.

The paper presents the analysis of nine architects design processes with different design experience and expertise levels. The main goal is to visualize and analyze patterns between their design processes, design phases, design iterations, and software used to support creative process in every design phase. This study focused on four major components for analysis: lengths of design processes, naming of design phases, flows of design directions, and the number and types of software used in each phase. The future goal is to investigate whether specific software capabilities would increase or hinder creative behavior patterns during the design process.

This paper presents an empirical study of two student pairs collaborating on two small products design session in both face-to-face and distributed settings while using computer-mediated communication (CMC) technologies and a collaborative virtual environment (CVE). To gain insight about the way designers communicate and collaborate, the observation focused on how much time the students worked together and individually in the design process. The study shows that teams worked together more in the distributed setting than in the face-to-face setting. In the post-test questionnaires, participants reported that they found the distributed setting a more engaging environment to work with teammates. Findings of the study suggest plausible design criteria for a communication system for distributed collaboration that supports interaction and sharing design information.

Easigami is a novel tangible user interface (TUI) and interactive system intended to help children to learn to fold 3D geometric forms and to explore 2D-3D transformations. We present the design of Easigami's physical interface: a reconfigurable system of thin flat polygon pieces connected by electronically instrumented hinges. Using the Easigami TUI as an input device, we have developed early prototype applications to develop children's visualization and spatial recognition skills. We discuss our experience in integrating traditional craft with computation, which may inform future tangible and graspable user interface design.

We present a methodology for building objects-to-think-computationally-with and illustrate its application in developing our Escape Machine game. The input mechanism for this game is a tangible state machine built with Posey, our computationally enhanced construction kit. Through manipulating this state machine children create an algorithmic specification for the behavior of both the avatar and its enemies in an attempt to navigate a maze without being eaten. We outline several strategies for success at Escape Machine and discuss how it embeds an important computational thinking concept in interaction with a tangible device.

We describe Posey, a computationally-enhanced hub-and-strut construction kit for learning and play. Posey employs a ball and socket connection that allows users to move the parts of an assembled model. Hubs and struts are optocoupled through the ball and socket joints using infrared LEDs and photosensors. Wireless transmitters in the hubs send connection and geometry information to a host computer. The host computer assembles a representation of the physical model as the user creates and configures it. Application programs can then use this representation to control computational models in particular domains.

We have, with our students, engaged in cross-disciplinary research in design. We describe parameters and principles that we have found helpful in organizing and conducting this kind of work. A variety of projects that have been developed in our group illustrate these parameters and principles. Our group focuses on making and we have come to see creativity as grounded in the ability to make things.

Flow selection is a time-based modeless selection and operation technique for freehand drawing and sketch tools. We offer flow selection as a modeless technique to address the observation that modal selection requires too much cognitive effort and causes breakdowns in creative flow. Flow selection provides input to a new class of operations by assigning increasing, fractional selection strengths to objects over time. We discuss the current prototype system and possible applications for this novel technique for interacting with sketches.

The Navigational Blocks project demonstrates a tangible user interface that facilitates retrieval of historical stories in a tourist spot. Orientation, movement, and relative positions of physical Blocks support visitor navigation and exploration in a virtual gallery. The Navigational Blocks system provides a physical embodiment of digital information through tactile manipulation and haptic feedback. The simple cubic form of the Blocks is easy to understand and therefore easy to use to manipulate complex digital information. Electromagnets embedded in the Blocks and wireless communication encourage users to quickly rearrange the Blocks to form different database queries.

This paper reports on our progress and findings in building a Web annotation system for non-immersive 3D virtual environments. Over the last two years, we developed and tested two systems for collaborating designers to comment on virtual 3D models. Our first system, Redliner [12] lets design team members browse and leave text annotations on surfaces in three-dimensional models. Experience with Redliner, including two user evaluations in different settings, led us to develop Space Pen [13], a second annotation system with improved interaction capabilities. It goes beyond the post-it note metaphor, allowing users to draw in and on the virtual environment.

Interfaces for conceptualand creative design should recognize and interpret drawings.They should also capture users' intended ambiguity, vagueness, and imprecision and convey these qualities visually and through interactive behavior.Freehand drawing can provide this information and it is a natural input mode for design.We describe a pen-based interface that acquires information about ambiguity and precision from freehand input, represents it internally, and echoes it to users visually and through constraint based edit behavior.

Architectural designers are voracious consumers of visual images, which play a crucial role especially in conceptual and creative design. Consequently architectural education revolves around visual references. Yet key word, texture and color retrieval schemes do not suit designers' needs. Designers need shape based retrieval that is driven by free hand drawing, and ways to integrate retrieved images into their design environment. The paper describes Drawing Analogies, an image retrieval scheme for design based on the need for image retrieval that can be integrated with the act of free hand drawing. Our scheme does not perform feature extraction of stored images, but relies on users to create an index of diagram keys.