Virtual reality (VR) has become an increasingly popular way for learning and training. The assessment of the amount of mental effort, or cognitive load required to perform a task, is essential to create adaptive VR experiences. In this work, we conducted a large-scale study (N=738) to collect behavioral and physiological measures under different cognitive load conditions in a VR environment, and developed a novel machine learning solution to predict cognitive load in real time. Our model predicts cognitive load as a continuous value in the range from 0 to 1, where 0 and 1 correspond to the lowest and highest reported cognitive loads across all participants. On top of the point estimation of cognitive load, our model quantifies prediction uncertainty using a prediction interval. We propose a novel dual-branch attention model to accurately predict the cognitive load. We achieve a MAE (Mean Absolute Error) of 0.11. The result indicates that, with a combination of behavioral and physiological indicators, we can reliably predict cognitive load in real-time, without calibration. To support further research, we are releasing a test dataset comprising data from 100 participants for use by researchers and developers interested in machine learning, virtual reality, learning & memory, cognition, or psychophysiology. This dataset includes recordings from multiple sensors (including pupillometry, eye-tracking, and pulse plethysmography), self-reported cognitive effort, behavioral task performance, and demographic information on the sample.

Haptic feedback in VR is important for realistic simulation in virtual reality. However, recreating the haptic experience for hand tools in VR traditionally requires hardware with precise actuators, adding complexity to the system. We propose Ungrounded Haptic Retargeting, an interaction technique that provides a realistic haptic experience for grabbing tools using only passive mechanisms. This technique leverages the ungrounded feedback inherent in grabbing tools combined with dynamic visual adjustments of their position in virtual reality to create an illusion of physical presence for virtual objects. To demonstrate the capabilities of this technique, we created VR Grabbers, an exemplary passive VR controller, similar to training chopsticks, with haptic feedback for precise object selection and manipulation. We conducted two user studies based on VR Grabbers. The first study probed the perceptual limits of the illusion; we found that the maximum position difference between the virtual and physical world acceptable to the user is (-1.48, 1.95) cm. The second study showed that task performance of the VR Grabbers controller with Ungrounded Haptic Retargeting enabled outperforms the same controller with Ungrounded Haptic Retargeting disabled.

Electric Agents is a multi-player game that uses augmented reality on mobile phones to facilitate in-room collaboration and promote joint media engagement around interactive educational television content in the home. Joint media engagement describes 'collaborative' consumption of media and has potential learning benefits. Our research examines whether technology can support joint media engagement, through the use of a series of prototypes that combine handheld augmented reality with web-based video. In pilot testing, sibling play patterns showed evidence of the kind of in-room collaboration and joint media engagement that previous research has shown to facilitate children in making sense of media content.

We introduce People in Books with FlashCam technology, a system that supports children and long-distance family members to act as characters in children's storybooks while they read stories together over a distance. By segmenting the video chat streams of the child and remote family member from their background surroundings, we create the illusion that the child and adult reader are immersed among the storybook illustrations. The illusion of inhabiting a shared story environment helps remote family members feel a sense of togetherness and encourages active reading behaviors for children ages three to five. People In Books is designed to fit into families' traditional reading practices, such as reading ebooks on couches or in bed via netbook or tablet computers. To accommodate this goal we implemented FlashCam, a computationally cost effective and physically small background subtraction system for mobile devices that allows users to move locations and change lighting conditions while they engage in background-subtracted video communications. A lab evaluation compared People in Books with a conventional remote reading application. Results show that People in Books motivates parents and children to be more performative readers and encourages open-ended play beyond the story, while creating a strong sense of togetherness.

To effectively design for families, we must understand familial relationships, which exert a significant influence on children's growth, learning, and play. In particular, siblings can be influential play partners and teachers, providing important scaffolding to each other. We report our observations of eight sibling pairs between ages 6 and 10, playing four popular games of different gaming paradigms. We found that certain patterns of sibling behavior persisted through all game sessions, regardless of the play patterns afforded by the different games, and that parents reports were consistent with our observations. We also observed instances where game design seemed to influence sibling play dynamics. We share our insights into considerations for designing for sibling play, including specialized social dynamics, opportunities for scaffolding, and the particular challenges they present.

Electric Agents is a cross-media game that presents new ways for children to actively engage with television content. In the game Manny, a member of the Pranksters, steals words out of the mouth of Hector, a member of The Electric Company team, and hides them to prevent the story from progressing. Children collaborate through a mobile augmented reality experience to find and collect the missing vocabulary words relevant to the show narrative. The players return the stolen words back to the show by "throwing" the words towards the television using their mobile devices. This blend of a narrative and a game strives to make educational television content more engaging and participatory while fostering collaborative play and use of specific vocabulary words.

This paper outlines a series of experiments to develop asynchronous messaging systems for preschool aged children. Three unique systems build on a foundational design called Toaster, a jack-in-the box toy with embedded mobile phone that allows children to playfully take and share electronic media. Orange Toaster allows children to create and share self-portraits; Family Toast allows children to browse family photos with physical tokens, and shares their self-portrait reactions with remote family members; Play with Elmo allows children and distant adults to asynchronously share playful video messages. Observations with over 30 children suggest that asynchronous photographic or video messaging with very young children is possible. The results of these studies indicate specific guidelines including (1) children's UI's need to be playful and immediate (2) UI designs for children should create the "here and now" feel of real-time interaction, and (3) adults' UI's must provide emotionally meaningful feedback from children to engage adult users.

How might we build on the physical interactivity of children's play to help children communicate over a distance? Pokaboo is a networked toy for children ages 2--5 to physically play even when they are far apart. Envisioned almost like a low-frame rate video chat, the system combines physically-linked buttons with photo and audio communication. A child will press a button down to take their own photo, and their self-portrait will pop up on their partner's device. The device was tested with both photo sharing and video chat. Children were most engaged when the buttons were part of a mobile video chat, where one child could press a button down and see their partner's button pop up in front of their far-away playmate. When the playmate responded with a button press, their button would magically pop up in front of them, in a form of physical call-and-response. Pokaboo shows how networked toys can help children to form engaging connections through physical play over a distance.

StoryVisit allows children and long-distance adults to experience a sense of togetherness by reading children's story books together over a distance. StoryVisit combines video conferencing and connected books: remote grown-up and child readers can see and hear each other, and can also see and control the same e-book. We report on research with 61 families - over 200 users including parents, children and long-distance readers - who used StoryVisit in their homes with a long-distance reader for at least one reading session. In addition, we report qualitative findings regarding nineteen of the families who participated in telephone interviews and four families who were monitored and interviewed by researchers at home. Results show that connected e-book video chat sessions last about five times as long as the typical video chats reported in previous research on families with young children. Moreover, the addition of an animated character increased session lengths by another 50%. StoryVisit usage peaked for families with three year olds, showing that sustained distance interactions with very young children are possible if communication technologies incorporate joint activities that engage children and adults.

In this work, we introduce Interactive Rich Reading, a new enhanced book experience designed to run on smartphones and a tablet device. Interactive Rich Reading is characterized by a video-based conversational agent that asks questions or makes comments about the current page and is specifically designed to promote engagement with the contents of children's books. We use video compositing techniques to overlay the conversational agent directly over the book contents, creating a magical experience for children by bringing the book to life. We describe technical issues related to enabling this experience on mobile platforms for easier adoption of this technique by other researchers and practitioners.

Long-distance families are increasingly staying connected with free video conferencing tools. However research has highlighted a need for shared activities for long-distance family communication. While video technology is reportedly superior to audio-only tools for children under age 7, the tools themselves are not designed to accommodate children's or families' needs. This paper introduces four design explorations of shared play activities over video conferencing that support family togetherness between children and remote adult family members. We build on research in CSCW and child development to create opportunities for silliness and open-ended play between adults and young children. Our goal is to scaffold interaction across distance and generations.

What if children's make-believe characters could keep in touch when the children were apart? We propose a novel concept for children's use of technology through imagination play: user interfaces designed to be used by children's character toys rather than directly by the children ("doll-computer interfaces"). We apply this model to the challenge of remote communication for children with an enhanced dollhouse containing small-scale interfaces for the dolls with a variety of fully functional multimodal communication functions. Using this interface as a technology probe, we explore a variety of design decisions with remote pairs of children. Our preliminary results suggest that toy-perspective and manipulable toy elements are particularly helpful in supporting play and successful use of communication technologies, while the "true-to-life" toy aspects are sensitive to individual frames of reference and more flexible interfaces that still fit within the toy context lead to creative communication strategies. We found that different communication channels offered interesting tradeoffs between uninterrupted play and rich verbal description. We also learned that the concept appeals to a wide age range but that the youngest children may need additional scaffolding for successful remote play.

We introduce Family Story Play, a system that supports grandparents to read books together with their grandchildren over the Internet. Family Story Play is designed to improve communication across generations and over a distance, and to support parents and grandparents in fostering the literacy development of young children. The interface encourages active child participation in the book reading experience by combining a paper book, a sensor-enhanced frame, video conferencing technology, and video content of a Sesame Street Muppet (Elmo). Results with users indicate that Family Story Play improves child engagement in long-distance communication and increases the quality of interaction between young children and distant grandparents. Additionally, Family Story Play encourages dialogic reading styles that are linked with literacy development. Ultimately, reading with Family Story Play becomes a creative shared activity that suggests a new kind of collaborative story telling.

Increasingly, smartphones are being used to access all manner of information: email messages, Facebook status updates, tweets, RSS feeds, photographs and more. Approaches to dealing with this multi-faceted information stream developed on the desktop, such as switching between multiple applications or multiple browser windows, are unwieldy and scale poorly for mobile devices. In this paper, we propose the combination of the universal inbox and a system called 'Lenses' for extracting information of interest as part of a solution to this problem. These mechanisms allow the user to easily specify ways to sort, filter and manage their universal inbox in an intuitive way. We culminate with a discussion of implications for mobile phone interface design.

We interviewed and observed families in their homes to understand how they communicate across generations and across distances. The phone is still the most common way for keeping children in touch with distant relatives. However, many children can't talk on the phone by themselves until 7 or 8 years old. This paper examines the challenges children have with phone conversations, and looks at how families are currently working around these issues. These findings can help inform the design of future family communications technologies.

Gesture recognition is emerging as an engaging interaction technique in mobile scenarios, and high recognition rates promote user acceptance. Several factors influence recognition rates including the nature of the gesture set and the suitability of the gesture recognition algorithm. This work explores how seamfulness in gesture stroke visualization affects recognition rates. We present the results of a user evaluation of a gesture recognition system that shows that raw (seamful) visualization of low-delity gesture stroke data has recognition rates comparable to no feedback. Providing filtered (seamless) stroke visualization to the user, while retaining the un-filtered input data for recognition, resulted in a 34.9% improvement in gesture recognition rate over raw stroke data. The results provide insights into the broader design space of seamful design, and identifies areas where seamlessness is advantageous.

Mobile phones with integrated video cameras have become ubiquitous tools that people use both to document everyday surroundings and to express themselves artistically. In this paper we report the findings of a user study on user created mobile videos, where the actions of 11 active mobile video users were documented for 2 weeks, the collected material including diaries, device logs, and altogether 255 videos. We describe the patterns related to the creation, sharing and consuming mobile videos, revealing characteristics of both context and content of the video material.

iStuff Mobile is the first rapid prototyping framework that helps explore new sensor-based interfaces with existing mobile phones. It focuses on sensor-enhanced physical interfaces for ubiquitous computing scenarios. The framework includes sensor network platforms, mobile phone software, and a proven rapid prototyping framework. Interaction designers can use iStuff Mobile to quickly create and test functional prototypes of novel interfaces without making internal hardware or software modifications to the handset. A visual programming paradigm provides a low threshold for prototyping activities: the system is not difficult to learn. At the same time, the range of examples built using the toolkit demonstrates a high ceiling for prototyping activities: the toolkit places few limits on prototype complexity. A user study shows that the visual programming metaphor enables prototypes to be built faster and encourages more iterations than a previous approach.

Ubiquitous computing environments accrete slowly over time rather than springing into existence all at once. Mechanisms are needed for incremental integration- the problem of how to incrementally add or modify behaviors in existing ubicomp environments. Examples include adding new input modalities and choreographing the behavior of existing independent applications. The iROS event heap, via its publish-subscribe coordination mechanism, provides the foundation for interoperation through event intermediation, but does not directly provide facilities for expressing these intermediations. The patch panel provides a general facility for retargeting event flow. Intermediations can be expressed as simple event translation mappings or as more complex finite-state machines. We describe an implemented prototype of the patch panel, including examples of its use drawn from real life applications in production use in the iRoom ubiquitous computing environment.

The iStuff toolkit of physical devices, and the flexible software infrastructure to support it, were designed to simplify the exploration of novel interaction techniques in the post-desktop era of multiple users, devices, systems and applications collaborating in an interactive environment. The toolkit leverages an existing interactive workspace in-frastructure, making it lightweight and platform independent. The supporting software framework includes a dynamically configurable intermediary to simplify the mapping of devices to applications. We describe the iStuff architecture and provide several examples of iStuff, organized into a design space of ubiquitous computing interaction components. The main contribution is a physical toolkit for distributed, heterogeneous environments with run-time retargetable device data flow. We conclude with some insights and experiences derived from using this toolkit and framework to prototype experimental interaction techniques for ubiquitous computing environments.