Haptic technology can be used as a tool for learning. Can even the haptic elements in a smartwatch teach a new skill? Here we present a case of using a smartwatch for passive tactile learning. We use the Sony Smartwatch 3 to teach users Morse code while they wear the watch but focus on unrelated tasks. An initial hypothesis forecasted that the stimulation from the smartwatch, typically used for message alerts, would be too subtle to enable haptic learning; however, we find significant improvements in six participants using the technique. Furthermore, we expose participants to two different durations of stimulation and find different results.

The CopyCat game is an interactive educational adventure game to help deaf children improve their language and memory abilities. As part of the CopyCat project, several computer-assisted language learning games have been designed, one of the games “Alien” is shown in Figure 1. Each game entails some sort of quest by the hero to collect items in order to solve a problem. In each quest, the child interacts with the hero (Iris the white cat) via American Sign Language (ASL) to warn her of a villain or identify where a hidden object is located. The child may view the tutor repeatedly if they so choose (see Figure 1). After the child talks to the hero, the child's signing is classified as correct or incorrect. If the child's signing is incorrect, a question mark appears above the hero's head to simulate misunderstanding by the hero, and the child must try again to communicate accurately. If the child's sign is correct, the hero, with the wave of a paw, “poofs” the villain, turning it into an innocuous item, and the hero continues on the quest.

We investigate the potential of the Kinect depth-mapping camera for sign language recognition and verification for educational games for deaf children. We compare a prototype Kinect-based system to our current CopyCat system which uses colored gloves and embedded accelerometers to track children's hand movements. If successful, a Kinect-based approach could improve interactivity, user comfort, system robustness, system sustainability, cost, and ease of deployment. We collected a total of 1000 American Sign Language (ASL) phrases across both systems. On adult data, the Kinect system resulted in 51.5% and 76.12% sentence verification rates when the users were seated and standing respectively. These rates are comparable to the 74.82% verification rate when using the current(seated) CopyCat system. While the Kinect computer vision system requires more tuning for seated use, the results suggest that the Kinect may be a viable option for sign verification.

We perform real-time American Sign Language (ASL) phrase verification for an educational game, CopyCat, which is designed to improve deaf children's signing skills. Taking advantage of context information in the game we verify a phrase, using Hidden Markov Models (HMMs), by applying a rejection threshold on the probability of the observed sequence for each sign in the phrase. We tested this approach using 1204 signed phrase samples from 11 deaf children playing the game during the phase two deployment of CopyCat. The CopyCat data set is particularly challenging because sign samples are collected during live game play and contain many variations in signing and disfluencies. We achieved a phrase verification accuracy of 83% compared to 90% real-time performance by a sign linguist. We report on the techniques required to reach this level of performance.

We present a concept for augmented reality entertainment, called AR Karaoke, where users perform their favorite dramatic scenes with virtual actors. AR Karaoke is the acting equivalent of traditional Karaoke, where the goal is to facilitate an acting experience for the user that is entertaining for both the user and audience. Prototype implementations were created to evaluate various user interfaces and design approach reveal guidelines that are relevant to the design of mixed reality applications in the domains of gaming, performance, and entertainment.